/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

14 votes115 views3 answers
Oracle sql limit results
SQL
0 vote105 views2 answers
How to update date in oracle
SQL
7 votes102 views4 answers
What is my mysql version
SQL
0 vote115 views1 answer
Mysql 8 error on identified by
SQL
0 vote116 views1 answer
Connect laravel to mysql on mac
SQL
2 votes112 views1 answer
Postgresql format date dd/mm/yyyy
SQL
1 vote104 views1 answer
Get table which have specific columns in mysql
SQL
0 vote106 views1 answer
Nvl2 syntax
SQL
0 vote117 views1 answer
Drop database using terminal postgres
SQL
0 vote123 views1 answer
. ERROR! The server quit without updating PID file (/usr/local/var/mysql/MacBook-Pro-de-Quentin.local.pid).
SQL