/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote99 views1 answer
Dasebase_url-postgres for windows
SQL
7 votes113 views2 answers
Mysql get year from date
SQL
2 votes109 views1 answer
Mysql virtual column
SQL
3 votes107 views1 answer
Oracle sleep
SQL
18 votes109 views6 answers
Laravel form validation
SQL
3 votes122 views1 answer
Where id is in list sql
SQL
20 votes107 views4 answers
Reset auto increment in MySQL
SQL
0 vote108 views2 answers
Zsh: command not found: mysq
SQL
0 vote115 views1 answer
Oracle concat datetime
SQL
-1 vote105 views2 answers
Postgresql user permissions to database
SQL