/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote111 views1 answer
Flask sqlalchemy update row
SQL
1 vote117 views1 answer
Mysql random limit 1
SQL
0 vote119 views1 answer
Sql row number in result set
SQL
0 vote163 views1 answer
Osm2pgsql mac
SQL
3 votes127 views1 answer
Sql datetime as date
SQL
0 vote108 views1 answer
Show specific events on mysql
SQL
1 vote111 views2 answers
Sql server backup table
SQL
1 vote122 views2 answers
Snowflake alter column data type
SQL
25 votes114 views6 answers
Import mysql database command line
SQL
0 vote123 views1 answer
What is between keyword used for
SQL