/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote146 views1 answer
Oracle running queries sql
SQL
0 vote139 views1 answer
How to check default value of column in sql server
SQL
4 votes100 views1 answer
Oracle sleep
SQL
1 vote136 views1 answer
Create another table from existing table sql oracle
SQL
0 vote137 views1 answer
Price-colour
SQL
5 votes151 views1 answer
Oracle create table comment
SQL
1 vote89 views1 answer
Metada in sql
SQL
37 votes178 views6 answers
Mysql update
SQL
94 votes185 views4 answers
Create database postgres
SQL
20 votes137 views3 answers
SQL query rename table
SQL