/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote122 views2 answers
Starting mysql service from mac
SQL
0 vote130 views1 answer
How to add unique key constraint in mysql
SQL
0 vote122 views2 answers
Pop sql insert value into
SQL
0 vote142 views2 answers
Reseed sql identity
SQL
0 vote114 views1 answer
Tsql generate rows
SQL
4 votes120 views2 answers
Insert current date in mysql
SQL
0 vote147 views1 answer
Convert sqlalchemy.util._collections to list of string
SQL
0 vote133 views1 answer
Mysql local password denied
SQL
0 vote115 views1 answer
SQL Server format datetime
SQL
2 votes118 views1 answer
Convert series number to date in sql
SQL