/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote136 views1 answer
Laravel eloquent get generated sql
SQL
0 vote130 views1 answer
Apache Derby: Create SQL Dump with data
SQL
6 votes135 views2 answers
Mysql interval 1 day
SQL
2 votes129 views1 answer
Possgress drop if exists table
SQL
0 vote214 views1 answer
Drop check command
SQL
0 vote127 views1 answer
Mysql identified by syntax error
SQL
0 vote143 views1 answer
Sql getdate minus 1 day without time
SQL
36 votes133 views6 answers
Rename table sql
SQL
1 vote171 views2 answers
Sql split string by space
SQL
2 votes144 views2 answers
How to run postgresql in linux
SQL