/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

1 vote205 views1 answer
Tsql replace value in json
SQL
0 vote219 views1 answer
Postgres regex word boundary
SQL
2 votes287 views3 answers
Postgres create view
SQL
0 vote255 views1 answer
Soql last week
SQL
1 vote206 views1 answer
Isnull sql server
SQL
0 vote191 views1 answer
Possgress drop if exists view
SQL
0 vote195 views1 answer
Get last date join sql
SQL
7 votes157 views1 answer
Alter table add key sql
SQL
1 vote215 views1 answer
Create new index takes long time postgresql
SQL
0 vote188 views4 answers
Restart MySQL
SQL