/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote183 views1 answer
New rails app with mysql
SQL
0 vote179 views1 answer
Drop a field in psql django
SQL
11 votes191 views2 answers
Oracle saurce code
SQL
1 vote169 views1 answer
How to between in sql
SQL
0 vote247 views1 answer
Consulta alias con inner join
SQL
6 votes201 views4 answers
Mysql group by month
SQL
1 vote198 views1 answer
Load mysql dumb
SQL
3 votes191 views1 answer
Start mysql server by terminal in linux
SQL
0 vote189 views1 answer
Get the Db column names from a SqlDataReader
SQL
5 votes184 views2 answers
Oracle apex debug mode
SQL