/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote193 views1 answer
Oracle list of long running queries
SQL
6 votes201 views4 answers
Eloquent to sql with bindings
SQL
18 votes286 views3 answers
Get the list of all tables in sql server
SQL
0 vote273 views1 answer
Postgres row expiration
SQL
18 votes201 views6 answers
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
SQL
12 votes247 views6 answers
How to install sqlite3 in python
SQL
-1 vote234 views4 answers
Restart mysql server ubuntu
SQL
0 vote203 views3 answers
Create and attach user to a postgresql database
SQL
0 vote211 views1 answer
Netstat -tln mysql
SQL
0 vote206 views1 answer
Find most frequent value in SQL column
SQL