/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

6 votes200 views3 answers
Can you update NULL in sql
SQL
9 votes226 views3 answers
Sequence postgresql
SQL
7 votes185 views6 answers
Group_concat mysql
SQL
3 votes196 views3 answers
Django.db.utils.OperationalError: no such table:
SQL
3 votes195 views1 answer
How to find names includes in sql
SQL
0 vote233 views1 answer
Transaction and commit trong sql server
SQL
4 votes188 views2 answers
Postgresql reset sequence
SQL
1 vote193 views1 answer
Mamp mysql path mac
SQL
20 votes214 views6 answers
Creating a table in sql
SQL
0 vote194 views1 answer
SQL isnumeric DB2
SQL