/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

4 votes141 views2 answers
Oracle set date format
SQL
1 vote148 views1 answer
Sqlalchemy order_by
SQL
0 vote150 views1 answer
Postgresql linux password
SQL
2 votes161 views2 answers
Mysql write into table
SQL
0 vote144 views1 answer
Oracle session statistics
SQL
9 votes148 views3 answers
Find all tables with column name
SQL
0 vote170 views1 answer
Wordpress sql find and replace
SQL
3 votes143 views1 answer
Oracle drop user
SQL
0 vote139 views1 answer
How to fetch unique records from two tables
SQL
12 votes149 views6 answers
Select distinct
SQL