/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

1 vote123 views1 answer
Strict in postgres SQL
SQL
0 vote243 views1 answer
Duplicate table: 7 ERROR: relation "migrations" already exists (SQL: create table "migrations" ("id" serial primary key not null, "migration" varchar(255) not null, "batch" integer not null))
SQL
0 vote155 views1 answer
Redudancy in SQL
SQL
0 vote152 views1 answer
Print boolean in plsql
SQL
0 vote192 views1 answer
Ms sql print more than 1 variable
SQL
0 vote155 views1 answer
Add column not null with default value postgres
SQL
3 votes212 views2 answers
Mysql user permission database
SQL
0 vote141 views1 answer
How to change null display in psql
SQL
1 vote152 views1 answer
Mysql copy table1 to table2
SQL
0 vote165 views1 answer
Define a variable in mysql from select
SQL