/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

4 votes178 views2 answers
Postgresql default value
SQL
14 votes1.2K views1 answer
Sql value of string
SQL
5 votes173 views1 answer
Oracle sysdate
SQL
0 vote188 views1 answer
Sql new line output
SQL
1 vote187 views1 answer
Mysql workbench in ubuntu 14.04
SQL
1 vote217 views1 answer
Sql where max date
SQL
0 vote171 views1 answer
Alter table myisam to innodb
SQL
0 vote172 views1 answer
Not between mysql
SQL
9 votes201 views2 answers
Mac mysql this is incompatible with sql_mode=only_full_group_by
SQL
2 votes173 views1 answer
Q operator in plsql
SQL