/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote147 views1 answer
Postgres statistics
SQL
7 votes155 views3 answers
Sql set
SQL
0 vote140 views2 answers
Search stored procedures
SQL
0 vote140 views1 answer
Sql date d'aujourd'hui
SQL
0 vote177 views1 answer
BigQuery Remove Duplicate Keys From Table
SQL
0 vote139 views1 answer
Oracle current timestamp
SQL
0 vote146 views1 answer
Cara menampilkan tabel yang tidak mengandung kata di sql server
SQL
1 vote146 views1 answer
Mysql size int
SQL
2 votes142 views1 answer
Oracle drop table free up space
SQL
1 vote132 views1 answer
Index a database column sql
SQL