/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote185 views1 answer
How to find columns with null values in sql
SQL
1 vote204 views1 answer
Apex select list ORA-20999
SQL
1 vote193 views1 answer
Rename table column name in mysql
SQL
0 vote177 views1 answer
Symfony\Component\Debug\Exception\FatalThrowableError : Class 'Doctrine\DBAL\Driver\PDOSqlite\Driver' not found
SQL
2 votes208 views2 answers
Spring where to put the data sql
SQL
9 votes193 views2 answers
Sql limit order by
SQL
0 vote218 views1 answer
Sql server beginning of month
SQL
0 vote179 views1 answer
Suse status MySQL
SQL
0 vote195 views1 answer
TSQL Code Snippet For Optimized Sales Query
SQL
0 vote349 views2 answers
Illuminate\Database\QueryExceptionSQLSTATE[HY000] [2002] No such file or directory
SQL