/Python/Convert pandas dataframe to spark dataframe

Convert pandas dataframe to spark dataframe

import pandas as pd
from pyspark.sql import SparkSession

filename = <'path to file'>
spark = SparkSession.build.appName('pandasToSpark').getOrCreate()
# Assuming file is csv
pandas_df = pd.read_csv(filename)
spark_df = spark.CreateDataFrame(pandas_df)

Dataframe pandas to spark

from pyspark.sql import SparkSession
#Create PySpark SparkSession
spark = SparkSession.builder \
    .master("local[1]") \
    .appName("SparkByExamples.com") \
    .getOrCreate()
#Create PySpark DataFrame from Pandas
sparkDF=spark.createDataFrame(pandasDF) 
sparkDF.printSchema()
sparkDF.show()

#Outputs below schema & DataFrame

root
 |-- Name: string (nullable = true)
 |-- Age: long (nullable = true)

+------+---+
|  Name|Age|
+------+---+
| Scott| 50|
|  Jeff| 45|
|Thomas| 54|
|   Ann| 34|
+------+---+

Source: sparkbyexamples.com

Similar codes

0 vote148 views2 answers
Numpy random normal seed time
Python
0 vote156 views1 answer
Remove column by index
Python
0 vote155 views2 answers
How to create fibonacci sequence in python
Python
0 vote151 views1 answer
Python mark function as no return
Python
0 vote152 views1 answer
Quoto x discord selfbot
Python
7 votes148 views2 answers
Whatapp
Python
0 vote152 views1 answer
Ouvrir une autre fenetre tkinter
Python
0 vote111 views1 answer
End without space in python
Python
0 vote199 views1 answer
Django blog new post
Python Django
3 votes121 views2 answers
Python replace values in a matrix
Python