Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

8 votes171 views2 answers
Combine two dataframe in pandas
Python
0 vote154 views1 answer
Base conversion python
Python
0 vote166 views1 answer
Python run async function without await
Python
0 vote166 views1 answer
Inicair venv python
Python
2 votes166 views1 answer
Normalise list python
Python
2 votes187 views1 answer
How to open file dialog in pytohn
Python
1 vote168 views1 answer
Run code with different verions of python
Python
0 vote150 views1 answer
Virtualbox python manage.py not found
Python
5 votes153 views1 answer
How to use a string variable as a variable name in python
Python
1 vote159 views1 answer
How to set gui position tkinter python
Python