Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

17 votes196 views3 answers
Making log files in python
Python
0 vote201 views1 answer
How do I iterate over a subfolder in Python
Python
0 vote305 views1 answer
Djangobook.com jwd django restfremwork plugin
Python Django
0 vote194 views1 answer
While attempts > 0:
Python
1 vote254 views2 answers
Call c# from python
Python
1 vote194 views1 answer
What is the meaning of illiteral with base 10
Python
57 votes190 views6 answers
Timedelta python
Python
0 vote210 views1 answer
Optimal alpha jupyter
Python
0 vote174 views1 answer
Binary string to hex python
Python
4 votes208 views4 answers
Get guild by id discord.py
Python