Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

0 vote126 views1 answer
Pandas count show one column
Python
80 votes120 views5 answers
Random.shuffle
Python
2 votes136 views1 answer
Pandas find top 10 values in column
Python
22 votes136 views4 answers
How to remove all characters after a specific character in python?
Python
8 votes126 views3 answers
Len python
Python
3 votes125 views5 answers
Calc
Python
0 vote124 views1 answer
Pyqt5 qpushbutton disable
Python
0 vote131 views1 answer
Jetson nx unable to install matplotlib
Python
0 vote127 views1 answer
Scrapy create project
Python
189 votes203 views10 answers
New Year's Eve
Python