Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

0 vote103 views1 answer
Convert fisheye video to normal python
Python
1 vote103 views1 answer
To_csv drop index
Python
31 votes94 views1 answer
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 6148: character maps to <undefined>
Python
2 votes92 views1 answer
Directory name python
Python
72 votes99 views6 answers
How to reverse a string in python
Python
1 vote107 views2 answers
Django sum get 0 if none
Python
5 votes113 views1 answer
Dataframe from arrays python
Python
0 vote98 views1 answer
How to make a grading system in python
Python
0 vote101 views1 answer
Close all tables python
Python
0 vote116 views2 answers
Best computer language
Python