Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

1 vote117 views1 answer
Bgr2gray opencv
Python
0 vote122 views2 answers
Cron job python
Python
0 vote118 views1 answer
Python drop extension
Python
1 vote147 views3 answers
Check cuda version pytorch
Python
8 votes155 views5 answers
Pop vs remove python
Python
0 vote119 views1 answer
Import pygame module in pycharm
Python
0 vote121 views1 answer
Pyton recognize any datetime format
Python
21 votes143 views5 answers
Format string to 2 decimal places python
Python
1 vote125 views2 answers
How to run single loop iterations on same time in python
Python
0 vote166 views1 answer
Date format django template filter
Python