Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

4 votes128 views1 answer
Python path to python executable
Python
0 vote131 views1 answer
Access kwargs in template django
Python
0 vote127 views1 answer
Python type hint superclass
Python
6 votes198 views3 answers
Pandas remove rows with null in column
Python
1 vote140 views1 answer
Plot image without axes python
Python
0 vote118 views1 answer
Compare lists element wise python
Python
3 votes122 views2 answers
Discord.py make command admin only
Python
1 vote125 views1 answer
Python get timestamp 2020-04-23T12:00:00Z
Python
2 votes90 views1 answer
Python check if email is valid
Python
1 vote126 views1 answer
Cv2 draw box
Python