Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

13 votes134 views2 answers
Year to datetime python
Python
0 vote183 views1 answer
Convert numpy array to tfrecord and back
Python
3 votes193 views2 answers
Flask db migrate
Python
0 vote289 views1 answer
Python PyDrive service account credentials
Python
4 votes170 views1 answer
Python dictionary add key-value pair
Python
1 vote255 views1 answer
Easiest way to position labels in tkinter
Python
0 vote182 views1 answer
Access django server from another machine
Python
0 vote192 views1 answer
Python as-lookup
Python
0 vote183 views1 answer
Python int in list
Python
-2 votes185 views1 answer
Ipython install
Python