Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

0 vote107 views1 answer
Label with list comprehension python
Python
0 vote111 views1 answer
Python break list into n lists
Python
0 vote115 views1 answer
Distance of a point from a line python
Python
7 votes118 views3 answers
Divmod
Python
0 vote112 views1 answer
Input code for python
Python
0 vote108 views1 answer
Waitkey in python
Python
-1 vote135 views1 answer
Django template filter
Python Django
0 vote120 views1 answer
Python tensorflow is not defined
Python
0 vote109 views1 answer
Python print functoin
Python
2 votes126 views2 answers
Convert list to generator python
Python