Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

6 votes189 views1 answer
How to open any program on python
Python
12 votes142 views2 answers
Python sort array of dictionary by value
Python
29 votes298 views4 answers
Basic flask app python
Python Flask
0 vote142 views1 answer
Affinity propagation cosine similarity python
Python
2 votes162 views2 answers
AttributeError: module 'django.contrib.auth.views' has no attribute 'login'
Python Django
0 vote134 views1 answer
Pandas nat to null?
Python
0 vote154 views1 answer
Python get image's text from web request
Python
1 vote175 views1 answer
Only include top ten items django for loop
Python Django
0 vote145 views1 answer
TypeError: Object of type DictProxy is not JSON serializable
Python
5 votes141 views2 answers
Pandas replace inf with 0
Python