Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

5 votes201 views2 answers
Python get min max value from a dictionary
Python
1 vote241 views1 answer
How to redirect to previous page in django
Python Django
41 votes236 views6 answers
Check if element in list Python
Python
0 vote200 views1 answer
PyQT5 reset color
Python
1 vote185 views1 answer
Sns title
Python
31 votes207 views2 answers
Anaconda create environment python version
Python
3 votes198 views3 answers
Pandas date_range
Python
9 votes201 views2 answers
Python readlines without n
Python
0 vote192 views1 answer
Random forest classifier classification report
Python
0 vote176 views1 answer
Import data from website pandas python medium
Python