Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

75 votes129 views3 answers
How to get a random element from an array in python
Python
0 vote141 views1 answer
Django.core.exceptions.ImproperlyConfigured: Field name is not valid for model
Python
0 vote138 views1 answer
Create a python file and import it as library in other file
Python
3 votes144 views2 answers
Python os module
Python
0 vote132 views1 answer
Settings urls
Python
12 votes130 views2 answers
Isinstance float or int
Python
14 votes133 views5 answers
Read excel sheet in python
Python
1 vote158 views1 answer
How to Add a overall Title to Seaborn Plots
Python
104 votes173 views4 answers
Python import text file
Python
0 vote166 views1 answer
Django insert bulk data
Python Django