Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

0 vote184 views1 answer
Créer fonction python
Python
2 votes167 views1 answer
Python divide every element in a list by a number
Python
16 votes179 views6 answers
List.add in python
Python
1 vote174 views1 answer
Python png library
Python
14 votes181 views3 answers
Pipenv
Python
0 vote172 views1 answer
Rqg3
Python
1 vote206 views1 answer
User group template tag django
Python Django
1 vote170 views2 answers
Number of columns with no missing values
Python
2 votes181 views1 answer
Register modal in django admin
Python Django
0 vote211 views1 answer
Flask wtforms multiple select
Python