Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

8 votes108 views2 answers
Python kivy
Python
1 vote111 views1 answer
Def dft(self, array): stack = Stack() visited = set() stack.append(self) while len(stack) > 0 and while len(visited) >= 0: current = stack.pop() array.append(current) visited.add(current) return array
Python
13 votes116 views2 answers
Python programming online editor
Python
4 votes116 views3 answers
Plt vertical line
Python
0 vote102 views1 answer
Django update field after save
Python
-1 vote152 views2 answers
Ai assistant in python
Python
0 vote150 views1 answer
Pypi modules for 3d gui
Python
0 vote112 views1 answer
Python boolean ungleich
Python
2 votes109 views1 answer
How to return total elements in database django
Python
-1 vote111 views1 answer
Global list python
Python