Pyspark mapreduce dataframe

df.rdd \
  .filter(lambda x: x[1] == "france") \ # only french stations
  .map(lambda x: (x[0], x[2])) \ # select station & temp
  .mapValues(lambda x: (x, 1)) \ # generate count
  .reduceByKey(lambda x, y: (x[0]+y[0], x[1]+y[1])) \ # calculate sum & count
  .mapValues(lambda x: x[0]/x[1]) \ # calculate average
  .sortBy(lambda x: x[1], ascending = False) \ # sort
  .take(100)

Source: stackoverflow.com

Similar codes

5 votes201 views2 answers
Allauth django
Python Django
-1 vote141 views1 answer
Percent chart seaborn
Python
0 vote147 views1 answer
How to open html file in python
Python
0 vote154 views1 answer
How to find the ascii value of character in python
Python
0 vote162 views1 answer
Sns time series plot
Python
2 votes152 views2 answers
Can immigrants vote in uk
Python
1 vote162 views1 answer
Python change base function
Python
0 vote138 views1 answer
Nested For loop in Python
Python
3 votes184 views1 answer
Django flush
Python Django
3 votes145 views1 answer
Python get day name
Python