Wordcount pyspark

text_file = sc.textFile("hdfs://...")
counts = text_file.flatMap(lambda line: line.split(" ")) \
             .map(lambda word: (word, 1)) \
             .reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("hdfs://...")

Source: spark.apache.org

Similar codes

11 votes143 views2 answers
Pandas shuffle rows
Python
0 vote127 views1 answer
Python in a nutshell
Python
7 votes124 views1 answer
How to find where python is located
Python
7 votes133 views4 answers
How to use lambda function in python
Python
0 vote138 views1 answer
What hormone causes the feeling of love
Python
0 vote161 views1 answer
Default ordering django
Python Django
0 vote131 views1 answer
Python login to O365
Python
18 votes121 views5 answers
Python remove punctuation
Python
27 votes128 views6 answers
Install pandas
Python
0 vote123 views1 answer
How to save python variables locally
Python