Wordcount pyspark

text_file = sc.textFile("hdfs://...")
counts = text_file.flatMap(lambda line: line.split(" ")) \
             .map(lambda word: (word, 1)) \
             .reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("hdfs://...")

Source: spark.apache.org

Similar codes

0 vote139 views1 answer
Pyplot width
Python
0 vote142 views1 answer
Array with zeros python
Python
0 vote150 views1 answer
Get_permissions
Python
0 vote144 views1 answer
Read the specific length of characters in a text file using the read() function
Python
0 vote165 views1 answer
Parallel loops in python
Python
0 vote132 views1 answer
Django form field add attrs
Python
0 vote145 views1 answer
Python - How to execute a program or call a system command?
Python
0 vote138 views1 answer
How to show rosbag file python
Python
11 votes134 views3 answers
How to send a message in a specific channel discord.py
Python
0 vote142 views1 answer
Adding extra fields to user model django
Python