Wordcount pyspark

text_file = sc.textFile("hdfs://...")
counts = text_file.flatMap(lambda line: line.split(" ")) \
             .map(lambda word: (word, 1)) \
             .reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("hdfs://...")

Source: spark.apache.org

Similar codes

16 votes122 views4 answers
Tqdm library
Python
1 vote162 views1 answer
Reload all extensions discord.py
Python
3 votes110 views1 answer
Pandas sheet name
Python
0 vote165 views1 answer
What is tf.repeat()
Python
0 vote148 views1 answer
Python, variables, Print() advanced, Input(), Variables ,INT, STR, FLOAT, BOOL, Casting
Python
0 vote148 views1 answer
Python - remove columns with same name and keep first
Python
1 vote162 views1 answer
Django migrate fake zero
Python
1 vote208 views1 answer
Opposite of .isin pandas
Python
3 votes154 views1 answer
Find angle mbc in python
Python
0 vote139 views1 answer
Ignore bad lines pandas
Python