Wordcount pyspark

text_file = sc.textFile("hdfs://...")
counts = text_file.flatMap(lambda line: line.split(" ")) \
             .map(lambda word: (word, 1)) \
             .reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("hdfs://...")

Source: spark.apache.org

Similar codes

0 vote147 views1 answer
Open tar file pandas
Python
0 vote146 views1 answer
Dataproc initialization_actions error
Python
0 vote144 views1 answer
Python load file with multiple jsons
Python
0 vote137 views1 answer
Pandas print dataframe dtypes
Python
3 votes157 views2 answers
Python get function docstring
Python
0 vote152 views1 answer
Pandas dataframe to excel hyperlink length limit
Python
0 vote144 views1 answer
Console code page (437) differs from Windows code page (1252) 8-bit characters might not work correctly
Python
0 vote181 views1 answer
Condtion for equal time in selenium python
Python
0 vote134 views1 answer
Convert ascii to binary python
Python
0 vote146 views1 answer
Python strip txt
Python