Python site-packages pyspark

import os

def myfun(x):`
        os.system("pip install shapely")
        return x
rdd = sc.parallelize([1,2,3,4]) ## assuming 4 worker nodes
rdd.map(lambda x: myfun(x)).collect() 
## call each cluster to run the code to import the library

Source: stackoverflow.com

Similar codes

0 vote206 views1 answer
Python child class init
Python
1 vote219 views1 answer
How to do key sensing in python
Python
0 vote197 views1 answer
Python cgi get raw post data
Python
0 vote232 views1 answer
SparkSession pyspark
Python
0 vote207 views1 answer
Python reverse split only once
Python
5 votes186 views1 answer
How to delete na values in a dataframe
Python
0 vote196 views2 answers
Comment faire un long commentaire en python
Python
0 vote176 views1 answer
Python running standard deviation
Python
0 vote183 views1 answer
Django.db.backends.mysql install
Python
0 vote189 views1 answer
Split the string using the separator
Python