Python site-packages pyspark

import os

def myfun(x):`
        os.system("pip install shapely")
        return x
rdd = sc.parallelize([1,2,3,4]) ## assuming 4 worker nodes
rdd.map(lambda x: myfun(x)).collect() 
## call each cluster to run the code to import the library

Source: stackoverflow.com

Similar codes

36 votes151 views6 answers
Python pandas
Python
0 vote155 views2 answers
Python capture stdout
Python
-1 vote144 views1 answer
Python if else interview questions
Python
1 vote158 views1 answer
Pandas percentage change across multiple periods
Python
3 votes194 views4 answers
Django setup in windows
Python Django
1 vote156 views2 answers
Types of all columns pandas
Python
16 votes205 views6 answers
Django models
Python Django
0 vote238 views1 answer
Pandas df where row has na
Python
0 vote150 views1 answer
Merge two Querysets in Python Django while preserving Queryset methods
Python
8 votes171 views1 answer
Iterative binary search python
Python