Connect to spark cluster

from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, Catalog

spark_conf = SparkConf()
spark_conf.setAll([
    ('spark.master', 'spark://<master-ip>:7077'),
    ('spark.app.name', 'myApp'),
    ('spark.submit.deployMode', 'client'),
    ('spark.ui.showConsoleProgress', 'true'),
    ('spark.eventLog.enabled', 'false'),
    ('spark.logConf', 'false'),
    ('spark.driver.bindAddress', '0.0.0.0'),
    ('spark.driver.host', '<master-ip>')
])

spark = SparkSession.builder.config(conf=spark_conf).getOrCreate()

Similar codes

11 votes172 views5 answers
Python discord mention user
Python
3 votes143 views1 answer
Input array of string in python
Python
0 vote213 views1 answer
Flask get uploaded file size
Python Flask
1 vote154 views1 answer
Stop a function from continuing when a condition is met python
Python
0 vote144 views1 answer
Rename pandas columns with list of new names
Python
62 votes151 views3 answers
Python how to loop in a list
Python
2 votes152 views2 answers
Print the number of times that the substring occurs in the given string
Python
0 vote145 views1 answer
Como escribir letras griegas en python
Python
0 vote138 views1 answer
Port python script to jupyter notebook
Python
1 vote158 views2 answers
Pandas slicing from one column to another
Python