Connect to spark cluster

from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, Catalog

spark_conf = SparkConf()
spark_conf.setAll([
    ('spark.master', 'spark://<master-ip>:7077'),
    ('spark.app.name', 'myApp'),
    ('spark.submit.deployMode', 'client'),
    ('spark.ui.showConsoleProgress', 'true'),
    ('spark.eventLog.enabled', 'false'),
    ('spark.logConf', 'false'),
    ('spark.driver.bindAddress', '0.0.0.0'),
    ('spark.driver.host', '<master-ip>')
])

spark = SparkSession.builder.config(conf=spark_conf).getOrCreate()

Similar codes

0 vote131 views1 answer
How to print class attributes in python
Python
0 vote143 views2 answers
How to Send WhatsApp API using python
Python
1 vote173 views2 answers
Django drop database postgres
Python Django
0 vote153 views1 answer
Python profile is not defined line_profiler
Python
5 votes128 views5 answers
Python space separated input
Python
0 vote141 views1 answer
Python list of deeper paths
Python
3 votes139 views1 answer
Access the value in settings django
Python
0 vote145 views1 answer
Python check if file is writable
Python
6 votes149 views6 answers
What is tuple in python
Python
9 votes165 views2 answers
Default flask app
Python