Apache Spark

What Is Apache Spark?

Apache Spark is an open-source engine for processing data across one computer or a cluster of computers. It provides APIs for SQL queries, batch processing, stream processing, machine learning, and graph workloads. Applications can be written in Python, Scala, Java, or R.

Spark divides a large computation into smaller tasks and schedules those tasks across available CPU cores and cluster nodes. Its execution engine can retain reusable intermediate data in memory, but Spark is not an in-memory-only system. It can read from and write to disk, object storage, distributed file systems, databases, and other supported data sources.

The official project website is spark.apache.org, and the source code is maintained in the Apache Spark GitHub repository.

What Apache Spark Is Used For

Apache Spark is used when a workload requires parallel data processing, a unified interface for several analytics tasks, or the ability to scale beyond the resources of one process. Common use cases include:

• Batch data processing: Cleaning, transforming, joining, and aggregating large datasets.
• Extract, transform, and load pipelines: Reading raw data, applying business rules, and writing curated tables or files.
• Interactive SQL analytics: Querying structured data through Spark SQL and DataFrames.
• Stream processing: Processing continuously arriving events with Structured Streaming.
• Machine learning: Building feature-processing and model-training pipelines with MLlib.
• Graph processing: Representing and analyzing graphs with the Scala-based GraphX API.
• Exploratory analysis: Working with data interactively from notebooks or language shells before converting the analysis into a scheduled application.

Spark is not a database or a permanent storage system. It is a computation engine that normally works with data stored elsewhere.

Apache Spark Components and APIs

Spark Component	Purpose	Typical API
Spark Core	Task scheduling, memory management, fault recovery, input/output, and the RDD abstraction	RDD APIs
Spark SQL	Structured-data processing, SQL queries, DataFrames, and table operations	SQL, DataFrame, and Dataset APIs
Structured Streaming	Incremental processing of streaming data using the structured APIs	DataFrame and Dataset APIs
MLlib	Feature engineering, machine-learning algorithms, evaluation, and pipelines	DataFrame-based machine-learning API
GraphX	Graph-parallel computation and graph algorithms	Scala graph API

These components share the Spark execution engine. A project can therefore use SQL transformations, streaming inputs, and machine-learning stages without operating a separate distributed engine for every stage.

Apache Spark Architecture: Driver, Executors, and Cluster Manager

A Spark application consists of a driver process and one or more executor processes. A cluster manager supplies resources when the application runs on a cluster.

• Driver: Runs the application’s main logic, creates the Spark session or context, builds execution plans, requests resources, and schedules work.
• Cluster manager: Allocates CPU and memory resources to Spark applications. Spark can work with its standalone cluster manager, Apache Hadoop YARN, or Kubernetes.
• Executors: Processes launched for an application on worker nodes. Executors run tasks and can cache data for reuse.
• Tasks: Units of work sent to executor slots. A task usually processes one data partition for one stage.

The driver must remain available while the application is running because it coordinates execution. Executors are normally dedicated to one Spark application and are released when that application ends.

How a Spark Job Moves from Code to Executors

1. The application creates a SparkSession.
2. The driver records transformations such as filters, joins, and aggregations.
3. An action requests a result and causes Spark to prepare an execution plan.
4. Spark separates the work into jobs, stages, and tasks according to dependencies and data exchanges.
5. The scheduler sends tasks to executors, generally one task per partition for a given stage.
6. Executors process their partitions and return results to the driver or write output to external storage.

Spark Jobs, Stages, Tasks, and Partitions

A partition is a logical slice of a distributed dataset. Spark processes different partitions in parallel when sufficient executor slots are available.

Execution Term	Meaning
Application	The complete Spark program containing a driver and its executors
Job	Work initiated by an action such as count(), collect(), or a write operation
Stage	A group of tasks that can run without an intervening shuffle boundary
Task	Work performed on one partition during a stage
Partition	A logical portion of a distributed dataset
Shuffle	Redistribution of data between partitions, often required by joins, grouping, sorting, or repartitioning

Partition count affects parallelism and task overhead. Too few partitions can leave CPU cores idle, while many very small partitions can add scheduling and file-management overhead. There is no universal partition count; it depends on data size, cluster resources, source layout, and the operations being performed.

Lazy Evaluation, Transformations, and Actions in Spark

Most Spark transformations are evaluated lazily. Calling a transformation records how a dataset should be produced but does not immediately process all its rows. An action asks Spark to calculate or store a result.

Operation Category	Examples	Effect
Transformation	select, filter, withColumn, groupBy, join	Returns a new logical dataset and contributes to the execution plan
Action	count, collect, show, and write operations	Triggers computation or materializes output

Lazy evaluation allows Spark SQL to optimize a series of DataFrame operations before executing them. It also means that an input or transformation error may not appear until the application calls an action.

RDD, DataFrame, and Dataset APIs in Apache Spark

Abstraction	Description	When It Is Used
RDD	A distributed collection of JVM or language objects processed through functional transformations	Low-level control, unstructured records, or code that depends on RDD-specific behavior
DataFrame	A distributed table with named columns and a schema	Most structured batch, SQL, ETL, and streaming workloads
Dataset	A strongly typed structured API available in Scala and Java	JVM applications that require typed records while retaining Spark SQL planning

Python does not expose the typed Dataset API in the same form as Scala and Java. PySpark applications generally use DataFrames for structured workloads. RDDs remain available, but DataFrames give the optimizer more information about columns, data types, filters, and aggregations.

Installing Apache Spark for Local Development

For a local Python tutorial, install a supported Python version and Java runtime, then install PySpark in an isolated environment. Consult the current Apache Spark documentation for the Java and Python versions supported by the Spark release you intend to use.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install pyspark
python -c "import pyspark; print(pyspark.__version__)"

On Windows PowerShell, activate the virtual environment with .venv\Scripts\Activate.ps1. If the project must match an existing cluster, install the PySpark version compatible with that cluster instead of automatically choosing a different version.

Another option is to download a packaged Spark distribution from the official Apache Spark download page. A packaged distribution includes command-line tools such as spark-submit, pyspark, and spark-shell.

First PySpark DataFrame Program

The following program creates a local Spark session, builds a DataFrame, filters rows, groups the remaining records by department, and calculates an average salary.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .appName("EmployeeSummary")
    .master("local[*]")
    .getOrCreate()
)

employees = [
    (101, "Anita", "Sales", 72000),
    (102, "Ravi", "Engineering", 95000),
    (103, "Meera", "Sales", 68000),
    (104, "Ishaan", "Engineering", 105000),
    (105, "Nisha", "Support", 54000),
]

columns = ["employee_id", "name", "department", "salary"]
df = spark.createDataFrame(employees, columns)

summary = (
    df.filter(F.col("salary") >= 60000)
      .groupBy("department")
      .agg(
          F.count("employee_id").alias("employee_count"),
          F.round(F.avg("salary"), 2).alias("average_salary"),
      )
      .orderBy("department")
)

summary.show()
spark.stop()

PySpark DataFrame Program Output

+-----------+--------------+--------------+
| department|employee_count|average_salary|
+-----------+--------------+--------------+
|Engineering|             2|      100000.0|
|      Sales|             2|       70000.0|
+-----------+--------------+--------------+

The local[*] master runs Spark locally and allows it to use the available logical CPU cores. Cluster applications normally obtain their master setting through deployment configuration rather than hard-coding it in application code.

Running a PySpark Application with spark-submit

Save the previous Python program as employee_summary.py. If a Spark distribution is installed and its bin directory is available on the command path, submit the program locally with:

spark-submit --master "local[*]" employee_summary.py

The same application can be submitted to a configured standalone, YARN, or Kubernetes environment by supplying the appropriate master and deployment options. Cluster-specific settings include executor memory, driver memory, executor cores, dependency packages, configuration files, and application arguments.

Reading and Writing Files with Spark DataFrames

Spark DataFrames can work with formats such as CSV, JSON, Parquet, ORC, and text. Additional connectors can provide access to databases, object stores, messaging systems, and table formats.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("SalesFiles").getOrCreate()

sales = (
    spark.read
         .option("header", True)
         .option("inferSchema", True)
         .csv("input/sales.csv")
)

valid_sales = sales.filter("quantity > 0 AND unit_price >= 0")

(
    valid_sales.write
               .mode("overwrite")
               .partitionBy("sale_year")
               .parquet("output/valid_sales")
)

spark.stop()

Schema inference is convenient for a small tutorial, but production pipelines commonly provide an explicit schema. An explicit schema avoids extra inference work and prevents malformed or inconsistent input from silently changing the intended column types.

Querying a Spark DataFrame with Spark SQL

A DataFrame can be registered as a temporary view and queried with SQL. The temporary view belongs to the current Spark session and does not by itself create a permanent table.

employees_df.createOrReplaceTempView("employees")

result_df = spark.sql("""
    SELECT
        department,
        COUNT(*) AS employee_count,
        ROUND(AVG(salary), 2) AS average_salary
    FROM employees
    WHERE salary >= 60000
    GROUP BY department
    ORDER BY department
""")

result_df.show()

The SQL query and equivalent DataFrame expressions use the same structured execution engine. Teams can choose the interface that makes a particular transformation easier to review and maintain.

Apache Spark Structured Streaming

Structured Streaming applies DataFrame-style operations to unbounded input. Developers describe a streaming query using familiar selections, filters, aggregations, joins, and output operations. Spark then executes the query incrementally as new data arrives.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("SocketWordCount").getOrCreate()

lines = (
    spark.readStream
         .format("socket")
         .option("host", "localhost")
         .option("port", 9999)
         .load()
)

words = lines.select(
    F.explode(F.split(F.col("value"), r"\s+")).alias("word")
)

counts = words.filter(F.col("word") != "").groupBy("word").count()

query = (
    counts.writeStream
          .outputMode("complete")
          .format("console")
          .option("checkpointLocation", "checkpoints/word-count")
          .start()
)

query.awaitTermination()

The socket source and console sink are appropriate for local demonstrations, not durable production pipelines. A production design should choose supported sources and sinks, define checkpoint storage, plan for late or duplicate data, and determine whether event-time watermarks are required.

Apache Spark Deployment Modes

Deployment Choice	Role in a Spark Application	Typical Context
Local mode	Runs the driver and execution threads on one machine	Learning, development, unit tests, and small workloads
Spark standalone cluster	Uses Spark’s included cluster manager	A dedicated Spark cluster with relatively direct setup
Hadoop YARN	Allocates resources through a Hadoop cluster’s resource manager	Organizations already operating a YARN environment
Kubernetes	Runs Spark driver and executor processes in pods	Container-based infrastructure and Kubernetes operations

Cluster mode and client mode describe where the driver runs relative to the submission client. In cluster mode, the cluster infrastructure launches the driver. In client mode, the driver remains in the submitting client process, while executors run on cluster nodes. Supported combinations and command options depend on the selected cluster manager.

Apache Spark Caching and Persistence

Caching can reduce repeated computation when the same DataFrame or RDD is reused by multiple actions. It should be applied selectively because cached data consumes executor storage memory and can displace other blocks.

active_customers = customers_df.filter("status = 'ACTIVE'").cache()

active_count = active_customers.count()
country_totals = active_customers.groupBy("country").count()
country_totals.show()

active_customers.unpersist()

The first action materializes the cache. Later actions can reuse it if the cached blocks remain available. Call unpersist() when the reused dataset is no longer needed. Caching data used by only one action often adds work without providing a benefit.

Apache Spark Performance Practices

• Prefer structured APIs for structured data: DataFrames expose schema and expression information that Spark can use while planning execution.
• Read only required columns: Select needed columns early and use file formats that support column pruning.
• Filter data early: Reducing rows before joins and aggregations can lower subsequent processing and shuffle volume.
• Inspect query plans: Use DataFrame.explain() and the Spark web interface to examine scans, joins, exchanges, stages, and task behavior.
• Avoid unrestricted collect() calls: collect() transfers every result row to the driver and can exhaust driver memory. Use it only when the result is known to be small.
• Control shuffle partitions: Review partition counts after large joins or aggregations instead of relying on one setting for every dataset.
• Address data skew: A heavily repeated join or grouping key can create one or more tasks much larger than the others.
• Use broadcast joins selectively: Broadcasting a genuinely small relation can avoid a large shuffle, but broadcasting an unexpectedly large relation can create memory pressure.
• Avoid many tiny output files: Excessive partitioning can create file-management overhead in the storage system and downstream readers.
• Cache only reused results: Persist an expensive intermediate dataset when multiple actions will reuse it, then remove it when finished.

Apache Spark Fault Tolerance

Spark records how distributed datasets are derived from their inputs. If an executor loses a partition, Spark can often recompute that partition from the recorded lineage. Shuffle files, cached blocks, external systems, and streaming state introduce additional recovery considerations.

Structured Streaming uses checkpoint data to preserve query progress and state across compatible restarts. Reliable recovery also depends on the guarantees of the selected source, sink, application logic, and storage used for checkpoints. A checkpoint directory should be durable and isolated for the specific streaming query.

Apache Spark vs Hadoop MapReduce

Apache Spark and Hadoop MapReduce are distributed computation frameworks, but they use different programming and execution models. Hadoop itself is a broader ecosystem that also includes storage and resource-management components such as HDFS and YARN. Spark can use HDFS for storage and YARN for resource allocation, so Spark does not necessarily replace an entire Hadoop environment.

Area	Apache Spark	Hadoop MapReduce
Programming model	DataFrames, SQL, Datasets, RDDs, and multi-stage directed execution plans	Map and reduce phases with intermediate processing between them
Intermediate data	Can remain in memory, spill to disk, or be recomputed	Traditionally materialized through disk-oriented stage boundaries
Workload APIs	Batch, SQL, streaming, machine learning, and graph APIs in one project	Primarily batch processing through the MapReduce model
Storage	Uses external storage systems	Frequently paired with HDFS but remains a computation framework
Resource management	Can run with standalone, YARN, or Kubernetes resource management	Commonly runs through YARN in modern Hadoop environments

Performance depends on the workload, data layout, implementation, configuration, and hardware. It is more accurate to compare a representative pipeline than to rely on a universal speed multiplier.

Apache Spark vs Databricks

Apache Spark is an open-source processing engine. Databricks is a managed data and analytics platform that provides Spark-based runtimes together with platform services such as workspace management, notebooks, job orchestration, governance integrations, and managed infrastructure capabilities.

They are therefore not equivalent product categories. A team can operate Apache Spark directly on supported infrastructure or use a managed platform that includes Spark. The decision depends on operational ownership, integration requirements, governance, workload features, support expectations, and cost.

When Apache Spark May Not Be the Right Tool

• A small dataset can be processed more simply and economically by a single-process program or database query.
• A transactional application requires row-level inserts and updates with database transaction guarantees rather than distributed analytical computation.
• A request-response service requires consistently low per-request latency and should not incur distributed job scheduling overhead.
• The workload requires specialized stream-processing semantics or latency targets that should be evaluated against dedicated streaming engines.
• The organization cannot operate or monitor the necessary cluster, storage, security, and dependency infrastructure and has not selected a managed service.

Apache Spark Tutorial Learning Path

1. Install Spark locally and create a SparkSession.
2. Create DataFrames and inspect their schemas.
3. Practice column selection, filtering, expressions, grouping, and aggregation.
4. Read and write CSV, JSON, and Parquet data with explicit schemas.
5. Learn joins, null handling, window functions, and Spark SQL.
6. Study partitions, shuffles, caching, query plans, and the Spark web interface.
7. Package an application and run it through spark-submit.
8. Learn the selected cluster manager and its driver, executor, resource, and deployment settings.
9. Add Structured Streaming or MLlib only after the DataFrame and execution fundamentals are clear.

Apache Spark Frequently Asked Questions

What is Apache Spark used for?

Apache Spark is used for distributed batch processing, data transformation, SQL analytics, stream processing, machine-learning pipelines, and graph computation. It is commonly placed between external data sources and storage destinations in analytical data pipelines.

Is Apache Spark a database?

No. Spark is a computation engine. It can query tables and files, cache intermediate results, and connect to many storage systems, but permanent storage and database transaction responsibilities belong to external systems.

Which programming languages does Apache Spark support?

Spark provides application APIs for Scala, Java, Python, and R, as well as SQL interfaces for structured data. API availability differs by language; for example, typed Datasets are available in Scala and Java, while PySpark normally uses DataFrames.

Can Apache Spark run on one computer?

Yes. Local mode runs Spark on one computer and is suitable for learning, development, tests, and datasets that fit the machine’s resources. A cluster is required only when the workload needs resources or parallelism beyond one machine, or when it must operate in an existing distributed environment.

Does Apache Spark process all data in memory?

No. Spark can cache selected data in memory, but it also reads and writes external storage, uses shuffle files, spills data to disk when necessary, and recomputes partitions from lineage. Available memory, storage levels, transformations, and configuration determine how a particular application uses memory and disk.

Apache Spark Editorial QA Checklist

• Verify that Spark is described as a processing engine rather than a database or distributed storage system.
• Confirm that architecture explanations distinguish the driver, cluster manager, executors, tasks, stages, and partitions.
• Test every PySpark example with the documented Spark, Python, and Java combination.
• Check that local examples do not present socket sources, console sinks, local checkpoints, or inferred schemas as production defaults.
• Confirm that RDD, DataFrame, and Dataset language support is described accurately.
• Review spark-submit, cluster-manager, and dependency instructions against the Spark version used by the target environment.
• Avoid universal performance claims; evaluate Spark against representative data, transformations, storage, and infrastructure.
• Check that Spark and Databricks are presented as an open-source engine and a managed platform, respectively.

Apache Spark Tutorial Summary

Apache Spark coordinates distributed data processing through a driver, executors, partitions, stages, and tasks. DataFrames and Spark SQL are the usual starting points for structured workloads, while Structured Streaming, MLlib, GraphX, and RDDs address more specific requirements. Effective Spark applications depend on clear schemas, suitable partitioning, controlled shuffles, careful driver usage, measured caching, and deployment settings matched to the available infrastructure.