Apache Flink

Apache Flink is an open-source distributed processing engine for stateful computations over bounded and unbounded data streams. It is commonly used to build real-time data pipelines, event-driven applications, streaming analytics systems, and batch data-processing jobs.

This Apache Flink tutorial explains its processing model, architecture, state management, event-time support, installation process, and a basic Java streaming example. It also clarifies how Flink relates to Apache Kafka and Apache Spark.

What Is Apache Flink?

Apache Flink is a distributed data-processing framework designed around stream processing. It can process continuous streams that have no defined end and bounded datasets that eventually finish.

In Flink, batch processing is treated as a special case of stream processing. A bounded input can be processed using the same general runtime and programming concepts used for an unbounded stream.

• Unbounded streams: Continuously arriving events such as application logs, transactions, sensor readings, or user activity.
• Bounded streams: Finite datasets such as files, archived events, database snapshots, or historical records.
• Stateful processing: Computations can retain information between events, such as counts, sessions, running totals, and detected patterns.
• Distributed execution: A Flink job can run across multiple processes and machines.

What Apache Flink Is Used For

Apache Flink is suitable when an application must process data as it arrives, maintain state across events, or produce continuously updated results.

Real-Time Streaming Analytics with Apache Flink

Flink can calculate metrics from incoming events without waiting for a complete daily or hourly dataset. Examples include transaction totals, website activity, operational dashboards, and continuously updated aggregates.

Apache Flink Event-Driven Applications

An event-driven application responds to events and may retain state about earlier events. Flink can support applications such as rule evaluation, anomaly detection, alert generation, workflow monitoring, and pattern detection.

Apache Flink Data Pipelines and ETL

Flink can read records from a source, transform or enrich them, and write the results to another system. A pipeline may parse events, remove invalid records, join reference data, calculate fields, and send the processed output to storage or another message system.

Batch Processing with Apache Flink

Flink can also process finite data collections. This is useful for historical analysis, data backfills, periodic report generation, and rebuilding derived datasets from archived events.

Apache Flink Processing Concepts

Data Streams and Flink Transformations

A Flink application reads data from one or more sources and applies transformations to create new streams. Common transformations include mapping, filtering, grouping, joining, aggregating, and windowing.

• map converts each input record into another record.
• filter keeps records that satisfy a condition.
• keyBy logically partitions records by a key.
• Window operations group events into finite sections for calculation.
• Process functions provide lower-level access to state, timers, and event-processing behavior.

Keyed Streams in Apache Flink

A keyed stream groups records that share the same key. For example, transactions can be keyed by account ID, sensor readings by device ID, or orders by customer ID.

Records with the same key are routed to the same logical partition. This enables Flink to maintain separate state for each account, device, customer, or other business key.

Windowing in Apache Flink

An unbounded stream does not naturally finish, so Flink uses windows to divide it into finite groups for aggregation. The appropriate window depends on the business requirement.

• Tumbling window: Creates fixed-size, non-overlapping windows.
• Sliding window: Creates windows that can overlap because they advance by a separate slide interval.
• Session window: Groups activity separated by periods of inactivity.
• Global window: Places records into a shared window and relies on a custom trigger to decide when to evaluate it.

Apache Flink Event Time, Processing Time, and Watermarks

Distributed event streams may arrive late or out of order. Flink distinguishes between the time when an event occurred and the time when the system processed it.

• Event time: A timestamp associated with when the event occurred at its source.
• Processing time: The system time of the Flink task that processes the event.
• Ingestion time: A timestamp assigned when data enters the Flink pipeline.

Watermarks are progress indicators used in event-time processing. They help Flink estimate how far event time has advanced and determine when an event-time window can be evaluated.

A watermark strategy should reflect the expected delay and ordering characteristics of the source. An overly aggressive watermark can treat valid delayed events as late, while an overly conservative strategy can delay results.

State Management in Apache Flink

State allows an operator to remember information from earlier records. Stateful operations are required for running totals, deduplication, fraud rules, joins, sessions, and many other streaming computations.

• Keyed state: State associated with records after a stream has been partitioned by a key.
• Operator state: State associated with a parallel operator instance rather than an individual key.
• Managed state: State managed by the Flink runtime so that it can participate in recovery and redistribution.

State may become large in a long-running application. The job design should therefore consider state growth, expiration, checkpoint storage, available memory, and the selected state backend.

Apache Flink Checkpoints, Savepoints, and Recovery

Flink uses snapshots of application state to support recovery and operational changes.

Apache Flink Checkpoints

Checkpoints are runtime-managed state snapshots created at configured intervals. If a recoverable failure occurs, Flink can restore operator state and continue processing from a consistent point, provided the sources and sinks support the required recovery behavior.

Apache Flink Savepoints

Savepoints are manually triggered snapshots intended for operational tasks such as stopping and resuming an application, upgrading job code, changing parallelism, or migrating a job to another cluster.

Checkpoint and savepoint compatibility can depend on operator identifiers, serializer compatibility, job topology, and version-specific behavior. Test restoration before performing a production upgrade.

Apache Flink Architecture

A Flink application is converted into a distributed job and executed by runtime components that coordinate scheduling, resource allocation, data exchange, state, and recovery.

Flink Client

The client prepares the application for submission. It builds the job representation, packages required dependencies when applicable, and submits the job to the Flink deployment.

Flink JobManager

The JobManager coordinates job execution. Its responsibilities include scheduling tasks, coordinating checkpoints, handling recovery, and communicating with worker processes.

Flink TaskManagers and Task Slots

TaskManagers execute the parallel tasks that perform the actual data processing. A TaskManager provides task slots, which represent portions of its processing resources available to execute job subtasks.

The parallelism configured for a job determines how many parallel instances of an operator can run. Effective parallelism also depends on available task slots, source partitions, operator limits, and resource configuration.

Apache Flink APIs

Flink provides multiple APIs for different levels of abstraction and different types of data processing.

• DataStream API: Used to build event-processing applications with transformations, keys, windows, state, and timers.
• Table API: Provides a relational API for working with dynamic tables.
• Flink SQL: Allows streaming and batch transformations to be expressed using SQL.
• Process functions: Provide lower-level control over keyed state, event timestamps, side outputs, and timers.

The API selected for a project should match the required control level. SQL and Table API are suitable for many relational transformations, while the DataStream API and process functions are useful when the application requires custom event-level logic.

Download and Run Apache Flink Locally

For local learning, download a Flink binary distribution that is compatible with the Java version installed on your machine. Extract the archive and use the included scripts to start a local Flink deployment.

The exact filename, Java requirements, and supported commands depend on the selected Flink release. Check the documentation supplied for that release before installation.

Check the Installed Java Version

java -version

Extract the Apache Flink Archive

Replace the placeholder filename with the archive you downloaded.

tar -xzf flink-<version>-bin-scala_<version>.tgz
cd flink-<version>

Start the Local Apache Flink Cluster

./bin/start-cluster.sh

After the cluster starts, open the address reported by the installation documentation for the Flink Web UI. In many local configurations, the interface is available on port 8081.

Stop the Local Apache Flink Cluster

./bin/stop-cluster.sh

Create a Basic Apache Flink Java Streaming Job

The following Java example creates a bounded stream from several strings, converts each value to uppercase, and prints the transformed records. It demonstrates the structure of a small DataStream job without requiring an external source or sink.

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class UppercaseJob {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
                StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromData("flink", "stream processing", "apache")
                .map(String::toUpperCase)
                .print();

        env.execute("Uppercase Flink Job");
    }
}

The application performs four steps:

1. Creates a StreamExecutionEnvironment.
2. Creates a bounded source containing three strings.
3. Maps each string to uppercase and sends it to the print sink.
4. Calls execute to submit the job for execution.

The console output can include task identifiers or parallel subtask prefixes depending on the execution environment. The transformed values will resemble the following:

FLINK
STREAM PROCESSING
APACHE

Add Flink dependencies that match the target runtime version. When submitting a packaged job to a separate Flink cluster, follow that release’s dependency-packaging guidance to avoid bundling incompatible runtime libraries.

Apache Flink Word Count Example

A word-count job shows how to flatten input records, assign a key, and maintain a running count for each word.

import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

import java.util.Arrays;

public class WordCountJob {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
                StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromData(
                    "apache flink processes streams",
                    "flink maintains state"
                )
                .flatMap((String line, org.apache.flink.util.Collector<Tuple2<String, Integer>> out) -> {
                    Arrays.stream(line.toLowerCase().split("\\s+"))
                            .filter(word -> !word.isEmpty())
                            .forEach(word -> out.collect(Tuple2.of(word, 1)));
                })
                .returns(Types.TUPLE(Types.STRING, Types.INT))
                .keyBy(value -> value.f0)
                .sum(1)
                .print();

        env.execute("Apache Flink Word Count");
    }
}

The flatMap operation emits one tuple for every word. keyBy groups tuples by the word in field f0, and sum(1) updates the count stored in field f1.

Apache Flink Sources and Sinks

A source brings records into a Flink job, while a sink sends processed records to another system. Connectors are available for common messaging systems, databases, filesystems, and data platforms, but connector support and packaging vary by Flink release.

• A source may read from a message topic, file, database change stream, socket, or generated collection.
• A sink may write to a database, object store, filesystem, message topic, search system, or custom service.
• Some connectors participate in checkpoint-based recovery and provide stronger delivery guarantees than others.
• Connector compatibility must be checked against the exact Flink runtime version.

Apache Flink and Apache Kafka

Apache Kafka and Apache Flink solve different parts of a streaming architecture and are frequently used together.

Area	Apache Kafka	Apache Flink
Primary role	Durable event storage and transport	Distributed event processing and computation
Data handling	Stores ordered records in partitioned topics	Transforms, joins, aggregates, and evaluates records
Application state	Retains topic records according to configured policies	Maintains managed operator and keyed state
Typical relationship	Acts as a source or sink for event data	Consumes from and produces to Kafka topics

Kafka is not replaced merely because Flink is used. Kafka can provide the event log, while Flink performs stateful calculations over the events. The correct architecture depends on retention, replay, processing, availability, and operational requirements.

Apache Flink and Apache Spark

Apache Flink and Apache Spark are distributed data-processing systems that support both streaming and bounded workloads. Their APIs, runtimes, scheduling behavior, state mechanisms, deployment models, and operational ecosystems differ.

Consideration	Apache Flink	Apache Spark
Processing model	Designed around stream processing for bounded and unbounded data	Supports batch and streaming workloads through its processing APIs
Stateful streaming	Managed keyed state, timers, checkpoints, and event-time operations	Provides stateful streaming features through its streaming APIs
Common evaluation areas	Event-time logic, continuous processing, state size, latency, connectors	Existing data platform, analytics ecosystem, libraries, team experience

A technology choice should be based on a representative workload rather than a general claim that one engine is always faster or better. Test event volume, state size, recovery time, latency, resource use, connector behavior, and operational complexity.

Apache Flink Deployment Options

Flink can be run locally for development or deployed to a cluster environment. Available deployment approaches depend on the selected Flink release and infrastructure.

• Local deployment: Suitable for tutorials, development, and basic testing.
• Standalone cluster: Flink manages execution using its own cluster components.
• Kubernetes deployment: Runs Flink components in a Kubernetes environment.
• Resource-manager integration: Flink releases may support deployment through compatible cluster resource managers.
• Managed services: Cloud providers and data-platform vendors may provide managed Flink offerings with provider-specific configuration and operational behavior.

For production deployment, define resource limits, checkpoint storage, high availability, restart behavior, secrets management, network access, monitoring, logging, application upgrades, and recovery procedures.

Apache Flink Development and Production Practices

• Pin compatible versions: Keep the runtime, connectors, client dependencies, and Java version compatible.
• Assign stable operator identifiers: Stable identifiers can be important when restoring state after changing job code.
• Bound state growth: Use appropriate retention, cleanup, state time-to-live settings, or bounded business logic.
• Configure checkpoints deliberately: Consider interval, timeout, storage location, concurrent checkpoints, and failure handling.
• Plan for late events: Define watermark behavior and decide whether late events should be dropped, updated, or sent to another output.
• Avoid blocking operator threads: Slow synchronous calls can reduce throughput and create backpressure.
• Monitor backpressure: Sustained backpressure can indicate a slow sink, uneven key distribution, insufficient resources, or expensive processing.
• Test recovery: Verify that the job can restore from checkpoints or savepoints before relying on the procedure in production.
• Protect sensitive data: Secure credentials, network connections, state storage, logs, and external systems.

Apache Flink Frequently Asked Questions

What is Apache Flink used for?

Apache Flink is used for stateful stream processing, real-time analytics, event-driven applications, continuous ETL pipelines, fraud or anomaly rules, event enrichment, and bounded batch-processing jobs.

Can Apache Flink process both streaming and batch data?

Yes. Flink processes unbounded streams and bounded datasets. Its runtime is designed around stream processing, with a bounded dataset treated as a stream that has a defined end.

Does Apache Flink replace Apache Kafka?

Not generally. Kafka commonly stores and transports events, while Flink performs computations over those events. A Flink job can consume records from Kafka, transform them, and write results back to Kafka or another system.

What programming languages can be used with Apache Flink?

Java is a primary language for Flink applications, and Flink also provides APIs and interfaces for other supported languages and SQL-based development. Language and API support can vary by release, so verify the documentation for the version being used.

What is the difference between a Flink checkpoint and a savepoint?

A checkpoint is normally created automatically for failure recovery. A savepoint is manually initiated and is generally used for planned operations such as upgrades, migration, stopping and resuming a job, or changing parallelism.

Apache Flink Tutorial QA Checklist

• Confirm that the stated Java version is supported by the Apache Flink release used in the tutorial.
• Verify that all Flink dependencies and connectors match the target cluster runtime version.
• Run the local cluster start and stop commands on the documented operating system.
• Compile and execute the Java examples with the APIs available in the selected Flink version.
• Confirm that the uppercase example produces all three transformed records.
• Test the word-count job with repeated words and verify that counts are grouped by key.
• Validate the explanation of event time, watermarks, windows, keyed state, checkpoints, and savepoints.
• Test checkpoint restoration and savepoint restoration before documenting an upgrade procedure.
• Check connector delivery guarantees against the specific source and sink configurations.
• Verify that Kafka and Spark comparisons describe different roles without unsupported performance claims.
• Review production guidance for state growth, backpressure, high availability, checkpoint storage, and access control.