A Local Flink SQL Environment + Reading Files

Time: ~10 minutes. Tangible win: run Flink SQL in Docker and read a directory of job-history files into a table — your first real pipeline, verified on a live cluster. Every command and every output below is from a real run (Flink 1.18.1).

Lessons 1–2 were the model. Now you run it. By the end you'll have a Flink SQL Client talking to a live cluster, reading the client's job-history-style JSON files. The Iceberg sink comes in Lesson 5 — this lesson stops at reading the source, so the one new skill (standing up Flink) lands cleanly.

Primary source Flink SQL Client docs — how the interactive client and the -f script.sql runner work. Dremio's Getting Started with Flink SQL and Iceberg mirrors this setup.

Step 1 — Sample job-history files

Create a directory with two small JSON files, one object per line (newline-delimited JSON, which Flink's json format reads row by row):

mkdir -p ~/flink-lab/data/job-history ~/flink-lab/warehouse

cat > ~/flink-lab/data/job-history/2026-10-13-batch1.json <<'EOF'
{"job_id":"job-001","engine":"preview","status":"COMPLETED","duration_ms":1240}
{"job_id":"job-002","engine":"preview","status":"FAILED","duration_ms":308}
{"job_id":"job-003","engine":"general","status":"COMPLETED","duration_ms":9820}
EOF

cat > ~/flink-lab/data/job-history/2026-10-13-batch2.json <<'EOF'
{"job_id":"job-004","engine":"general","status":"FAILED","duration_ms":512}
{"job_id":"job-005","engine":"preview","status":"COMPLETED","duration_ms":2044}
EOF

Step 2 — The two JARs Flink needs

The stock Flink image already ships the FileSystem and JSON connectors. For the Iceberg work in later lessons you add two JARs now, so the environment is ready once:

mkdir -p ~/flink-lab/jars && cd ~/flink-lab/jars
curl -sSLO https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.18/1.6.1/iceberg-flink-runtime-1.18-1.6.1.jar
curl -sSLO https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
Why these two, specifically iceberg-flink-runtime-1.18 is the Iceberg↔Flink bridge, built to match Flink 1.18 — the connector version must track your Flink version. The flink-shaded-hadoop uber JAR supplies the Hadoop classes Iceberg's Hadoop catalog and the Parquet writer need. Miss the Hadoop JAR and the Iceberg catalog fails to initialise.

Step 3 — The source table and read

Put this in ~/flink-lab/read.sql. It declares the FileSystem source from Lesson 1 and reads it:

SET 'execution.runtime-mode' = 'batch';
SET 'sql-client.execution.result-mode' = 'tableau';

CREATE TABLE job_history (
  job_id      STRING,
  engine      STRING,
  status      STRING,
  duration_ms BIGINT
) WITH (
  'connector' = 'filesystem',
  'path'      = 'file:///data/job-history/',
  'format'    = 'json'
);

SELECT * FROM job_history;

Step 4 — Run it

One container starts a local cluster, then runs your script. The volume mounts map your lab directories to the paths named in the SQL:

docker run --rm --user root \
  -v ~/flink-lab/data:/data \
  -v ~/flink-lab/warehouse:/warehouse \
  -v ~/flink-lab/jars:/extra-jars \
  -v ~/flink-lab/read.sql:/read.sql \
  flink:1.18.1-scala_2.12-java11 \
  bash -lc '
    cp /extra-jars/*.jar $FLINK_HOME/lib/
    $FLINK_HOME/bin/start-cluster.sh
    sleep 10
    $FLINK_HOME/bin/sql-client.sh -f /read.sql
  '

Verified output from a real run. The row order can differ (Flink reads files in parallel); the five rows will not.

+---------+---------+-----------+-------------+
|  job_id |  engine |    status | duration_ms |
+---------+---------+-----------+-------------+
| job-004 | general |    FAILED |         512 |
| job-005 | preview | COMPLETED |        2044 |
| job-001 | preview | COMPLETED |        1240 |
| job-002 | preview |    FAILED |         308 |
| job-003 | general | COMPLETED |        9820 |
+---------+---------+-----------+-------------+
5 rows in set

Five rows across two files, read as one table. That is the FileSystem source doing its whole job: it treats the directory as the table, not any single file.

Run as --user root for the lab Copying JARs into $FLINK_HOME/lib and editing config needs write access the default container user doesn't have. --user root is fine for a throwaway local lab; a real deployment bakes the JARs into a custom image instead of copying at runtime.

Why batch mode here

You set execution.runtime-mode = batch because a fixed set of files is a bounded source — it has an end. The job reads everything and stops. Lesson 4 flips this to continuous mode, where the same directory becomes an unbounded source the job watches forever. Same connector, different boundedness — exactly the axis from Lesson 2.

Retrieval check

In the source table, what does the 'path' pointing at a directory mean?

Correct. The FileSystem connector reads the whole directory as a single table — two files of job history became five rows of one job_history table.

Not quite. The connector reads the entire directory as one table, combining the rows from every file it contains.

Why is execution.runtime-mode = batch the right choice for this read?

Correct. A fixed directory is bounded — the job reads it all and finishes. Continuous watching would be unbounded (Lesson 4).

No — it's about boundedness, not speed or format. A fixed set of files is finite, so batch mode fits.

Practice

  1. Run the four steps on your own machine. Confirm you get five rows back.
  2. Add a third JSON file with two more jobs, re-run, and watch the row count rise to seven — proof the directory is the table.
  3. Change SELECT * to SELECT engine, COUNT(*) FROM job_history GROUP BY engine. What do you expect before you run it?
Ask the agent: "The cluster logs printed jdk.compiler warnings — do those matter?" (They don't; they're harmless module warnings on Java 11.) Next lesson keeps this exact source but makes it continuous — Flink watches the directory and picks up new files as they land.