A Local Flink SQL Environment + Reading Files
Lessons 1–2 were the model. Now you run it. By the end you'll have a Flink SQL Client talking to a live cluster, reading the client's job-history-style JSON files. The Iceberg sink comes in Lesson 5 — this lesson stops at reading the source, so the one new skill (standing up Flink) lands cleanly.
-f script.sql runner work. Dremio's Getting Started with Flink SQL and Iceberg mirrors this setup.
Step 1 — Sample job-history files
Create a directory with two small JSON files, one object per line (newline-delimited JSON, which Flink's json format reads row by row):
mkdir -p ~/flink-lab/data/job-history ~/flink-lab/warehouse
cat > ~/flink-lab/data/job-history/2026-10-13-batch1.json <<'EOF'
{"job_id":"job-001","engine":"preview","status":"COMPLETED","duration_ms":1240}
{"job_id":"job-002","engine":"preview","status":"FAILED","duration_ms":308}
{"job_id":"job-003","engine":"general","status":"COMPLETED","duration_ms":9820}
EOF
cat > ~/flink-lab/data/job-history/2026-10-13-batch2.json <<'EOF'
{"job_id":"job-004","engine":"general","status":"FAILED","duration_ms":512}
{"job_id":"job-005","engine":"preview","status":"COMPLETED","duration_ms":2044}
EOF
Step 2 — The two JARs Flink needs
The stock Flink image already ships the FileSystem and JSON connectors. For the Iceberg work in later lessons you add two JARs now, so the environment is ready once:
mkdir -p ~/flink-lab/jars && cd ~/flink-lab/jars
curl -sSLO https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.18/1.6.1/iceberg-flink-runtime-1.18-1.6.1.jar
curl -sSLO https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
iceberg-flink-runtime-1.18 is the Iceberg↔Flink bridge, built to match Flink 1.18 — the connector version must track your Flink version. The flink-shaded-hadoop uber JAR supplies the Hadoop classes Iceberg's Hadoop catalog and the Parquet writer need. Miss the Hadoop JAR and the Iceberg catalog fails to initialise.
Step 3 — The source table and read
Put this in ~/flink-lab/read.sql. It declares the FileSystem source from Lesson 1 and reads it:
SET 'execution.runtime-mode' = 'batch';
SET 'sql-client.execution.result-mode' = 'tableau';
CREATE TABLE job_history (
job_id STRING,
engine STRING,
status STRING,
duration_ms BIGINT
) WITH (
'connector' = 'filesystem',
'path' = 'file:///data/job-history/',
'format' = 'json'
);
SELECT * FROM job_history;
Step 4 — Run it
One container starts a local cluster, then runs your script. The volume mounts map your lab directories to the paths named in the SQL:
docker run --rm --user root \
-v ~/flink-lab/data:/data \
-v ~/flink-lab/warehouse:/warehouse \
-v ~/flink-lab/jars:/extra-jars \
-v ~/flink-lab/read.sql:/read.sql \
flink:1.18.1-scala_2.12-java11 \
bash -lc '
cp /extra-jars/*.jar $FLINK_HOME/lib/
$FLINK_HOME/bin/start-cluster.sh
sleep 10
$FLINK_HOME/bin/sql-client.sh -f /read.sql
'
+---------+---------+-----------+-------------+
| job_id | engine | status | duration_ms |
+---------+---------+-----------+-------------+
| job-004 | general | FAILED | 512 |
| job-005 | preview | COMPLETED | 2044 |
| job-001 | preview | COMPLETED | 1240 |
| job-002 | preview | FAILED | 308 |
| job-003 | general | COMPLETED | 9820 |
+---------+---------+-----------+-------------+
5 rows in set
Five rows across two files, read as one table. That is the FileSystem source doing its whole job: it treats the directory as the table, not any single file.
--user root for the lab
Copying JARs into $FLINK_HOME/lib and editing config needs write access the default container user doesn't have. --user root is fine for a throwaway local lab; a real deployment bakes the JARs into a custom image instead of copying at runtime.
Why batch mode here
You set execution.runtime-mode = batch because a fixed set of files is a bounded source — it has an end. The job reads everything and stops. Lesson 4 flips this to continuous mode, where the same directory becomes an unbounded source the job watches forever. Same connector, different boundedness — exactly the axis from Lesson 2.
Retrieval check
In the source table, what does the 'path' pointing at a directory mean?
Correct. The FileSystem connector reads the whole directory as a single table — two files of job history became five rows of one job_history table.
Not quite. The connector reads the entire directory as one table, combining the rows from every file it contains.
Why is execution.runtime-mode = batch the right choice for this read?
Correct. A fixed directory is bounded — the job reads it all and finishes. Continuous watching would be unbounded (Lesson 4).
No — it's about boundedness, not speed or format. A fixed set of files is finite, so batch mode fits.
Practice
- Run the four steps on your own machine. Confirm you get five rows back.
- Add a third JSON file with two more jobs, re-run, and watch the row count rise to seven — proof the directory is the table.
- Change
SELECT *toSELECT engine, COUNT(*) FROM job_history GROUP BY engine. What do you expect before you run it?
jdk.compiler warnings — do those matter?" (They don't; they're harmless module warnings on Java 11.) Next lesson keeps this exact source but makes it continuous — Flink watches the directory and picks up new files as they land.