The Iceberg Sink

Time: ~10 minutes. Tangible win: write a Flink table into a real Iceberg table through a catalog, then see the snapshot, manifests, and Parquet files it produced on disk. Every output below is from a real run — including the files Dremio would read.

You can read files (Lessons 3–4). Now you write the result into Iceberg — the output the whole engagement exists for, because Dremio queries Iceberg natively. The new idea here is the catalog: Flink never writes to an Iceberg table directly, only through a catalog that tracks where tables live.

Primary source Iceberg Flink Getting Started — CREATE CATALOG, DDL, and read/write. Dremio's Flink SQL + Iceberg guide walks the same path.

Step 1 — Register an Iceberg catalog

A catalog is a first-class object you create before any Iceberg table exists. This one is a Hadoop catalog backed by a local directory — the simplest catalog, zero external services:

CREATE CATALOG iceberg WITH (
  'type'         = 'iceberg',
  'catalog-type' = 'hadoop',
  'warehouse'    = 'file:///warehouse'
);

CREATE DATABASE IF NOT EXISTS iceberg.dremio;
The catalog is the decision, not the detail catalog-type = 'hadoop' is the laziest choice that works locally. In the client's world it becomes 'rest' (Apache Polaris — Dremio's Open Catalog), 'nessie', or AWS Glue. The CREATE TABLE and INSERT that follow are identical regardless — only this one block changes. Lesson 7 is entirely about this choice.

Step 2 — Create the Iceberg table and write to it

Note the fully-qualified name: iceberg.dremio.failed_jobs is catalog.database.table. The INSERT is the same transform from Lesson 1 — keep only failed jobs:

CREATE TABLE IF NOT EXISTS iceberg.dremio.failed_jobs (
  job_id      STRING,
  engine      STRING,
  duration_ms BIGINT
);

INSERT INTO iceberg.dremio.failed_jobs
SELECT job_id, engine, duration_ms
FROM job_history
WHERE status = 'FAILED';
Real gotcha: the INSERT returns before the data lands In script mode (sql-client.sh -f), an INSERT is submitted as an asynchronous job and the runner moves on immediately. A SELECT placed right after it in the same script ran before the commit and returned Empty set — on a verified run. The write wasn't broken; the read was simply too early. Read the table back in a separate invocation, or wait for the job to finish, and the rows are there. This surprises nearly everyone the first time.

Step 3 — Read it back (fresh run)

Verified output — reading iceberg.dremio.failed_jobs in a separate run, after the commit:

+---------+---------+-------------+
|  job_id |  engine | duration_ms |
+---------+---------+-------------+
| job-004 | general |         512 |
| job-002 | preview |         308 |
+---------+---------+-------------+
2 rows in set

Two FAILED jobs out of the five source rows — the transform worked, and the result is now a durable Iceberg table.

What Flink actually wrote on disk

This is the payoff worth internalising: an Iceberg "table" is a tree of files, not a database object. Here is the real on-disk layout the write produced:

warehouse/dremio/failed_jobs/
  data/
    00000-0-42eaeb5c-...-00001.parquet        ← the actual rows
  metadata/
    v1.metadata.json                          ← table before the write (empty)
    v2.metadata.json                          ← table after the commit
    snap-4469045629319297262-...-.avro        ← manifest list for the snapshot
    a9aab11c-...-m0.avro                       ← manifest (lists data files)
    version-hint.text                          ← points at the current metadata

Every commit writes a new metadata.json and a new snapshot — that's where Iceberg's time travel and atomic commits come from. The real snapshot summary Flink recorded:

operation         : append
added-records     : 2
total-records     : 2
added-data-files  : 1
engine-name       : flink
engine-version    : 1.18.1
iceberg-version   : Apache Iceberg 1.6.1
This is exactly what Dremio reads When Dremio queries this table, it reads version-hint.text → the current metadata.json → the snapshot's manifest list → the manifests → the Parquet data files. Flink produced that whole structure; nothing extra is needed to make it Dremio-readable. The glossary defines each piece. The underlying format is the same one covered in the Apache Polaris course.

Retrieval check

Why must you run CREATE CATALOG before creating an Iceberg table?

Correct. The catalog is the system of record for where tables live; Flink has no way to address an Iceberg table without one.

Not quite. The catalog isn't about compression or formats — it's the required registry of where Iceberg tables live.

A SELECT right after an INSERT in one script returned Empty set. Why?

Correct. The INSERT is submitted async and the script moves on; the read ran before the commit landed. Read in a separate run and the rows appear.

No — the write succeeded (the files were on disk). The read was simply too early, because the insert job runs asynchronously.

Practice

  1. Run the full pipeline (source from Lesson 3 + this sink). Then find warehouse/ -type f and match each file to the tree above.
  2. Open the latest metadata.json and find the current-snapshot-id and the summary. Confirm added-records matches your FAILED count.
  3. Insert again (append a new FAILED job to the source first). Watch a v3.metadata.json and a second snapshot appear — proof every commit is a new snapshot.
Ask the agent: "If two Flink jobs write to the same Iceberg table at once, what stops them clobbering each other?" (Iceberg's atomic commit + optimistic concurrency.) Next lesson switches sources entirely — MongoDB, and the sharp difference between a plain read and a CDC changelog.