The Iceberg Sink
You can read files (Lessons 3–4). Now you write the result into Iceberg — the output the whole engagement exists for, because Dremio queries Iceberg natively. The new idea here is the catalog: Flink never writes to an Iceberg table directly, only through a catalog that tracks where tables live.
CREATE CATALOG, DDL, and read/write. Dremio's Flink SQL + Iceberg guide walks the same path.
Step 1 — Register an Iceberg catalog
A catalog is a first-class object you create before any Iceberg table exists. This one is a Hadoop catalog backed by a local directory — the simplest catalog, zero external services:
CREATE CATALOG iceberg WITH (
'type' = 'iceberg',
'catalog-type' = 'hadoop',
'warehouse' = 'file:///warehouse'
);
CREATE DATABASE IF NOT EXISTS iceberg.dremio;
catalog-type = 'hadoop' is the laziest choice that works locally. In the client's world it becomes 'rest' (Apache Polaris — Dremio's Open Catalog), 'nessie', or AWS Glue. The CREATE TABLE and INSERT that follow are identical regardless — only this one block changes. Lesson 7 is entirely about this choice.
Step 2 — Create the Iceberg table and write to it
Note the fully-qualified name: iceberg.dremio.failed_jobs is catalog.database.table. The INSERT is the same transform from Lesson 1 — keep only failed jobs:
CREATE TABLE IF NOT EXISTS iceberg.dremio.failed_jobs (
job_id STRING,
engine STRING,
duration_ms BIGINT
);
INSERT INTO iceberg.dremio.failed_jobs
SELECT job_id, engine, duration_ms
FROM job_history
WHERE status = 'FAILED';
sql-client.sh -f), an INSERT is submitted as an asynchronous job and the runner moves on immediately. A SELECT placed right after it in the same script ran before the commit and returned Empty set — on a verified run. The write wasn't broken; the read was simply too early. Read the table back in a separate invocation, or wait for the job to finish, and the rows are there. This surprises nearly everyone the first time.
Step 3 — Read it back (fresh run)
+---------+---------+-------------+
| job_id | engine | duration_ms |
+---------+---------+-------------+
| job-004 | general | 512 |
| job-002 | preview | 308 |
+---------+---------+-------------+
2 rows in set
Two FAILED jobs out of the five source rows — the transform worked, and the result is now a durable Iceberg table.
What Flink actually wrote on disk
This is the payoff worth internalising: an Iceberg "table" is a tree of files, not a database object. Here is the real on-disk layout the write produced:
warehouse/dremio/failed_jobs/
data/
00000-0-42eaeb5c-...-00001.parquet ← the actual rows
metadata/
v1.metadata.json ← table before the write (empty)
v2.metadata.json ← table after the commit
snap-4469045629319297262-...-.avro ← manifest list for the snapshot
a9aab11c-...-m0.avro ← manifest (lists data files)
version-hint.text ← points at the current metadata
Every commit writes a new metadata.json and a new snapshot — that's where Iceberg's time travel and atomic commits come from. The real snapshot summary Flink recorded:
operation : append
added-records : 2
total-records : 2
added-data-files : 1
engine-name : flink
engine-version : 1.18.1
iceberg-version : Apache Iceberg 1.6.1
version-hint.text → the current metadata.json → the snapshot's manifest list → the manifests → the Parquet data files. Flink produced that whole structure; nothing extra is needed to make it Dremio-readable. The glossary defines each piece. The underlying format is the same one covered in the Apache Polaris course.
Retrieval check
Why must you run CREATE CATALOG before creating an Iceberg table?
Correct. The catalog is the system of record for where tables live; Flink has no way to address an Iceberg table without one.
Not quite. The catalog isn't about compression or formats — it's the required registry of where Iceberg tables live.
A SELECT right after an INSERT in one script returned Empty set. Why?
Correct. The INSERT is submitted async and the script moves on; the read ran before the commit landed. Read in a separate run and the rows appear.
No — the write succeeded (the files were on disk). The read was simply too early, because the insert job runs asynchronously.
Practice
- Run the full pipeline (source from Lesson 3 + this sink). Then
find warehouse/ -type fand match each file to the tree above. - Open the latest
metadata.jsonand find thecurrent-snapshot-idand thesummary. Confirmadded-recordsmatches your FAILED count. - Insert again (append a new FAILED job to the source first). Watch a
v3.metadata.jsonand a second snapshot appear — proof every commit is a new snapshot.