Catalogs and the Dremio Seam
In Lesson 5 you wrote through a Hadoop catalog, with a note that only the CREATE CATALOG block changes in production. This lesson cashes that in. The catalog is the seam between the two halves of the engagement: Flink writes through it, Dremio reads through it. Pick the same catalog on both sides and the loop closes.
'catalog-type'='rest'. See also the Apache Polaris course in this repo.
What a catalog is, restated
An Iceberg catalog is the system of record for where tables live and what their current snapshot is. It is not storage — the Parquet and metadata files you saw in Lesson 5 live in object storage regardless. The catalog just answers "for table dremio.failed_jobs, which metadata.json is current?" Every engine that agrees on the catalog sees the same tables.
The catalog options, same table and INSERT either way
The crucial thing: swapping catalogs changes only the CREATE CATALOG block. Your CREATE TABLE and INSERT are untouched.
| Catalog type | catalog-type | Who tracks tables | When to reach for it |
|---|---|---|---|
| Hadoop / filesystem | hadoop | Files in a warehouse dir | Local dev, single writer, no services (Lesson 5) |
| REST (Apache Polaris) | rest | A Polaris server over HTTP | Dremio's Open Catalog — the engagement's likely target |
| Nessie | nessie | A Nessie server, Git-like | Branch/tag versioning of catalog state |
| AWS Glue | glue | The Glue Data Catalog | AWS-native lakehouse |
The one block that changes: pointing Flink at Polaris
Because Dremio's Open Catalog is Apache Polaris, the production form of Lesson 5's catalog is a REST catalog with OAuth credentials:
CREATE CATALOG dremio_catalog WITH (
'type' = 'iceberg',
'catalog-type' = 'rest',
'uri' = 'https://<polaris-host>/api/catalog',
'warehouse' = 'dremio',
'credential' = '<client-id>:<client-secret>'
);
Everything after this — CREATE TABLE dremio_catalog.dremio.failed_jobs, the INSERT — is byte-for-byte the pipeline you already ran. That's the whole point of the catalog abstraction: the laziness you took locally (Hadoop) costs you nothing when you graduate to Polaris.
Why not always just use the filesystem catalog?
The Hadoop catalog is simplest but weakest: it assumes effectively one writer and has no access control or multi-engine coordination. The moment two engines (Flink writing, Dremio reading, maybe Spark too) share tables, you want a service-backed catalog — REST/Polaris, Nessie, or Glue — which is exactly the multi-engine payoff the Polaris course opens with.
Retrieval check
When you switch from a Hadoop catalog to a Polaris REST catalog, what changes?
Correct. The catalog abstraction means only the CREATE CATALOG block changes — the table DDL and the write are identical.
Not quite. That's the value of the abstraction: only the catalog block changes, not the table, the insert, or the files.
Why might Dremio see nothing after Flink successfully writes a table?
Correct. The data files exist, but if no shared catalog records them, Dremio's catalog doesn't know the table — both sides must agree on one catalog.
No — Dremio reads Iceberg natively and tables aren't private. The issue is the two engines using different catalogs.
Practice
- Take your working Lesson 5 script and rewrite only the
CREATE CATALOGblock to the Polaris REST form. Note that nothing else needs to change. - In one sentence, explain to the client why "Flink wrote it but Dremio can't see it" is almost always a catalog-mismatch, not a data problem.
- Skim the Polaris course intro and connect its "one catalog, many engines" claim to this lesson's seam.
'catalog-type'='rest' block here?" Next lesson tackles correctness: what exactly-once means when files flow into Iceberg, and why Flink + Iceberg can promise it.