Catalogs and the Dremio Seam

Time: ~8 minutes. Tangible win: explain why the catalog choice — not the write code — is what lets Dremio read what Flink produced, and name the right catalog for the engagement.

In Lesson 5 you wrote through a Hadoop catalog, with a note that only the CREATE CATALOG block changes in production. This lesson cashes that in. The catalog is the seam between the two halves of the engagement: Flink writes through it, Dremio reads through it. Pick the same catalog on both sides and the loop closes.

Primary source Dremio: Apache Polaris with Apache Flink — Flink writing to Iceberg through Polaris with 'catalog-type'='rest'. See also the Apache Polaris course in this repo.

What a catalog is, restated

An Iceberg catalog is the system of record for where tables live and what their current snapshot is. It is not storage — the Parquet and metadata files you saw in Lesson 5 live in object storage regardless. The catalog just answers "for table dremio.failed_jobs, which metadata.json is current?" Every engine that agrees on the catalog sees the same tables.

The catalog options, same table and INSERT either way

The crucial thing: swapping catalogs changes only the CREATE CATALOG block. Your CREATE TABLE and INSERT are untouched.

Catalog typecatalog-typeWho tracks tablesWhen to reach for it
Hadoop / filesystemhadoopFiles in a warehouse dirLocal dev, single writer, no services (Lesson 5)
REST (Apache Polaris)restA Polaris server over HTTPDremio's Open Catalog — the engagement's likely target
NessienessieA Nessie server, Git-likeBranch/tag versioning of catalog state
AWS GlueglueThe Glue Data CatalogAWS-native lakehouse

The one block that changes: pointing Flink at Polaris

Because Dremio's Open Catalog is Apache Polaris, the production form of Lesson 5's catalog is a REST catalog with OAuth credentials:

CREATE CATALOG dremio_catalog WITH (
  'type'          = 'iceberg',
  'catalog-type'  = 'rest',
  'uri'           = 'https://<polaris-host>/api/catalog',
  'warehouse'     = 'dremio',
  'credential'    = '<client-id>:<client-secret>'
);

Everything after this — CREATE TABLE dremio_catalog.dremio.failed_jobs, the INSERT — is byte-for-byte the pipeline you already ran. That's the whole point of the catalog abstraction: the laziness you took locally (Hadoop) costs you nothing when you graduate to Polaris.

This is where the two courses meet Lesson 5's on-disk snapshot and metadata files are exactly what the Polaris course teaches from the catalog's side. There you learned how Polaris tracks catalogs, namespaces, and tables and vends credentials; here you learned how Flink writes into that. Flink fills the tables; Polaris governs them; Dremio reads them. Same Iceberg table, three roles.
Both sides must point at the same catalog Dremio can only query what its catalog knows about. If Flink writes through a Hadoop catalog on a laptop and Dremio looks at a Polaris catalog, Dremio sees nothing — the data files exist, but no shared catalog records them. "Point both at the same catalog" is the entire integration requirement, and the most common reason a freshly written table is invisible to the query engine.

Why not always just use the filesystem catalog?

The Hadoop catalog is simplest but weakest: it assumes effectively one writer and has no access control or multi-engine coordination. The moment two engines (Flink writing, Dremio reading, maybe Spark too) share tables, you want a service-backed catalog — REST/Polaris, Nessie, or Glue — which is exactly the multi-engine payoff the Polaris course opens with.

Retrieval check

When you switch from a Hadoop catalog to a Polaris REST catalog, what changes?

Correct. The catalog abstraction means only the CREATE CATALOG block changes — the table DDL and the write are identical.

Not quite. That's the value of the abstraction: only the catalog block changes, not the table, the insert, or the files.

Why might Dremio see nothing after Flink successfully writes a table?

Correct. The data files exist, but if no shared catalog records them, Dremio's catalog doesn't know the table — both sides must agree on one catalog.

No — Dremio reads Iceberg natively and tables aren't private. The issue is the two engines using different catalogs.

Practice

  1. Take your working Lesson 5 script and rewrite only the CREATE CATALOG block to the Polaris REST form. Note that nothing else needs to change.
  2. In one sentence, explain to the client why "Flink wrote it but Dremio can't see it" is almost always a catalog-mismatch, not a data problem.
  3. Skim the Polaris course intro and connect its "one catalog, many engines" claim to this lesson's seam.
Ask the agent: "If I already did the Polaris course, which lesson there maps to the 'catalog-type'='rest' block here?" Next lesson tackles correctness: what exactly-once means when files flow into Iceberg, and why Flink + Iceberg can promise it.