Design Recommendation

Time: ~9 minutes. Tangible win: a defensible, written recommendation for the client's two pipelines — source mode, delivery guarantee, catalog — with every choice traceable to an earlier lesson. This is the deliverable.

Eight lessons of parts; now the assembly. The client has two inputs — job-history log files and MongoDB operational data — both landing in Iceberg for Dremio. This lesson makes the calls and justifies each, the way you would in a design review.

Primary source Your own earlier lessons — each recommendation below cites the lesson that establishes it. For the end-to-end reference pattern, Dremio's Flink SQL + Iceberg guide.

Pipeline A — Dremio job-history logs → Iceberg

Files landing continuously as engines run jobs, needed in Iceberg for analysis in Dremio.

DecisionRecommendationWhy (lesson)
Connectorfilesystem + json formatLogs are files; connector and format are separate choices (L1)
Source modeTwo jobs: bounded backfill, then continuousBackfill the history once; keep up with new files via source.monitor-interval (L3, L4)
SinkIceberg, plain append tableA file read is an append stream — no upserts needed (L2, L5)
GuaranteeExactly-once (checkpointing on)Checkpoints + Iceberg atomic commits; no dropped or doubled job records (L8)
CatalogPolaris REST in prod; Hadoop locallyDremio's Open Catalog is Polaris; only the catalog block changes (L7)
The one flag to raise with the client The FileSystem source assumes write-once files (L4). If an engine rewrites a job-history file in place rather than rolling a new one, continuous mode won't re-read it. Confirm the logs are append-only/rolled before committing to this design — it's the question most likely to bite later.

Pipeline B — MongoDB operational data → Iceberg

Here the recommendation forks on a question only the client can answer: do they need a live view, or a periodic snapshot?

DecisionIf live view neededIf periodic snapshot
Connectormongodb-cdc (source-only)mongodb (bounded)
Stream kindUpsert changelogAppend
Iceberg tablev2, upsert enabled, PRIMARY KEY(_id)Plain append table
Infra costReplica set + always-on jobAny Mongo + scheduled run
LessonL2, L6L2, L6

Recommendation: default to the plain mongodb bounded read unless the client has a concrete need for sub-minute freshness. CDC is strictly more powerful and strictly more operationally expensive — a replica set, an always-running job, and the upsert-table complexity. Reach for it only when "live" is a real requirement, not a nice-to-have. This is the laziest-that-works call, and it's defensible precisely because it names what CDC costs.

Both pipelines share one catalog — that's the architecture Pipelines A and B write to different Iceberg tables but through the same catalog, and Dremio reads them through that same catalog (L7). That single shared catalog is the whole integration: Flink fills, the catalog governs, Dremio reads. If you remember one diagram from this course, make it that triangle.

What you can now do

Retrieval check

Why default to plain mongodb over mongodb-cdc for Pipeline B?

Correct. CDC needs a replica set, an always-on job, and an upsert table — worth it only when sub-minute freshness is a genuine requirement.

Not quite. CDC can feed Iceberg (via upsert); the point is it costs more, so default to the simpler read unless "live" is truly needed.

What ties both pipelines into one coherent architecture?

Correct. Different tables, same catalog — Flink fills, the catalog governs, Dremio reads. The shared catalog is the integration.

No — they use different sources and can be separate jobs. The shared catalog is what unifies them for Dremio.

Practice — the deliverable

  1. Write the two recommendation tables above in your own words, from memory, as a one-page design note for the client.
  2. For Pipeline B, write the single question you'd ask the client to settle the live-vs-snapshot fork.
  3. Draw the Flink → catalog → Dremio triangle, and mark where each of this course's nine lessons sits on it.
Ask the agent: "Review my one-page design note — where would a Dremio architect push back?" You've finished the connector course. The natural next step is the Java DataStream API (for logic beyond SQL) and closing the loop by querying these tables from a real Dremio instance — both flagged in MISSION.md as follow-ons.