Design Recommendation
Eight lessons of parts; now the assembly. The client has two inputs — job-history log files and MongoDB operational data — both landing in Iceberg for Dremio. This lesson makes the calls and justifies each, the way you would in a design review.
Pipeline A — Dremio job-history logs → Iceberg
Files landing continuously as engines run jobs, needed in Iceberg for analysis in Dremio.
| Decision | Recommendation | Why (lesson) |
|---|---|---|
| Connector | filesystem + json format | Logs are files; connector and format are separate choices (L1) |
| Source mode | Two jobs: bounded backfill, then continuous | Backfill the history once; keep up with new files via source.monitor-interval (L3, L4) |
| Sink | Iceberg, plain append table | A file read is an append stream — no upserts needed (L2, L5) |
| Guarantee | Exactly-once (checkpointing on) | Checkpoints + Iceberg atomic commits; no dropped or doubled job records (L8) |
| Catalog | Polaris REST in prod; Hadoop locally | Dremio's Open Catalog is Polaris; only the catalog block changes (L7) |
Pipeline B — MongoDB operational data → Iceberg
Here the recommendation forks on a question only the client can answer: do they need a live view, or a periodic snapshot?
| Decision | If live view needed | If periodic snapshot |
|---|---|---|
| Connector | mongodb-cdc (source-only) | mongodb (bounded) |
| Stream kind | Upsert changelog | Append |
| Iceberg table | v2, upsert enabled, PRIMARY KEY(_id) | Plain append table |
| Infra cost | Replica set + always-on job | Any Mongo + scheduled run |
| Lesson | L2, L6 | L2, L6 |
Recommendation: default to the plain mongodb bounded read unless the client has a concrete need for sub-minute freshness. CDC is strictly more powerful and strictly more operationally expensive — a replica set, an always-running job, and the upsert-table complexity. Reach for it only when "live" is a real requirement, not a nice-to-have. This is the laziest-that-works call, and it's defensible precisely because it names what CDC costs.
What you can now do
- Classify any source a client names by two axes — direction and boundedness (L2).
- Stand up Flink SQL and run a real file-to-Iceberg pipeline, and read the files it produced (L3–L5).
- Choose a source mode, a delivery guarantee, and a catalog, and defend each (L4, L7, L8).
- Explain why CDC changes the Iceberg write, and when its cost is justified (L6, L9).
Retrieval check
Why default to plain mongodb over mongodb-cdc for Pipeline B?
Correct. CDC needs a replica set, an always-on job, and an upsert table — worth it only when sub-minute freshness is a genuine requirement.
Not quite. CDC can feed Iceberg (via upsert); the point is it costs more, so default to the simpler read unless "live" is truly needed.
What ties both pipelines into one coherent architecture?
Correct. Different tables, same catalog — Flink fills, the catalog governs, Dremio reads. The shared catalog is the integration.
No — they use different sources and can be separate jobs. The shared catalog is what unifies them for Dremio.
Practice — the deliverable
- Write the two recommendation tables above in your own words, from memory, as a one-page design note for the client.
- For Pipeline B, write the single question you'd ask the client to settle the live-vs-snapshot fork.
- Draw the Flink → catalog → Dremio triangle, and mark where each of this course's nine lessons sits on it.
MISSION.md as follow-ons.