The Connector Map
Lesson 1 gave you the skeleton: sources → transforms → sinks, declared as tables. This lesson populates it with the specific connectors the Dremio engagement calls for, and — more importantly — the single defining tradeoff of each. Memorising option names is low value; knowing why you'd pick one is what a client pays for.
Two axes that classify every connector
Before the map, two questions sort any connector you'll meet:
- Direction — can it be a source, a sink, or both? Some are strictly one.
- Boundedness — does it produce a bounded stream (finite, ends) or an unbounded one (continuous)? And if unbounded, is it an append stream or a changelog stream?
Those two axes are the whole reason the four connectors below behave differently. Keep them in mind as you read the map.
The map for this engagement
| Connector | Direction | Produces | Defining tradeoff |
|---|---|---|---|
filesystemjob-history logs |
Source & sink | Bounded by default; unbounded append in continuous mode | Simplest possible source — but you choose bounded (read once) vs. continuous (watch for new files), and you must pick a format. |
mongodboperational data |
Source & sink | Bounded append (point-in-time read) | Reads the collection as it is now. Simple, but it does not see later changes — rerun to refresh. |
mongodb-cdcoperational data, live |
Source only | Unbounded changelog (snapshot, then live changes) | Captures inserts/updates/deletes continuously via Debezium — but cannot be a sink, needs a Mongo replica set, and emits a changelog, not plain rows. |
icebergthe output |
Source & sink | Table-format writes through a catalog | What Dremio reads natively — but every write goes through a catalog, so the catalog choice is a real decision (covered later). |
mongodb and mongodb-cdc are two different connectors, not one connector with a flag. The plain mongodb connector gives you a bounded snapshot and can also sink. The mongodb-cdc connector gives you a live changelog and is source-only. Lesson 6 drills exactly how their output differs.
Why "append vs. changelog" is the hinge
The deepest split in the map is not file-vs-database — it's what the stream means:
- An append stream says "here is another row." A bounded file read and a plain Mongo read are both append streams. Writing them to Iceberg is a straight accumulation of inserts.
- A changelog stream says "this row was inserted / updated / deleted." A CDC source emits this, because it reflects edits to rows that already exist. Writing it to Iceberg means the sink has to apply updates and deletes, not just append.
Beyond the engagement (breadth, not depth)
Flink ships many more connectors — kafka, jdbc, kinesis, firehose, dynamodb, elasticsearch, hbase, and testing helpers like datagen and print. You won't drill these, but knowing they exist — and that they classify on the same two axes — means you can place any new source a client mentions. The overview table marks the direction of each.
Retrieval check
Which statement about the two MongoDB connectors is correct?
Correct. The plain connector reads a point-in-time snapshot (and can sink); the CDC connector streams inserts/updates/deletes and cannot be a sink.
Not quite. Plain mongodb is bounded append and can sink; mongodb-cdc is a source-only changelog stream — don't swap their roles.
Why does a changelog source change how you write to Iceberg?
Correct. A changelog carries insert/update/delete, so the Iceberg table needs upsert/delete handling and a primary key — not plain append.
No — it's about meaning, not speed. A changelog carries updates and deletes, so the sink must apply them, which append-only writes can't.
Practice
- From memory, write the four connectors and one tradeoff each. Check against the table above and the glossary.
- For each of the client's two inputs — job-history files and MongoDB — decide: bounded or continuous? append or changelog? Note why.
- A teammate says "just use the MongoDB connector for live changes." State the one-sentence correction.