The Connector Map

Time: ~8 minutes. Tangible win: name the four connectors this engagement needs and the one tradeoff that defines each — enough to say "reach for this one, because…" in a design conversation.

Lesson 1 gave you the skeleton: sources → transforms → sinks, declared as tables. This lesson populates it with the specific connectors the Dremio engagement calls for, and — more importantly — the single defining tradeoff of each. Memorising option names is low value; knowing why you'd pick one is what a client pays for.

Primary source Flink SQL Connectors — Overview — the built-in connector list, including which are source-only, sink-only, or both.

Two axes that classify every connector

Before the map, two questions sort any connector you'll meet:

Those two axes are the whole reason the four connectors below behave differently. Keep them in mind as you read the map.

The map for this engagement

ConnectorDirectionProducesDefining tradeoff
filesystem
job-history logs
Source & sink Bounded by default; unbounded append in continuous mode Simplest possible source — but you choose bounded (read once) vs. continuous (watch for new files), and you must pick a format.
mongodb
operational data
Source & sink Bounded append (point-in-time read) Reads the collection as it is now. Simple, but it does not see later changes — rerun to refresh.
mongodb-cdc
operational data, live
Source only Unbounded changelog (snapshot, then live changes) Captures inserts/updates/deletes continuously via Debezium — but cannot be a sink, needs a Mongo replica set, and emits a changelog, not plain rows.
iceberg
the output
Source & sink Table-format writes through a catalog What Dremio reads natively — but every write goes through a catalog, so the catalog choice is a real decision (covered later).
The one asymmetry worth memorising mongodb and mongodb-cdc are two different connectors, not one connector with a flag. The plain mongodb connector gives you a bounded snapshot and can also sink. The mongodb-cdc connector gives you a live changelog and is source-only. Lesson 6 drills exactly how their output differs.

Why "append vs. changelog" is the hinge

The deepest split in the map is not file-vs-database — it's what the stream means:

This decides your Iceberg write mode Append sources map cleanly to append-only Iceberg writes. A changelog source (MongoDB CDC) requires an Iceberg table configured to accept upserts/deletes, with a primary key. If you only remember one consequence from this lesson, make it this one — it's the question that most often surprises people on their first CDC-to-Iceberg pipeline.

Beyond the engagement (breadth, not depth)

Flink ships many more connectors — kafka, jdbc, kinesis, firehose, dynamodb, elasticsearch, hbase, and testing helpers like datagen and print. You won't drill these, but knowing they exist — and that they classify on the same two axes — means you can place any new source a client mentions. The overview table marks the direction of each.

Retrieval check

Which statement about the two MongoDB connectors is correct?

Correct. The plain connector reads a point-in-time snapshot (and can sink); the CDC connector streams inserts/updates/deletes and cannot be a sink.

Not quite. Plain mongodb is bounded append and can sink; mongodb-cdc is a source-only changelog stream — don't swap their roles.

Why does a changelog source change how you write to Iceberg?

Correct. A changelog carries insert/update/delete, so the Iceberg table needs upsert/delete handling and a primary key — not plain append.

No — it's about meaning, not speed. A changelog carries updates and deletes, so the sink must apply them, which append-only writes can't.

Practice

  1. From memory, write the four connectors and one tradeoff each. Check against the table above and the glossary.
  2. For each of the client's two inputs — job-history files and MongoDB — decide: bounded or continuous? append or changelog? Note why.
  3. A teammate says "just use the MongoDB connector for live changes." State the one-sentence correction.
Ask the agent: "If the client needs live Mongo changes and a batch of old log files in the same table, is that one job or two?" Next up: standing up a local Flink SQL environment and actually reading a directory of job-history files — your first hands-on win.