MongoDB Source vs. MongoDB CDC
The client stores operational data in MongoDB. Flink reads it two ways, and they are different connectors with different semantics — the asymmetry flagged back in the connector map. Picking wrong is a real design error, so this lesson is about the difference, not the setup.
The plain mongodb source: a point-in-time read
The plain connector reads the collection as it is now and produces an append stream — the same shape as a bounded file read. Good for a snapshot or a lookup; it does not see later changes.
CREATE TABLE jobs_snapshot (
_id STRING,
engine STRING,
status STRING
) WITH (
'connector' = 'mongodb',
'uri' = 'mongodb://localhost:27017',
'database' = 'dremio',
'collection'= 'jobs'
);
The mongodb-cdc source: a live changelog
The CDC connector does a snapshot first, then streams every insert, update, and delete continuously using MongoDB's change streams (via Debezium). It is source-only — it can never be a sink. Note the two differences from the plain source: a different connector value, and a mandatory PRIMARY KEY.
CREATE TABLE jobs_cdc (
_id STRING,
engine STRING,
status STRING,
PRIMARY KEY(_id) NOT ENFORCED
) WITH (
'connector' = 'mongodb-cdc',
'hosts' = 'localhost:27017,localhost:27018,localhost:27019',
'username' = 'flinkuser',
'password' = 'flinkpw',
'database' = 'dremio',
'collection' = 'jobs'
);
_id as the primary key. You can't use any other column, because a delete event carries only _id. This isn't boilerplate — it's forced by how MongoDB emits changes.
The two sources side by side
mongodb | mongodb-cdc | |
|---|---|---|
| Direction | Source & sink | Source only |
| Boundedness | Bounded (reads now, stops) | Unbounded (snapshot, then live) |
| Stream kind | Append | Upsert changelog |
| Sees later changes? | No — rerun to refresh | Yes — insert/update/delete |
| Primary key | Not required | Required (_id) |
| Infra needed | Any MongoDB | Replica set / sharded cluster |
What this means for the Iceberg sink
This is where it connects to Lesson 5. An append source (plain Mongo, or files) maps to a plain append-only Iceberg write — the one you already ran. An upsert changelog source (Mongo CDC) can't just append: the Iceberg table has to apply updates and deletes, so it must be created with upsert enabled and a matching primary key:
CREATE TABLE iceberg.dremio.jobs_live (
_id STRING,
engine STRING,
status STRING,
PRIMARY KEY(_id) NOT ENFORCED
) WITH (
'format-version' = '2',
'write.upsert.enabled' = 'true'
);
Iceberg format-version 2 is what supports row-level deletes. Point an append-only table at a changelog source and the updates/deletes have nowhere to go — the single most common CDC-to-Iceberg mistake, and the reason the connector map called append-vs-changelog the hinge.
Retrieval check
Why must a MongoDB CDC table declare _id as its primary key?
Correct. MongoDB emits no before-image, so Flink produces an upsert changelog, which needs a unique key — and a delete event carries only _id.
Not quite. It's the upsert changelog that needs the key — a delete event carries only _id, so no other column works.
Which pairing of source and Iceberg write is correct?
Correct. Upsert changelog (Mongo CDC) requires an Iceberg v2 table with upsert enabled and a key; plain append sources write to a plain table.
No — append sources go to append tables; a changelog source is the one that needs upsert enabled and a primary key.
Practice
- From memory, fill the six-row comparison table. Check against the glossary entries for append vs. changelog.
- The client asks for a "live view of running jobs" in Iceberg. Which Mongo connector, and what must the Iceberg table declare? Answer in two sentences.
- If you only need last night's job counts once a day, which connector is the simpler fit, and why?