MongoDB Source vs. MongoDB CDC

Time: ~9 minutes. Tangible win: say exactly how a plain MongoDB read differs from a CDC changelog, and why CDC forces a primary key on your Iceberg table. This is the client's other source.

How this lesson is grounded Lessons 3–5 were run on a live cluster. This one is close reading of the official Flink CDC docs — a MongoDB CDC source needs a replica set, which this course doesn't stand up locally. Every option and behaviour below is quoted from the MongoDB CDC connector docs, not a local run. Treat the DDL as correct-by-source, not as something I executed here.

The client stores operational data in MongoDB. Flink reads it two ways, and they are different connectors with different semantics — the asymmetry flagged back in the connector map. Picking wrong is a real design error, so this lesson is about the difference, not the setup.

Primary source Flink CDC — MongoDB CDC Connector — DDL, connector options, and the change-stream behaviour.

The plain mongodb source: a point-in-time read

The plain connector reads the collection as it is now and produces an append stream — the same shape as a bounded file read. Good for a snapshot or a lookup; it does not see later changes.

CREATE TABLE jobs_snapshot (
  _id    STRING,
  engine STRING,
  status STRING
) WITH (
  'connector' = 'mongodb',
  'uri'       = 'mongodb://localhost:27017',
  'database'  = 'dremio',
  'collection'= 'jobs'
);

The mongodb-cdc source: a live changelog

The CDC connector does a snapshot first, then streams every insert, update, and delete continuously using MongoDB's change streams (via Debezium). It is source-only — it can never be a sink. Note the two differences from the plain source: a different connector value, and a mandatory PRIMARY KEY.

CREATE TABLE jobs_cdc (
  _id    STRING,
  engine STRING,
  status STRING,
  PRIMARY KEY(_id) NOT ENFORCED
) WITH (
  'connector'  = 'mongodb-cdc',
  'hosts'      = 'localhost:27017,localhost:27018,localhost:27019',
  'username'   = 'flinkuser',
  'password'   = 'flinkpw',
  'database'   = 'dremio',
  'collection' = 'jobs'
);
Why the primary key is mandatory — the key insight Per the docs: MongoDB's change event has no "before" image on updates. So Flink can't produce a full before/after changelog — it produces an UPSERT changelog instead. An upsert stream needs a unique key to know which row to replace or delete, so you must declare _id as the primary key. You can't use any other column, because a delete event carries only _id. This isn't boilerplate — it's forced by how MongoDB emits changes.

The two sources side by side

mongodbmongodb-cdc
DirectionSource & sinkSource only
BoundednessBounded (reads now, stops)Unbounded (snapshot, then live)
Stream kindAppendUpsert changelog
Sees later changes?No — rerun to refreshYes — insert/update/delete
Primary keyNot requiredRequired (_id)
Infra neededAny MongoDBReplica set / sharded cluster

What this means for the Iceberg sink

This is where it connects to Lesson 5. An append source (plain Mongo, or files) maps to a plain append-only Iceberg write — the one you already ran. An upsert changelog source (Mongo CDC) can't just append: the Iceberg table has to apply updates and deletes, so it must be created with upsert enabled and a matching primary key:

CREATE TABLE iceberg.dremio.jobs_live (
  _id    STRING,
  engine STRING,
  status STRING,
  PRIMARY KEY(_id) NOT ENFORCED
) WITH (
  'format-version' = '2',
  'write.upsert.enabled' = 'true'
);

Iceberg format-version 2 is what supports row-level deletes. Point an append-only table at a changelog source and the updates/deletes have nowhere to go — the single most common CDC-to-Iceberg mistake, and the reason the connector map called append-vs-changelog the hinge.

Retrieval check

Why must a MongoDB CDC table declare _id as its primary key?

Correct. MongoDB emits no before-image, so Flink produces an upsert changelog, which needs a unique key — and a delete event carries only _id.

Not quite. It's the upsert changelog that needs the key — a delete event carries only _id, so no other column works.

Which pairing of source and Iceberg write is correct?

Correct. Upsert changelog (Mongo CDC) requires an Iceberg v2 table with upsert enabled and a key; plain append sources write to a plain table.

No — append sources go to append tables; a changelog source is the one that needs upsert enabled and a primary key.

Practice

  1. From memory, fill the six-row comparison table. Check against the glossary entries for append vs. changelog.
  2. The client asks for a "live view of running jobs" in Iceberg. Which Mongo connector, and what must the Iceberg table declare? Answer in two sentences.
  3. If you only need last night's job counts once a day, which connector is the simpler fit, and why?
Ask the agent: "Walk me through what a single MongoDB update becomes as it travels CDC source → Flink → Iceberg upsert table." Next lesson is the catalog decision — the seam where this whole pipeline becomes readable by Dremio.