Delivery Guarantees
A client about to depend on a pipeline will ask the correctness question: if a job crashes mid-run, do we lose records, or double-count them? This lesson answers it for the file-to-Iceberg path you built. The good news: Flink and Iceberg together give exactly-once on the sink, and you already saw the fingerprint of the mechanism in Lesson 5's snapshot.
Three levels of guarantee
| Guarantee | Meaning on failure + replay | Risk |
|---|---|---|
| At-most-once | No replay; whatever was in flight is lost | Dropped records |
| At-least-once | Replays, but may re-emit already-processed records | Duplicate records |
| Exactly-once | Replays, and each record affects the result once | Neither — the goal |
"Exactly-once" doesn't mean each record is processed once — on recovery, work is redone. It means each record affects the result once. That distinction is what trips people up.
The two mechanisms that make it work
For file-to-Iceberg, exactly-once rests on two cooperating parts:
- Flink checkpoints — periodically, Flink takes a consistent snapshot of the whole job's state, including how far into each source file it has read. On failure, it rewinds every operator to the last checkpoint and resumes. Nothing before the checkpoint is reprocessed into the result.
- Iceberg atomic commits — the Iceberg sink doesn't make rows visible as it writes them. It stages data files, then commits them to a new snapshot atomically, tied to a Flink checkpoint. Either the whole checkpoint's data becomes one new snapshot, or none of it does.
flink.max-committed-checkpoint-id. That field is the sink recording which Flink checkpoint this Iceberg snapshot corresponds to — the exact handshake between the two mechanisms. The commit is per-checkpoint, which is why a partial failure can't leave half a batch of job records visible.
Why the combination matters
Checkpoints alone give at-least-once: on replay, a sink that already wrote some rows would write them again. Atomic commits alone don't help if the source can't rewind. Put together — rewindable source position plus all-or-nothing commit per checkpoint — and a crash leaves the Iceberg table at the last committed snapshot, with the job resuming exactly where that snapshot ended. No gap, no overlap.
The CDC case, briefly
For the MongoDB CDC path from Lesson 6, the same checkpoint machinery applies, and the connector docs note the CDC source supports exactly-once processing. Because that path is an upsert changelog keyed on _id, a replayed update is idempotent anyway — applying the same upsert twice lands the same row — which reinforces the guarantee rather than fighting it.
Retrieval check
What does "exactly-once" actually promise for the Iceberg result?
Correct. On recovery, work is redone — but each record affects the committed Iceberg result once. Processing-once and result-once are different claims.
Not quite. Work is reprocessed on replay; the promise is that the committed result reflects each record exactly once.
Which two mechanisms together give file-to-Iceberg exactly-once?
Correct. Checkpoints give a rewindable source position; atomic commits make each checkpoint's data one all-or-nothing snapshot — together, exactly-once.
No — it's checkpoints (rewindable position) plus Iceberg's atomic per-checkpoint commit. Memory, format, and catalog don't provide the guarantee.
Practice
- Open your Lesson 5
metadata.jsonagain and findflink.max-committed-checkpoint-idin the snapshot summary. Say what it ties together. - In one sentence each, define at-most-once, at-least-once, and exactly-once using the client's job records as the example.
- A teammate says "exactly-once means Flink never does any work twice." State the correction.