Delivery Guarantees

Time: ~8 minutes. Tangible win: explain what "exactly-once" means for a file-to-Iceberg pipeline, and point to the two mechanisms that deliver it — enough to answer "will we drop or duplicate job records?" with confidence.

A client about to depend on a pipeline will ask the correctness question: if a job crashes mid-run, do we lose records, or double-count them? This lesson answers it for the file-to-Iceberg path you built. The good news: Flink and Iceberg together give exactly-once on the sink, and you already saw the fingerprint of the mechanism in Lesson 5's snapshot.

Primary source Flink Checkpointing — the consistent-snapshot mechanism. See also FileSystem fault-tolerance guarantees.

Three levels of guarantee

GuaranteeMeaning on failure + replayRisk
At-most-onceNo replay; whatever was in flight is lostDropped records
At-least-onceReplays, but may re-emit already-processed recordsDuplicate records
Exactly-onceReplays, and each record affects the result onceNeither — the goal

"Exactly-once" doesn't mean each record is processed once — on recovery, work is redone. It means each record affects the result once. That distinction is what trips people up.

The two mechanisms that make it work

For file-to-Iceberg, exactly-once rests on two cooperating parts:

You already saw the fingerprint Lesson 5's real snapshot summary included flink.max-committed-checkpoint-id. That field is the sink recording which Flink checkpoint this Iceberg snapshot corresponds to — the exact handshake between the two mechanisms. The commit is per-checkpoint, which is why a partial failure can't leave half a batch of job records visible.

Why the combination matters

Checkpoints alone give at-least-once: on replay, a sink that already wrote some rows would write them again. Atomic commits alone don't help if the source can't rewind. Put together — rewindable source position plus all-or-nothing commit per checkpoint — and a crash leaves the Iceberg table at the last committed snapshot, with the job resuming exactly where that snapshot ended. No gap, no overlap.

Exactly-once is a sink property here, not a free global promise The guarantee covers what lands in Iceberg. It assumes checkpointing is enabled and the source position is replayable (the FileSystem source tracks which files it consumed — see Lesson 4). It does not cover side effects you bolt on outside Flink, like an external alert fired mid-processing. For the client's "files → Iceberg for Dremio" path, though, the end-to-end story is exactly-once.

The CDC case, briefly

For the MongoDB CDC path from Lesson 6, the same checkpoint machinery applies, and the connector docs note the CDC source supports exactly-once processing. Because that path is an upsert changelog keyed on _id, a replayed update is idempotent anyway — applying the same upsert twice lands the same row — which reinforces the guarantee rather than fighting it.

Retrieval check

What does "exactly-once" actually promise for the Iceberg result?

Correct. On recovery, work is redone — but each record affects the committed Iceberg result once. Processing-once and result-once are different claims.

Not quite. Work is reprocessed on replay; the promise is that the committed result reflects each record exactly once.

Which two mechanisms together give file-to-Iceberg exactly-once?

Correct. Checkpoints give a rewindable source position; atomic commits make each checkpoint's data one all-or-nothing snapshot — together, exactly-once.

No — it's checkpoints (rewindable position) plus Iceberg's atomic per-checkpoint commit. Memory, format, and catalog don't provide the guarantee.

Practice

  1. Open your Lesson 5 metadata.json again and find flink.max-committed-checkpoint-id in the snapshot summary. Say what it ties together.
  2. In one sentence each, define at-most-once, at-least-once, and exactly-once using the client's job records as the example.
  3. A teammate says "exactly-once means Flink never does any work twice." State the correction.
Ask the agent: "What breaks the exactly-once guarantee if I turn checkpointing off?" The final lesson pulls everything together into a design recommendation for the client's two real pipelines.