Glossary

Learn about the words and phrases related to ingesting data with Apache Flink and writing it to Apache Iceberg for Dremio.

Apache Iceberg

Apache Iceberg is an open table format for large analytic datasets, sitting between a query engine and data files on object storage. It adds ACID commits, schema evolution, hidden partitioning, and time travel. It is what Dremio reads.

append stream

An append stream is a stream where every record is a new row — insert-only. A bounded file read produces an append stream. Contrast with a changelog stream.

at-least-once

At-least-once means every record is processed, but a failure-and-replay can process some records more than once, so duplicates are possible.

at-most-once

At-most-once means there is no replay on failure — whatever was in flight is lost. It is the weakest guarantee, and risks dropped records.

atomic commit

An atomic commit is Iceberg making a write visible all-at-once as a new snapshot, never partially. It is tied to a Flink checkpoint through flink.max-committed-checkpoint-id in the snapshot summary — the handshake that gives exactly-once on the sink.

bounded stream

A bounded stream (batch) is a finite input with a known end — a fixed set of files read once. The job reads everything, finishes, and stops.

catalog (Iceberg)

An Iceberg catalog is the system of record for where Iceberg tables live and what they look like. Flink writes to Iceberg through a catalog. Options include filesystem/Hadoop (local), REST (for example, Apache Polaris), Nessie, AWS Glue, and Hive.

CDC (Change Data Capture)

CDC is capturing row-level changes from a database as they happen, rather than re-reading the whole table. Flink CDC uses Debezium to turn a database's change log into a Flink changelog stream.

changelog stream

A changelog stream is a stream whose records carry a change kind — insert, update-before, update-after, delete. A CDC source produces a changelog stream, because it reflects edits to existing rows, not just new rows.

checkpoint

A checkpoint is a periodic, consistent snapshot of a Flink job's state, used to recover after failure. It is the mechanism underpinning exactly-once.

connector

A connector is the pluggable integration that lets a Flink table talk to an external system. It is declared with WITH ('connector' = '...') and decides what system the data moves to or from — not how bytes are encoded.

CREATE CATALOG

CREATE CATALOG is the Flink SQL statement that registers an Iceberg catalog in a session, naming its type and connection properties, before any Iceberg table can be created or written.

Debezium

Debezium is the open-source change-capture engine Flink CDC connectors build on. It reads a database's internal change log (for MongoDB, the oplog and change streams) and emits structured change events.

exactly-once

Exactly-once means every record affects the result exactly one time, even across failures. For a file-to-Iceberg pipeline, Flink's checkpointing plus Iceberg's atomic commits give exactly-once on the sink side.

FileSystem connector

The FileSystem connector reads and writes directories of files (local, HDFS, S3), as a source or a sink. It is bounded by default; set source.monitor-interval to make it continuously watch for new files. Pair it with a format. It is the job-history-logs source in this course.

Flink job

A Flink job is a dataflow graph that reads from one or more sources, applies transforms, and writes to one or more sinks. In Flink SQL, you declare tables and then run INSERT INTO sink SELECT ... FROM source.

format

A format is how records are encoded within a connector's byte stream — csv, json, avro, parquet, debezium-json. It is declared with WITH ('format' = '...') and is orthogonal to the connector: FileSystem + JSON, FileSystem + CSV, and Kafka + Avro are all valid pairings.

format-version 2 / write.upsert.enabled

format-version 2 and write.upsert.enabled are Iceberg table properties that enable row-level updates and deletes. They are required when the source is an upsert changelog (MongoDB CDC); a plain append source needs neither.

Hadoop catalog

The Hadoop catalog is the simplest Iceberg catalog: table state is tracked by files in a warehouse directory (catalog-type = 'hadoop'), with no external service. It is good for local dev and a single writer, but weak for multi-engine sharing and access control. It is used in this course's hands-on lessons.

Iceberg connector (sink)

The Iceberg connector writes a Flink table into an Apache Iceberg table through a catalog. It is the output throughout this course, because Dremio queries Iceberg natively.

metadata file / manifest

The metadata file (JSON) records the current snapshot and schema; manifest files (Avro) list the data files in each snapshot. They are what a Flink write produces alongside the Parquet data files, and what Dremio reads to plan a query.

MongoDB CDC connector

The MongoDB CDC connector is a Flink CDC source that captures inserts, updates, and deletes from MongoDB as a changelog stream, using Debezium underneath. It is source-table only — it cannot be a sink. It does a snapshot first, then streams incremental changes.

MongoDB connector

The MongoDB connector reads a MongoDB collection as a bounded source, and can also act as a lookup source or a sink. It produces an append stream — a point-in-time read, not a change stream.

REST catalog

A REST catalog (catalog-type = 'rest') is an Iceberg catalog served over HTTP — for example, Apache Polaris, Dremio's Open Catalog. It is the production target for this engagement; swapping to it changes only the CREATE CATALOG block.

sink

A sink is where data leaves a Flink job — in this course, an Apache Iceberg table. It is a table whose connector can be written to.

snapshot

A snapshot is an immutable, point-in-time view of an Iceberg table's data. Every commit creates a new snapshot; time travel reads an older one.

source

A source is where data enters a Flink job — a file directory, a MongoDB collection, a Kafka topic. It is a table whose connector can be read from.

source.monitor-interval

source.monitor-interval is the FileSystem connector option that flips a bounded read into a continuous one: Flink re-scans the directory on this interval and ingests new files, so the job never finishes. Omit it and the read is bounded.

transform

A transform is any processing between source and sink: filter, map, join, window, aggregate. It is expressed as the SELECT body in Flink SQL.

unbounded stream

An unbounded stream (streaming) is an input with no end — a directory new files keep landing in, or a change stream from a database. The job runs continuously.

upsert changelog

An upsert changelog is a changelog with no update-before image — only insert/update-after and delete, keyed on a primary key. MongoDB CDC produces this (its change events carry no before-image), which is why a MongoDB CDC table must declare a primary key.

No results.