The Flink Connector Model
You are learning Flink to feed Apache Iceberg tables that a Dremio client will query — pulling from their MongoDB store and their job-history log files. Before any of those specifics, you need one mental model that makes the whole connector landscape read as structure instead of a list to memorise.
CREATE TABLE … WITH model.
Every Flink job is sources → transforms → sinks
A Flink job is a dataflow: data enters at one or more sources, passes through transforms (filter, join, aggregate), and leaves at one or more sinks. That is the whole shape. Your two client pipelines are both instances of it:
job-history log files ─▶ [transform] ─▶ Iceberg table
MongoDB collection ─▶ [transform] ─▶ Iceberg table
(source) (sink)
In Flink SQL you do not write a graph by hand. You declare a table for each end — the source and the sink — then connect them with a query. The query body is the transform.
CREATE TABLE in Flink, you are not creating a table that holds data. You are declaring a named connection to an external system, plus the schema Flink should expect. The data stays in the file directory, the Mongo collection, or the Iceberg table — Flink just reads or writes through the declaration.
A source, a sink, and the query that joins them
Here is a complete (if minimal) pipeline: read job-history files, keep the failures, write them to Iceberg. Read it top to bottom — declare source, declare sink, connect.
-- 1. SOURCE: a directory of JSON job-history files
CREATE TABLE job_history (
job_id STRING,
engine STRING,
status STRING,
duration_ms BIGINT
) WITH (
'connector' = 'filesystem',
'path' = 'file:///data/job-history/',
'format' = 'json'
);
-- 2. SINK: an Iceberg table (catalog setup comes in a later lesson)
CREATE TABLE failed_jobs (
job_id STRING,
engine STRING,
duration_ms BIGINT
) WITH (
'connector' = 'iceberg',
'catalog-name' = 'local',
'warehouse' = 'file:///data/warehouse'
);
-- 3. TRANSFORM + wire them together
INSERT INTO failed_jobs
SELECT job_id, engine, duration_ms
FROM job_history
WHERE status = 'FAILED';
That single INSERT INTO … SELECT … FROM … is the entire job. The FROM names the source, the INSERT INTO names the sink, and the SELECT/WHERE is the transform. Nothing else is required to describe a Flink pipeline at this level.
'connector' and 'format' as two words for the same idea. They are not — and keeping them separate is the single most useful distinction in this whole course.
Connector vs. format
These answer two different questions:
| Connector | Format | |
|---|---|---|
| Question it answers | Which system does data move to/from? | How are records encoded in that system's bytes? |
| Examples | filesystem, mongodb, kafka, iceberg | json, csv, avro, parquet |
| Declared with | 'connector' = '…' | 'format' = '…' |
They are orthogonal: FileSystem + JSON, FileSystem + CSV, and Kafka + Avro are all valid pairings. One connector can carry many formats. This is why "read the job logs" has two separate decisions baked in — filesystem (the connector) and which encoding (the format).
'format'. The iceberg connector and the mongodb connector talk to systems that already have their own on-disk/on-the-wire encoding, so they manage it internally. The 'format' option mainly matters for connectors that move raw byte streams — FileSystem and Kafka above all.
Retrieval check
In Flink SQL, what does CREATE TABLE … WITH ('connector' = 'filesystem') actually create?
Correct. A Flink SQL table is a declared connection plus a schema — the data stays in the external system; Flink reads or writes through the declaration.
Not quite. Flink doesn't store the rows itself — the table is a named connection to an external system, and the data stays where it already lives.
Which pair correctly separates connector from format?
Correct. The connector decides which system (filesystem, mongodb, iceberg); the format decides how records are encoded (json, csv, parquet) — orthogonal choices.
Reversed. The connector names the system; the format names the encoding. They vary independently.
Practice
- In the pipeline above, point to the exact tokens that are the source, the sink, and the transform. Say each one out loud.
- The job logs could arrive as CSV instead of JSON. Which single option changes, and which stays the same? (Answer: the
formatchanges, theconnectorstaysfilesystem.) - In one sentence each, state what the
connectorandformatoptions decide. Check yourself against the glossary.
WITH clause?" Next lesson lays out the full connector map for this engagement — FileSystem, MongoDB, MongoDB CDC, and Iceberg — and the one key tradeoff that defines each.