The Flink Connector Model

Time: ~7 minutes. Tangible win: read a whole Flink pipeline as one INSERT INTO sink SELECT … FROM source, and tell a connector apart from a format. Every later lesson is a detail hung on this skeleton.

You are learning Flink to feed Apache Iceberg tables that a Dremio client will query — pulling from their MongoDB store and their job-history log files. Before any of those specifics, you need one mental model that makes the whole connector landscape read as structure instead of a list to memorise.

Primary source Flink SQL & Table Connectors — Overview — the canonical list of built-in sources, sinks, and formats, and the CREATE TABLE … WITH model.

Every Flink job is sources → transforms → sinks

A Flink job is a dataflow: data enters at one or more sources, passes through transforms (filter, join, aggregate), and leaves at one or more sinks. That is the whole shape. Your two client pipelines are both instances of it:

job-history log files  ─▶ [transform] ─▶  Iceberg table
MongoDB collection     ─▶ [transform] ─▶  Iceberg table
      (source)                                (sink)

In Flink SQL you do not write a graph by hand. You declare a table for each end — the source and the sink — then connect them with a query. The query body is the transform.

A Flink SQL "table" is not storage When you CREATE TABLE in Flink, you are not creating a table that holds data. You are declaring a named connection to an external system, plus the schema Flink should expect. The data stays in the file directory, the Mongo collection, or the Iceberg table — Flink just reads or writes through the declaration.

A source, a sink, and the query that joins them

Here is a complete (if minimal) pipeline: read job-history files, keep the failures, write them to Iceberg. Read it top to bottom — declare source, declare sink, connect.

-- 1. SOURCE: a directory of JSON job-history files
CREATE TABLE job_history (
  job_id     STRING,
  engine     STRING,
  status     STRING,
  duration_ms BIGINT
) WITH (
  'connector' = 'filesystem',
  'path'      = 'file:///data/job-history/',
  'format'    = 'json'
);

-- 2. SINK: an Iceberg table (catalog setup comes in a later lesson)
CREATE TABLE failed_jobs (
  job_id     STRING,
  engine     STRING,
  duration_ms BIGINT
) WITH (
  'connector' = 'iceberg',
  'catalog-name' = 'local',
  'warehouse'    = 'file:///data/warehouse'
);

-- 3. TRANSFORM + wire them together
INSERT INTO failed_jobs
SELECT job_id, engine, duration_ms
FROM job_history
WHERE status = 'FAILED';

That single INSERT INTO … SELECT … FROM … is the entire job. The FROM names the source, the INSERT INTO names the sink, and the SELECT/WHERE is the transform. Nothing else is required to describe a Flink pipeline at this level.

The two options doing the real work: connector and format Beginners read 'connector' and 'format' as two words for the same idea. They are not — and keeping them separate is the single most useful distinction in this whole course.

Connector vs. format

These answer two different questions:

ConnectorFormat
Question it answersWhich system does data move to/from?How are records encoded in that system's bytes?
Examplesfilesystem, mongodb, kafka, icebergjson, csv, avro, parquet
Declared with'connector' = '…''format' = '…'

They are orthogonal: FileSystem + JSON, FileSystem + CSV, and Kafka + Avro are all valid pairings. One connector can carry many formats. This is why "read the job logs" has two separate decisions baked in — filesystem (the connector) and which encoding (the format).

Some connectors don't take a format Not every connector needs a 'format'. The iceberg connector and the mongodb connector talk to systems that already have their own on-disk/on-the-wire encoding, so they manage it internally. The 'format' option mainly matters for connectors that move raw byte streams — FileSystem and Kafka above all.

Retrieval check

In Flink SQL, what does CREATE TABLE … WITH ('connector' = 'filesystem') actually create?

Correct. A Flink SQL table is a declared connection plus a schema — the data stays in the external system; Flink reads or writes through the declaration.

Not quite. Flink doesn't store the rows itself — the table is a named connection to an external system, and the data stays where it already lives.

Which pair correctly separates connector from format?

Correct. The connector decides which system (filesystem, mongodb, iceberg); the format decides how records are encoded (json, csv, parquet) — orthogonal choices.

Reversed. The connector names the system; the format names the encoding. They vary independently.

Practice

  1. In the pipeline above, point to the exact tokens that are the source, the sink, and the transform. Say each one out loud.
  2. The job logs could arrive as CSV instead of JSON. Which single option changes, and which stays the same? (Answer: the format changes, the connector stays filesystem.)
  3. In one sentence each, state what the connector and format options decide. Check yourself against the glossary.
Ask the agent: "Show me the same pipeline but reading from MongoDB instead of files — what changes in the WITH clause?" Next lesson lays out the full connector map for this engagement — FileSystem, MongoDB, MongoDB CDC, and Iceberg — and the one key tradeoff that defines each.