Flink Connectors Course
A hands-on course on Apache Flink's connector surface — the input sources data arrives from and the output sinks it lands in — grounded in a real Dremio engagement: pulling from MongoDB and job-history log files, writing to Apache Iceberg for Dremio to query. Flink SQL first. Lessons 3–5 were run for real against a local Flink 1.18 + Iceberg Docker stack; Lessons 6–9 are close-reading and design, grounded in the official Flink/Iceberg docs. Work through them in order — each builds on the previous one's model.
- 01 The Flink Connector Model Sources → transforms → sinks, the Flink SQL table as a declaration, and connector vs. format. ~7 min
- 02 The Connector Map FileSystem, MongoDB, MongoDB CDC, and Iceberg — the defining tradeoff of each, and the append-vs-changelog hinge. ~8 min
- 03 A Local Flink SQL Environment + Reading Files Stand up Flink SQL in Docker and read a directory of job-history files into a table (bounded). ~10 min
- 04 The FileSystem Source: Bounded vs. Continuous Read once vs. watch a directory for new files, and what changes when the job never ends. ~8 min
- 05 The Iceberg Sink Write a Flink table into Iceberg through a catalog, and see the snapshot, manifests, and Parquet files it produced. ~10 min
- 06 MongoDB Source vs. MongoDB CDC A bounded append read against a live changelog stream — and why CDC forces a primary key on your Iceberg table. ~9 min
- 07 Catalogs and the Dremio Seam Filesystem, Polaris, Nessie, Glue — and why the catalog choice is what lets Dremio read the result. ~8 min
- 08 Delivery Guarantees What exactly-once means for a file-to-Iceberg pipeline, and the two mechanisms that deliver it. ~8 min
- 09 Design Recommendation The client's two pipelines decided and defended: source mode, guarantee, catalog — each traced to a lesson. ~9 min
Course: Flink Connectors — exploring Apache Flink's input sources and output sinks for a Dremio engagement (MongoDB + job-history logs → Iceberg).
9 lessons · ~77 minutes · Interactive checks in each lesson · Flink SQL first, Java later · Lessons 3–5 verified on a real Flink 1.18 + Iceberg run