The FileSystem Source: Bounded vs. Continuous

Time: ~8 minutes. Tangible win: turn the same source from a read-once batch into a never-ending stream with one option, and know which the client's job-history pipeline needs. The job-state difference below is from a real run.

Lesson 3's read finished and stopped — the directory was bounded. But the client's job-history logs don't stop; new files land as engines run jobs. This lesson makes the same FileSystem source continuous, so Flink watches the directory and ingests new files as they appear. One option name does it.

Primary source Flink FileSystem Connector — the source.monitor-interval option and streaming-source behaviour.

The one option: source.monitor-interval

Add source.monitor-interval to the WITH clause and the connector flips from "read once" to "scan the directory on this interval, pick up new files forever":

SET 'execution.runtime-mode' = 'streaming';

CREATE TABLE job_history_stream (
  job_id      STRING,
  engine      STRING,
  status      STRING,
  duration_ms BIGINT
) WITH (
  'connector' = 'filesystem',
  'path'      = 'file:///data/job-history/',
  'format'    = 'json',
  'source.monitor-interval' = '2s'
);

Everything else is identical to Lesson 3. The connector is still filesystem, the format still json, the path still the same directory. The only change in meaning is boundedness — exactly the axis from the connector map.

The observable difference: the job never finishes

This is the whole point, and you can see it in the cluster's job list. A bounded read runs and reaches FINISHED. A continuous read stays RUNNING, waiting for the next file.

Verified output — the continuous job's state, listed from a real cluster a few seconds after submission:

------------------ Running/Restarting Jobs -------------------
14.10.2026 05:08:15 : d8cc9896a7755740f1e424ad2afc340e :
    insert-into_default_catalog.default_database.sink_print (RUNNING)
No scheduled jobs.

RUNNING, not FINISHED. Drop another JSON file into the directory and within the 2-second interval its rows flow through. The job only stops when you cancel it.

Bounded is a special case of unbounded Think of it as one dial, not two modes. Without source.monitor-interval: scan once, emit, finish — bounded. With it: scan repeatedly, emit new files, never finish — unbounded. The source table's schema, format, and path don't change; only whether the job ends.

What continuous mode costs you

Watching forever isn't free, and these tradeoffs are the substance of a design conversation:

Bounded (batch)Continuous (streaming)
Job lifecycleReads all files, finishesRuns until cancelled
Latency to new dataRe-run the job manuallyPicked up within the interval
Resource useOnly while runningA slot held continuously
Fits the client's…Historical log backfillLive job-history ingestion
Each file is read once — and files are immutable here Continuous mode tracks which files it has already ingested and reads each new file exactly once. It does not re-read a file you edit in place. The FileSystem source assumes files are written once and never modified — which matches append-only log files, and does not match a file an engine rewrites. If the client's logs are rewritten rather than rolled, that's a design flag.

Mapping it to the client

The client has both needs, and they are two different jobs:

You can even do both into the same Iceberg table: backfill first, then start the continuous job. That's a hybrid source pattern — worth knowing the name, not drilling here.

Retrieval check

What single change makes the FileSystem source continuous instead of read-once?

Correct. source.monitor-interval tells the connector to re-scan the directory on that interval and ingest new files, so the job never finishes.

Not quite. The format and path don't control boundedness — source.monitor-interval does.

How do you observe that a source is unbounded on a running cluster?

Correct. A bounded read reaches FINISHED; a continuous read holds at RUNNING, waiting for the next file — exactly what the verified job list showed.

No — the tell is the job state. Unbounded jobs stay RUNNING rather than finishing.

Practice

  1. Run the continuous version with a print sink. In another terminal, cp a new JSON file into the directory and watch the rows appear within two seconds.
  2. Run flink list and confirm the job is RUNNING. Then cancel it with flink cancel <job-id>.
  3. For each of the client's two needs — backfill and live ingestion — state which mode and why, in one sentence each.
Ask the agent: "If I edit an already-ingested file in place, will continuous mode re-read it?" (No — it assumes write-once files. Rewrites are a design flag.) Next lesson adds the other half of the pipeline: the Iceberg sink, where this stream actually lands.