The FileSystem Source: Bounded vs. Continuous
Lesson 3's read finished and stopped — the directory was bounded. But the client's job-history logs don't stop; new files land as engines run jobs. This lesson makes the same FileSystem source continuous, so Flink watches the directory and ingests new files as they appear. One option name does it.
source.monitor-interval option and streaming-source behaviour.
The one option: source.monitor-interval
Add source.monitor-interval to the WITH clause and the connector flips from "read once" to "scan the directory on this interval, pick up new files forever":
SET 'execution.runtime-mode' = 'streaming';
CREATE TABLE job_history_stream (
job_id STRING,
engine STRING,
status STRING,
duration_ms BIGINT
) WITH (
'connector' = 'filesystem',
'path' = 'file:///data/job-history/',
'format' = 'json',
'source.monitor-interval' = '2s'
);
Everything else is identical to Lesson 3. The connector is still filesystem, the format still json, the path still the same directory. The only change in meaning is boundedness — exactly the axis from the connector map.
The observable difference: the job never finishes
This is the whole point, and you can see it in the cluster's job list. A bounded read runs and reaches FINISHED. A continuous read stays RUNNING, waiting for the next file.
------------------ Running/Restarting Jobs -------------------
14.10.2026 05:08:15 : d8cc9896a7755740f1e424ad2afc340e :
insert-into_default_catalog.default_database.sink_print (RUNNING)
No scheduled jobs.
RUNNING, not FINISHED. Drop another JSON file into the directory and within the 2-second interval its rows flow through. The job only stops when you cancel it.
source.monitor-interval: scan once, emit, finish — bounded. With it: scan repeatedly, emit new files, never finish — unbounded. The source table's schema, format, and path don't change; only whether the job ends.
What continuous mode costs you
Watching forever isn't free, and these tradeoffs are the substance of a design conversation:
| Bounded (batch) | Continuous (streaming) | |
|---|---|---|
| Job lifecycle | Reads all files, finishes | Runs until cancelled |
| Latency to new data | Re-run the job manually | Picked up within the interval |
| Resource use | Only while running | A slot held continuously |
| Fits the client's… | Historical log backfill | Live job-history ingestion |
Mapping it to the client
The client has both needs, and they are two different jobs:
- Backfill the pile of historical job-history logs → bounded, run once (Lesson 3's setup).
- Keep up with logs as engines produce them → continuous, left running (this lesson's setup).
You can even do both into the same Iceberg table: backfill first, then start the continuous job. That's a hybrid source pattern — worth knowing the name, not drilling here.
Retrieval check
What single change makes the FileSystem source continuous instead of read-once?
Correct. source.monitor-interval tells the connector to re-scan the directory on that interval and ingest new files, so the job never finishes.
Not quite. The format and path don't control boundedness — source.monitor-interval does.
How do you observe that a source is unbounded on a running cluster?
Correct. A bounded read reaches FINISHED; a continuous read holds at RUNNING, waiting for the next file — exactly what the verified job list showed.
No — the tell is the job state. Unbounded jobs stay RUNNING rather than finishing.
Practice
- Run the continuous version with a
printsink. In another terminal,cpa new JSON file into the directory and watch the rows appear within two seconds. - Run
flink listand confirm the job isRUNNING. Then cancel it withflink cancel <job-id>. - For each of the client's two needs — backfill and live ingestion — state which mode and why, in one sentence each.