Persistence and Production Topology

Time: ~8 minutes. Tangible win: explain what a Polaris persistence backend actually stores, why the quickstart's default isn't safe to run in production, and what changes operationally when Polaris serves many engines and teams instead of one laptop.

Everything in Lessons 3–6 ran against one Polaris process on one machine. That's the right way to learn the API; it's the wrong way to run it. This lesson is the "next capability" that turns this course into an actual deployment decision instead of a demo.

Primary source Polaris server configuration reference (1.7.0) — source for metaStoreManager.type and other persistence-backend settings.

Two separate failure domains

Polaris's own state — every catalog, namespace pointer, principal, role, and grant from Lessons 2–6 — lives in a metastore/persistence backend that is itself pluggable (metaStoreManager.type in the server config), separate from wherever the actual Iceberg table data (Parquet files, manifests) lives.
A default or in-memory/embedded persistence backend, useful for a quickstart, does not survive a container restart or scale past one process — every entity created in this course would be gone. Production deployments swap this for a real relational store (e.g. the EclipseLink-backed JDBC option) instead.
Because catalog metadata and table data are separate concerns, scaling Polaris horizontally (multiple Polaris instances behind a load balancer, all sharing one persistence backend) is a realistic topology — any instance can serve any request, because none of them hold state locally.
Multi-realm support (Lesson 3) is what lets one production Polaris deployment serve multiple isolated tenants (e.g., separate business units or environments) without standing up a separate fleet per tenant.

Exercise: inspect what your quickstart is actually persisting to

docker inspect polaris --format '{{json .Mounts}}' | python3 -m json.tool
docker exec polaris grep -A2 -i "metaStoreManager" polaris-server.yml

Verified output from a real local run:

[{"Type":"bind","Source":".../icebergdata","Destination":"/data","Mode":"rw","RW":true,...}]

metaStoreManager:
  type: in-memory
  # type: eclipse-link # uncomment to use eclipse-link as metastore

Only half of this quickstart is durable Only /data (table data) is a mounted, durable volume here. There is no volume backing catalog metadata at all — it's in-memory, with eclipse-link (a real JDBC-backed option) sitting right there, commented out.

Then answer: if you ran docker compose down -v right now (removing volumes) or just restarted this container, which of Lessons 2–6's objects would survive, and which would you have to recreate?

Evidence of competence

Paste your own inspection output and your answer. A correct answer recognizes that with metaStoreManager.type: in-memory, every catalog/principal/role/grant from Lessons 2–6 lives only in the running process's heap — a plain container restart loses all of it and forces a fresh bootstrap (Lesson 3), even though the actual table files under /data would survive because that's a separate, mounted volume.

Retrieval check

Your quickstart's metaStoreManager.type is in-memory and you restart the container. What happens to the course_engineer principal from Lesson 6?

Correct. Catalog metadata (principals, roles, grants) and table data are separate failure domains — an in-memory metastore loses everything on restart regardless of whether the table-data volume is mounted.

Not quite. With metaStoreManager.type: in-memory, nothing about principals or grants is written to disk — a restart forces a fresh bootstrap.

Why can multiple Polaris instances safely sit behind one load balancer, each serving any request?

Correct. Horizontal scaling works specifically because catalog state lives in a shared external persistence backend, not because instances talk to each other or diverge independently.

No. Independent per-instance state (or direct instance-to-instance sync) is not how this works — a shared persistence backend is what makes any instance interchangeable.

Practice

Run the inspection commands against your own quickstart and note whether table data, catalog metadata, both, or neither are on durable storage.
Restart your container and confirm which Lesson 2–6 objects actually survived.
Read the EclipseLink JDBC option in the configuration reference and note what you'd need (a running database) to switch to it.

Common failure modes Treating a working local quickstart as evidence the deployment is production-ready — the API surface is identical either way; only the persistence backend tells you whether state survives a restart. Also: standing up multiple Polaris instances without confirming they share the same persistence backend — that's not horizontal scaling, it's multiple catalogs silently diverging.

New terms — persistence backend, metaStoreManager, realm — are in the glossary.

Ask the agent: "What would it actually take to switch this quickstart's metastore from in-memory to eclipse-link locally, and confirm state survives a restart?" That's the natural next step once this course's foundations are solid — along with pointing a second engine (Trino, DuckDB, or Flink) at the same catalog from Lesson 5 with a scoped-down principal from Lesson 6, and confirming both engines see the same table state without either one knowing the other exists.