Connecting a Query Engine (Spark)

Time: ~9 minutes. Tangible win: create a namespace, create a table, insert rows, and query them entirely through Spark SQL, with Spark talking Iceberg REST underneath — no Polaris-specific code required.

The previous lesson proved the catalog works in isolation. This lesson proves the actual point of Lesson 1: an engine you didn't write any Polaris-specific code for can read and write the same catalog using a standard Iceberg Spark connector.

What the config actually says

• Spark needs the iceberg-spark-runtime package for your Spark/Scala version (this run: iceberg-spark-runtime-3.5_2.12:1.5.2 against Spark 3.5.2) plus the Iceberg SQL extensions enabled via spark.sql.extensions.
• The catalog is registered as spark.sql.catalog.<name>, set to org.apache.iceberg.spark.SparkCatalog with catalog-impl=org.apache.iceberg.rest.RESTCatalog — this is what tells Spark "talk Iceberg REST to this URI" instead of using a Hive Metastore or Spark's built-in catalog.
• Credentials travel as spark.sql.catalog.<name>.credential in clientId:clientSecret form, with token-refresh-enabled=true so Spark re-authenticates instead of failing when the token expires mid-session.
• Once configured, <name>.<namespace>.<table> in SQL resolves through Polaris exactly like coursecatalog/course_db did via curl — same entities, different client.

Exercise

import pyspark
from pyspark.sql import SparkSession

conf = (
    pyspark.SparkConf()
    .setAppName("polaris_demo")
    .set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .set("spark.sql.catalog.polaris", "org.apache.iceberg.spark.SparkCatalog")
    .set("spark.sql.catalog.polaris.warehouse", "coursecatalog")
    .set("spark.sql.catalog.polaris.catalog-impl", "org.apache.iceberg.rest.RESTCatalog")
    .set("spark.sql.catalog.polaris.uri", "http://polaris:8181/api/catalog")
    .set("spark.sql.catalog.polaris.credential", "<clientId>:<clientSecret>")
    .set("spark.sql.catalog.polaris.scope", "PRINCIPAL_ROLE:ALL")
    .set("spark.sql.catalog.polaris.token-refresh-enabled", "true")
)
spark = SparkSession.builder.config(conf=conf).getOrCreate()

spark.sql("CREATE NAMESPACE IF NOT EXISTS polaris.course_db")
spark.sql("CREATE TABLE IF NOT EXISTS polaris.course_db.people (name STRING, role STRING) USING iceberg")
spark.sql("INSERT INTO polaris.course_db.people VALUES ('Bilbo', 'engineer'), ('Gandalf', 'staff')")
spark.sql("SELECT * FROM polaris.course_db.people ORDER BY name").show()

Verified output:

SELECT RESULT:
+-------+--------+
|   name|    role|
+-------+--------+
|  Bilbo|engineer|
|Gandalf|   staff|
+-------+--------+

SNAPSHOT COUNT:
+--------------+
|snapshot_count|
+--------------+
|             1|
+--------------+

snapshot_count is 1, not 2, because Iceberg's CREATE TABLE doesn't produce a snapshot by itself (no data yet) — the single multi-row INSERT is the first and only commit.

Confirming the round trip: the same table is now visible from the previous lesson's raw REST API, even though it was only ever created from Spark —

curl -s "http://localhost:8181/api/catalog/v1/coursecatalog/namespaces/course_db/tables" -H "Authorization: Bearer $TOKEN"
# {"identifiers":[{"namespace":["course_db"],"name":"people"}],"next-page-token":null}

That's Lesson 1's thesis made concrete: Spark never talked to curl, curl never talked to Spark, and they still agree perfectly on what tables exist — because both went through the same Polaris REST catalog.

Retrieval check

Practice

1. Run this against your own Polaris catalog with a table of your own design (different columns, different rows).
2. Confirm the table is visible from the raw REST API too, even though you only created it from Spark.
3. If running Spark and Polaris as separate containers, confirm you're using the Docker service name (e.g. polaris) in the URI, not localhost — inside the Spark container, localhost means the Spark container itself.