Array and the Validity Bitmap

Time: ~8 minutes. Tangible win: construct an Arrow Array, explain how it represents nulls, and demonstrate that slicing is zero-copy.

Array is the actual buffer of column data that a Field/Schema only describes abstractly. This is where "columnar in-memory format" stops being a slogan and becomes bytes you can inspect.

pa.array(python_list) infers an Arrow type from the input and builds an immutable columnar buffer.
Nulls are not encoded with a sentinel value inside the data buffer. Arrow arrays carry a separate validity bitmap — one bit per element — alongside the data buffer. This is why null-heavy columns don't need "impossible value" hacks like -1 or NaN-as-null.
Arrays are immutable, and slicing does not copy the underlying data buffer — a slice is a view with a different offset/length into the same buffer. This is the array-level version of the zero-copy idea from Lesson 1.
A Table's columns aren't plain Arrays — they're ChunkedArrays (Lesson 5). Keep those two names distinct.

Primary source Arrow Columnar Format specification — the validity bitmap layout referenced here.

Exercise

import pyarrow as pa

arr = pa.array([1, 2, None, 4, 5])
print("Array:", arr)
print("null_count:", arr.null_count)
print("length:", len(arr))

sliced = arr.slice(3, 2)
print("Sliced:", sliced)

parent_data_buffer = arr.buffers()[1]
sliced_data_buffer = sliced.buffers()[1]
print("Same underlying data buffer (zero-copy slice):",
      parent_data_buffer.address == sliced_data_buffer.address)
print("Sliced array offset into parent:", sliced.offset)

Verified output (pyarrow 25.0.1):

Array: [
  1,
  2,
  null,
  4,
  5
]
null_count: 1
length: 5
Sliced: [
  4,
  5
]
Same underlying data buffer (zero-copy slice): True
Sliced array offset into parent: 3

A slice keeps the whole parent alive Don't assume a slice is an independent copy you can freely hand off expecting the parent to become garbage-collectable — the slice keeps the entire parent buffer alive, not just the sliced portion. Slicing a tiny window out of a huge array does not free the rest of the memory.

Retrieval check

How does an Arrow Array represent a null value in an int64 column?

Correct. The validity bitmap tracks null/not-null per element separately from the data buffer — no "impossible value" hacks needed.

Not quite. Arrow uses a dedicated validity bitmap rather than sentinel values, so the data buffer itself never needs a fake "null" number.

You slice a 10,000-element array down to 5 elements and drop your reference to the original array. Is the other 9,995 elements' memory now freed?

Correct. Slicing is zero-copy, which is fast — but it means the slice pins the whole original buffer in memory, not just its own window.

No. A slice is zero-copy: it's a view (offset + length) into the parent's buffer, so the full parent buffer stays alive as long as the slice does.

Practice

Build an array of at least 10 elements with 3+ nulls in different positions.
Slice out a range that includes at least one null.
Report: the null_count of the slice, and whether the slice's data buffer address still matches the parent's.

New terms — Array, validity bitmap, zero-copy — are in the glossary.

Ask the agent: "What does is_null() return for the slice in the exercise, and how does that relate to the validity bitmap?" Next lesson combines multiple typed Arrays under one Schema into the actual unit of interchange: RecordBatch.