Install & the Deterministic Sleep Experiment

Warm-up: retrieve before you build

Answer these from memory — no peeking at Lesson 1. Say your answers out loud (or to me) before clicking.

What you'll do

Install SkillOpt and run its deterministic validation experiment — the acceptance test the authors ship for the whole sleep-cycle idea. It uses a MockBackend: no API key, no cost, fully reproducible. According to the experiment source, one run proves three things:

1. Monotonic lift — over N sleep "nights", the held-out score climbs from a baseline (empty skill) toward 1.0 as the gate accepts the rules the tasks require.
2. Gate safety — an injected harmful edit is rejected, proving a bad nightly proposal can never be adopted.
3. Plumbing — harvest → mine → replay → consolidate → stage → adopt all run, and the adopted artifact retains the lift when re-scored.

Step 1 — Install

Create a clean virtualenv in the workspace (SkillOpt needs Python ≥ 3.10):

cd ~/learn-skillopt-harness
python3 -m venv .venv
source .venv/bin/activate
pip install skillopt
skillopt-sleep --help

If skillopt-sleep --help prints a usage banner, you're installed.

Step 2 — Run the experiment

There are two built-in personas — a researcher (wants arXiv ids in <answer> tags) and a programmer (wants imperative-mood commit subjects). Given your mission, run the programmer:

python -m skillopt_sleep.experiments.run_experiment --persona programmer --assert-improves

The --assert-improves flag makes it exit non-zero if the lift doesn't materialize — so a clean exit is the proof.

Step 3 — Read the trace like an optimizer

Don't just check the exit code. In the output, find and note down:

• Night 0 baseline — the held-out score with an empty skill. This is your "no-skill accuracy".
• Per-night trace — each night shows a holdout_score, a gate action, whether the proposal was accepted, and the actual text of applied edits. Read the edits: they are the rules being learned.
• Judge the edits like a researcher — score each learned rule against the three properties that actually predict skill utility (reference: What Makes a Good Skill): does it encode a failure mechanism with a remedy? Is it step-level specific? Does it forbid a specific harmful pattern? The mock persona's rules are simple, but the habit is the point — you'll use this rubric on every real staged proposal from Lesson 3 on.
• Rejected edits — nights where n_rejected > 0. The gate at work.
• The harmful-edit probe — confirm the known-bad rule was REJECTED. This is proof #2. It matters because ungated skill extraction degrades the agent in 25% of measured cases — the gate is your only defense.

Step 4 — One experiment of your own

Change one variable and predict the outcome before running:

python -m skillopt_sleep.experiments.run_experiment --persona programmer --nights 1

Prediction first: with only one night, will the held-out score reach the same level as the multi-night run? Why or why not? (Hint: think about the edit budget per night — the textual learning rate.) Run it, then check.