Sleep Cycle on Your Real Sessions

Lesson 3 · SkillOpt & Agent Harnesses · ~20 minutes · hands-on

Warm-up: retrieve before you build

From Lesson 2: the experiment injected a known-bad edit. What proved the gate was working?

The bad edit was included but scored lowest
The bad edit was rejected — never merged into the skill
The bad edit was accepted then auto-reverted next night

Right. The gate is a hard boundary: reject and discard. No "accept then undo" — that would mean live regression. The 25% negative transfer rate from SkillLens is what happens without a gate.

The gate doesn't rank — it's binary. A bad edit is rejected outright (never merged). There's no rollback mechanism because the edit never ships. That's the whole safety argument.

What you'll do

Run skillopt-sleep dry-run against your actual Claude Code or Codex transcripts. A dry-run harvests your sessions, mines recurring tasks from them, replays offline, and reports what it would propose — without changing anything. You'll then evaluate the staged proposal using the What Makes a Good Skill rubric.

harvest → mine → replay → consolidate → stage → adopt

The dry-run stops after replay — it consolidates and shows you the proposal but doesn't persist it. This is the safe exploration mode.

Prerequisites

You need at least 3–5 real agent sessions from the past week. SkillOpt-Sleep harvests transcripts from your agent's session store:

Claude Code: ~/.claude/projects/ — each project folder has session JSONs
Codex: ~/.codex/sessions/

If you don't have enough sessions yet, go use your agent normally for a day and come back. The more varied the tasks, the more interesting the mining phase.

Step 1 — Dry run

cd ~/learn-skillopt-harness
source .venv/bin/activate

# Dry run harvests, mines, replays — then reports without changing anything
skillopt-sleep dry-run

This will take 1–3 minutes depending on how many transcripts you have. Watch the output for each stage completing.

API cost note

dry-run uses your configured model (default: the optimizer model from ~/.skillopt/config.yaml) for mining and replay. Typical cost for 5 sessions is ~$0.10–$0.30. If you want to control spend, set SKILLOPT_OPTIMIZER_MODEL=gpt-4o-mini in your environment.

Step 2 — Read the harvest report

The dry-run output shows what was harvested. Look for:

Sessions found: How many transcripts were discovered
Tasks mined: The recurring patterns SkillOpt extracted — these are your task types
Replay results: For each task, how the current (empty or existing) skill performed

Note which task types it found. These are what SkillOpt thinks you do repeatedly. Do they match your actual workflow patterns?

Step 3 — Evaluate the staged proposal

The dry-run ends with a proposal — the text edit it would apply to your skill document. This is the moment to apply the rubric from What Makes a Good Skill. For each rule in the proposal, ask:

Dimension	Question	Red flag if…
Failure Mechanism	Does it name a specific failure mode and its remedy?	It's generic advice ("be careful with X")
Actionable Specificity	Is it step-level specific? Could the agent follow it mechanically?	It's vague or aspirational
High-Risk Blacklist	Does it forbid a specific harmful pattern?	Not every rule needs this — but the strongest ones do

Your judgment call: Would you adopt this proposal into your agent's system prompt? If not, why not? Write down your reasoning — you'll compare this to the gate's judgment in the full run (Lesson 4).

Step 4 — Check status and run a full cycle

If the proposal looks promising, try a full (non-dry) run:

# Full cycle — stages the proposal for review
skillopt-sleep run

# See what was staged
skillopt-sleep status

The status command shows the staged proposal and its held-out score. The gate decided: did the proposal strictly improve held-out accuracy? If yes, it's staged. If no, it's rejected — and the status tells you why.

Step 5 — Adopt or reject

If the gate accepted the proposal and you agree with it (your rubric check from Step 3), adopt it:

# Apply the staged proposal to your skill document
skillopt-sleep adopt

This writes the learned rules into your agent's skill file. From now on, every session with that agent benefits from what was learned — at zero inference-time cost (it's just a better system prompt).

The gate can reject good-looking proposals

The gate only cares about empirical improvement on held-out tasks. A proposal that reads beautifully but doesn't improve accuracy gets rejected. This is by design — format and plausibility don't predict utility. Trust the gate over your gut on this one.

Your dry-run proposes: "Always check file existence before reading." Does this pass the rubric?

Yes — it prevents a clear failure mode
Weak — no specific failure mechanism or remedy named
No — it's a blacklist rule and those are always bad

Right. "Always check file existence" is generic advice — it doesn't name when this fails (which command? what error?), doesn't give a specific remedy (check how? with what tool call?), and doesn't identify the failure pattern. A strong rule would be: "Before cat-ing a path from user input, call test -f; if missing, ask the user to confirm the path rather than erroring."

Blacklist rules aren't inherently bad — "never run rm -rf on paths containing variables without prior ls confirmation" is a strong blacklist. The issue here is specificity: "always check file existence" fails the Failure Mechanism dimension because it doesn't identify when/why this matters or what the specific remedy is.