Automate: Schedule & Monitor

Lesson 9 · SkillOpt & Agent Harnesses · ~15 minutes

Warm-up: retrieve

From Lesson 8: your skill shows +5pp on the test split but feels neutral in daily use. What's the most likely cause?

The skill document is too long for the context window
The test split doesn't represent your actual daily task distribution
The gate accepted a harmful edit by mistake

Distribution gap. Now let's make the system run itself — so your skill stays aligned with your evolving task distribution over time.

The gate can't make mistakes (strict empirical check), and context isn't the issue. It's a distribution gap: test split ≠ real usage. Automation (this lesson) keeps the system aligned by continuously harvesting real sessions.

What you'll learn

You've run manual sleep cycles. Now automate them: install a nightly cron, monitor skill evolution over weeks, and know when to intervene (reset, prune, or retrain). The goal is zero-maintenance skill improvement — the agent gets better while you sleep.

Step 1 — Schedule the nightly cycle

SkillOpt-Sleep ships a scheduler:

# Install a nightly cron entry for this project
skillopt-sleep schedule

# Verify it was installed
crontab -l | grep skillopt

By default this runs at 2:00 AM local time. The cycle harvests today's sessions, runs the consolidation pipeline, and stages a proposal. It does not auto-adopt — you still review and explicitly run skillopt-sleep adopt.

Auto-adopt mode

If you trust the gate fully, you can enable auto-adopt: skillopt-sleep schedule --auto-adopt. The gate is your safety net — if a proposal doesn't improve held-out accuracy, it's rejected regardless. Auto-adopt just means you skip the manual review step for proposals that pass the gate.

Step 2 — Monitor skill evolution

After a week of nightly runs, check the history:

# See the full history of proposals, acceptances, and rejections
skillopt-sleep status --history

# See held-out score over time
skillopt-sleep status --scores

What to look for:

Pattern	Meaning	Action
Scores climbing steadily	The system is working — skill is improving	Keep running, maybe increase `recall_k`
Scores flat for 5+ nights	Plateau — the current task set is saturated	Add new tasks to your data, or increase edit budget
All proposals rejected for 5+ nights	The optimizer can't find improvements	You may be at ceiling for this task domain
Score dropped after an adopt	Should be impossible if gate is on — investigate	Check if your test split changed, or rollback the adopt

Step 3 — Skill lifecycle management

Over weeks, your skill will grow. Here's when to intervene:

When to prune

If the skill exceeds ~2000 words, older rules may be obsolete (your workflow changed, tools updated). Review the oldest rules — do they still apply? Remove ones that don't. The gate will re-learn anything still relevant on the next cycle.

When to reset

If your task domain fundamentally changes (new language, new project, new tools), the accumulated skill may be counterproductive. Reset to a minimal seed and let the system retrain from scratch:

# Archive the current skill
cp ~/.pi/skills/best_skill.md ~/.pi/skills/best_skill_$(date +%Y%m%d).bak

# Reset to minimal seed
echo "# Coding Skill\nFollow project conventions." > ~/.pi/skills/best_skill.md

When to accumulate

If your domain is stable and scores are still climbing (even slowly), let it run. The validation gate prevents degradation — accumulation is safe as long as the gate is on.

Step 4 — Notifications (optional)

Get notified when a proposal is staged for review:

# Add to your cron entry or a wrapper script:
skillopt-sleep run && \
  skillopt-sleep status --latest | \
  mail -s "SkillOpt: proposal staged" you@example.com

Or pipe to a Slack webhook, desktop notification, or whatever fits your workflow.

Your nightly cycle has been running for 2 weeks. The last 7 nights all show "REJECTED" proposals. What's the best interpretation?

The optimizer is broken — switch to a different model
The gate threshold is too strict — lower it
The skill has saturated on the current task distribution — you've reached ceiling

Seven consecutive rejections means the optimizer can't find edits that improve held-out performance. The skill is at ceiling for these tasks. Next moves: add harder tasks, expand to new task types (Lesson 10), or increase the edit budget to allow larger changes.

The gate can't be "too strict" — it's a binary check (did the score improve? yes/no). And the optimizer isn't broken; it's proposing edits that don't help because the skill is saturated on the current task set. Next: add harder tasks or expand domains (Lesson 10).