Measuring Skill Impact

Warm-up: retrieve

What you'll learn

The gate tells you whether an edit improves held-out accuracy. But does the deployed skill actually make your daily agent better? Gates operate on synthetic benchmarks — your real workflow is noisier, more varied, and harder to measure. This lesson covers how to confirm your skill is delivering value in the real world.

The measurement problem

From SkillLens: format and plausibility don't predict utility. A skill that reads well can be neutral or harmful. The only reliable signal is empirical: does the agent produce better outputs with the skill than without? This applies after deployment too — not just during training.

Step 1 — Baseline evaluation

Before deploying a new skill, capture a baseline on your real tasks:

# Evaluate with NO skill (empty system prompt)
python scripts/evaluate.py \
  --config configs/my_env/default.yaml \
  --skill /dev/null \
  --split test

# Evaluate with your trained skill
python scripts/evaluate.py \
  --config configs/my_env/default.yaml \
  --skill outputs/my_env/skills/best_skill.md \
  --split test

Record both scores. The delta is your measured lift.

Step 2 — The noise floor

Run the same evaluation 3 times (same skill, same test split). The variance between runs is your noise floor. If your measured lift is smaller than 2× the noise floor, you can't confidently distinguish signal from random variation.

# Run 3 times, record scores
for i in 1 2 3; do
  python scripts/evaluate.py \
    --config configs/my_env/default.yaml \
    --skill outputs/my_env/skills/best_skill.md \
    --split test 2>&1 | grep "score"
done

From the Sleep README: "single-seed baseline variance is ±1–2 pts, so treat sub-~1.5pt differences as noise." If your lift is +4pp with ±1pp variance, you have real signal.

Step 3 — A/B in daily use

For real-world measurement, alternate days:

After a week, compare. This is crude but catches the big signal: is the skill helping with your actual daily tasks, not just the held-out benchmark?

Step 4 — The meta-skill boost

From SkillLens: embedding the quality rubric into the extractor prompt itself yields +1.55pp on average — the optimizer produces better proposals when it knows what "good" looks like. This is the meta-skill idea:

# Add to your skillopt config or system prompt for the optimizer:
# "A good skill rule must:
#  1. Name a specific failure mechanism and its remedy
#  2. Be step-level specific (the agent can follow it mechanically)
#  3. Forbid a specific harmful pattern when applicable
# Only propose rules that pass all three criteria."

Try this for a few nights and measure whether the gate acceptance rate improves. Higher acceptance rate with maintained or improved held-out scores = the meta-skill is working.