Reference: What Makes a Good Skill

Compressed from From Raw Experience to Skill Consumption (SkillLens, MSR 2026) — the systematic study behind SkillOpt, covering 5 domains × 6 targets × 5 extractors. Use this page whenever you write a seed skill, read a staged SkillOpt-Sleep proposal, or judge a best_skill.md.

The validated rubric — three properties that predict utility

Of seven candidate quality dimensions discovered by contrastive analysis of high-gap skill pairs, only three reliably aligned with downstream performance (Table 13 of the paper):

1. Failure Mechanism Encoding better-rate 65.5%

The skill identifies why agents fail, not just that they fail — and pairs each mechanism with an executable remedy. Example: "formula strings are not evaluated in headless execution → always precompute static values in Python and write the scalar."

2. Actionable Specificity better-rate 66.0%

Step-level procedures that reference concrete domain objects and tools ("search for anchor headers to determine indices; never hardcode cell references") rather than process-level advice ("edit minimally and safely").

3. High-Risk Action Blacklist better-rate 64.6%

Explicitly forbids specific harmful action patterns ("never delete rows top-to-bottom — iterate bottom-to-top to avoid index shifting"), not generic caution ("be careful with destructive edits").

Concrete remedies beat generic advice

In the paper's flagship contrastive case (SpreadsheetBench, Δ gap = 10.3 pts), the winning skill named three domain-specific failure mechanisms with fixes; the losing skill gave eight reasonable-sounding process rules ("resolve the contract before coding"). Reasonable ≠ useful.

Higher-utility skill sounds like

"Writing formula strings does not trigger calculation engines in headless environments. Always calculate the final static value in Python and write the scalar result. When deleting rows, iterate bottom-to-top to avoid index-shifting errors."

Lower-utility skill sounds like

"Resolve the contract before coding. Choose the simplest method that matches the runtime. Edit minimally and safely, keeping changes inside the intended scope. Validate the saved result."

Things that do NOT predict skill quality

Non-signal	Evidence
Format (list vs. checklist vs. prose)	Friedman test non-significant on every target (p > 0.34); extractor swap significant on 5/6 (p < 0.01). What a skill says matters, not how it looks.
How good the text "reads"	An unguided GPT-5.4 judge picked the better-performing skill 46.4% of the time — coin-flip. On the largest-gap pairs (δ ≥ 5 pts) it picked right only 15.8%: the skill that reads better often performs worse.
Extractor model scale / task strength	On SpreadsheetBench the lightweight Gemini-3.1-Flash-Lite was the best extractor; GPT-5.4 ranked last despite the strongest baseline. Extraction is a distinct capability from execution.
Plausibility-rubric guidance (clarity, tone, structure…)	Adding a 7-dim "good writing" rubric to the extractor hurt average performance (−0.59 pts, worse in 6/9 cells). The 3-dim validated rubric improved all 9 cells (+1.55 pts).

Non-signal

Evidence

Format (list vs. checklist vs. prose)

Friedman test non-significant on every target (p > 0.34); extractor swap significant on 5/6 (p < 0.01). What a skill says matters, not how it looks.

How good the text "reads"

An unguided GPT-5.4 judge picked the better-performing skill 46.4% of the time — coin-flip. On the largest-gap pairs (δ ≥ 5 pts) it picked right only 15.8%: the skill that reads better often performs worse.

Extractor model scale / task strength

On SpreadsheetBench the lightweight Gemini-3.1-Flash-Lite was the best extractor; GPT-5.4 ranked last despite the strongest baseline. Extraction is a distinct capability from execution.

Plausibility-rubric guidance (clarity, tone, structure…)

Adding a 7-dim "good writing" rubric to the extractor hurt average performance (−0.59 pts, worse in 6/9 cells). The 3-dim validated rubric improved all 9 cells (+1.55 pts).

Lifecycle facts to remember

Fact	Number	Implication for you
Ungated model-generated skills cause negative transfer	25% of 150 extractor–target cells (47% on the worst domain)	This is why SkillOpt's validation gate exists. Never adopt a skill without held-out evidence.
All-failure experience pools produce the worst skills	Worst in every domain tested	Make sure your transcript/experience pool contains successes; the optimal success ratio is domain-specific (coding peaked mostly-successful).
The same skill text gives different gains per consumer	+1.8 to +9.5 pts across six targets, same skill	A skill trained for Claude Code may transfer to pi/Kiro — but re-validate; consumption ability is a per-model property.
Skills act by reshaping the default policy	Trajectory analysis, §5.3	A skill doesn't add abilities; it steers choices. A weak model can be steered into workflows it can't execute robustly — one source of negative transfer.
Rubric-guided judging beats unguided	46.4% → 73.8% judge accuracy	When you review a staged SkillOpt-Sleep proposal, score it against the three validated dimensions — don't trust your sense of "reads well".

Fact

Number

Implication for you

Ungated model-generated skills cause negative transfer

25% of 150 extractor–target cells (47% on the worst domain)

This is why SkillOpt's validation gate exists. Never adopt a skill without held-out evidence.

All-failure experience pools produce the worst skills

Worst in every domain tested

Make sure your transcript/experience pool contains successes; the optimal success ratio is domain-specific (coding peaked mostly-successful).

The same skill text gives different gains per consumer

+1.8 to +9.5 pts across six targets, same skill

A skill trained for Claude Code may transfer to pi/Kiro — but re-validate; consumption ability is a per-model property.

Skills act by reshaping the default policy

Trajectory analysis, §5.3

A skill doesn't add abilities; it steers choices. A weak model can be steered into workflows it can't execute robustly — one source of negative transfer.

Rubric-guided judging beats unguided

46.4% → 73.8% judge accuracy

When you review a staged SkillOpt-Sleep proposal, score it against the three validated dimensions — don't trust your sense of "reads well".

Reviewer checklist

Before adopting any skill artifact (staged Sleep proposal, best_skill.md, or hand-written seed):

☐ Does it name WHY failures happen, each with an executable remedy? ☐ Are steps concrete — real tools, objects, commands — not process platitudes? ☐ Does it blacklist specific harmful action patterns? ☐ Is there held-out evidence it improves this exact target model? ☐ Was it distilled from a pool containing successful trajectories?