The Skill Audit Runbook

Lesson 6 · Building Better Skills & Agents · ~16 minutes

This is the capstone. Lessons 1–5 gave you vocabulary (axes), discipline (steering, pruning), perspective (lifecycles), and architecture (invocation). This lesson synthesises all of it into a publishable, repeatable audit method you can run on any skill — yours or someone else's.

The Two-Minute Audit

For monthly maintenance. Run this on every skill that's been modified or invoked recently. Total time: 120 seconds per skill.

5s

1. Invocation Check

Is it model-invoked or user-invoked? Is that the right mode? Is the description pulling its weight as a routing signal (model-invoked) or a discovery aid (user-invoked)?

10s

2. Hierarchy Scan

Can you find the workflow steps within 3 seconds of opening the file? Is reference material disclosed progressively or sprawled upfront? If you can't find the steps immediately, the agent can't either.

60s

3. Steering Sweep

Scan for negations (reframe positively). Check each step's completion criterion — is "done" defined? Look for leading words that cheaply anchor behaviour. Flag any step that could end prematurely.

45s

4. Pruning Pass

Run the deletion test on every rule: "If I remove this, what changes?" Flag no-ops and duplicates. Check for sediment from previous skill versions that was never cleaned up.

The two-minute target is real

With practice, you can audit a skill in under two minutes. The constraint is the point — it forces you to look at structure, not content. If a skill's quality isn't visible in two minutes, it has a hierarchy problem.

The Deep Audit

For quarterly reviews. Run this on your full portfolio — not skill-by-skill, but portfolio-wide.

Lifecycle Check

For each skill: should it be promoted, renamed, merged, or killed? Use the signals from Lesson 4:

Context Budget

Recalculate: how many model-invoked skills × average description length = total context rent. Has it grown since last quarter? Should any skills switch to user-invoked?

Cross-Skill Coherence

The Scoring Rubric

Rate each axis 1–3:

ScoreInvocationHierarchySteeringPruning
1 — Needs work Wrong mode, or description doesn't route Steps hard to find, reference sprawled Negations, missing completion criteria, no leading words Multiple no-ops, visible sediment, duplication
2 — Solid Right mode, description adequate Steps findable in 5s, some disclosure Few negations, most criteria clear, some leading words Minor sediment, no obvious no-ops
3 — Exemplary Right mode, description precisely triggers correct invocation Steps in 3s, progressive disclosure, no sprawl Zero negations, every step has completion criterion, strong leading words Every line is load-bearing, zero duplication

Worked Example: Auditing write-a-skill

Let's run the full audit on a real skill:

~/.kiro/skills/write-a-skill/SKILL.md name: write-a-skill description: Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, or build a new skill. dependencies: ["writing-great-skills"] <objective> Help the user create a well-structured skill. Gather requirements, draft it, review with the user. </objective> <workflow> ## 1. Gather requirements Ask: - What task/domain does the skill cover? - What specific use cases should it handle? - Does it need executable scripts or just instructions? - Any reference materials to include? ## 2. Draft the skill Structure: skill-name/ ├── SKILL.md ├── REFERENCE.md ├── EXAMPLES.md └── scripts/ ## 3. Review with user - Does this cover your use cases? - Anything missing or unclear? - Should any section be more/less detailed? </workflow>

Audit Results

write-a-skill — Scores

Invocation
3
Hierarchy
2
Steering
1
Pruning
2
AxisFindingScore
Invocation User-invoked (correct — skill authoring is a deliberate act). Description has clear triggers ("create, write, or build a new skill"). 26 words — efficient. 3
Hierarchy Steps findable within 5 seconds (numbered workflow). But the "Description requirements", "When to split files", and "Review checklist" sections are mixed into the workflow instead of being progressively disclosed in a REFERENCE.md. 2
Steering No completion criteria on any step. Step 1 asks 4 questions — but when is "gather requirements" done? After all 4? After 2? The agent can end prematurely or keep asking forever. No leading words anchor the drafting style. No branch handling (what if user already has a draft?). 1
Pruning Relatively lean (under 100 lines). But the "Review checklist" partially duplicates the "Description requirements" section (both mention triggers). Minor sediment: the file structure diagram is static reference that could be in REFERENCE.md. 2

Highest-Leverage Improvement

Add completion criteria to each workflow step. Step 1 is done when: all 4 questions are answered OR user says "just draft it." Step 2 is done when: SKILL.md is written and shown to user. Step 3 is done when: user confirms or provides revisions. This single change would move Steering from 1 to 2.

The Maintenance Calendar

CadenceWhatTime
MonthlyTwo-Minute Audit on every skill modified or invoked that month~30 min for 15 skills
QuarterlyDeep Audit — lifecycle check, context budget, cross-skill coherence~2 hours for full portfolio
AnnualLifecycle review — kill dormant skills, assess if the portfolio structure still matches your workflow~1 hour

The Reference Card

A one-page summary of the method, structured for printing or pinning:

Skill Audit — Quick Reference

  1. Invocation (5s): Right mode? Description routes correctly?
  2. Hierarchy (10s): Steps in 3 seconds? Reference disclosed progressively?
  3. Steering (60s): Negations? Completion criteria? Leading words?
  4. Pruning (45s): Deletion test every rule. No-ops? Sediment? Duplication?

Score: Rate each 1–3. Priority: fix lowest score first.

Lifecycle: Promote (proven + scoped + named) · Rename (name misleads) · Merge (unclear boundary) · Kill (60+ days unused)

Context Budget: model-invoked count × avg description words = total rent

Cadence: Monthly quick audit · Quarterly deep audit · Annual lifecycle cull

Check Your Understanding

You're running a Two-Minute Audit and find a skill with this workflow step: "Write good documentation." There's no completion criterion. What's the primary issue?
Correct. A step without a completion criterion is a steering failure — the agent has no signal for when to stop. "Write good documentation" could mean a one-line README or a 50-page manual. Add a criterion: "Done when: README.md exists with install, usage, and API sections."
This is primarily a steering issue. "Write good documentation" has no completion criterion — the agent doesn't know what "done" looks like. It might also be a no-op (pruning), but the steering failure is more critical because it causes premature completion or endless work.
During a quarterly Deep Audit, you find that both your "architect" skill and your "code-review" skill say "Apply DRY principles." What's this?
Correct. Cross-skill duplication means the same rule is maintained in multiple places. If you update how you think about DRY in one skill, you must remember to update the other. The fix: define it in one place (a shared dependency or the user's me.md principles) and reference it.
This is cross-skill duplication — a coherence issue found during deep audits. Having "Apply DRY" in two skills means two maintenance points. It's not a merge signal (the skills serve different purposes) and it's not acceptable reinforcement (repetition between skills creates drift). Put it in one place.
A skill scores: Invocation 3, Hierarchy 3, Steering 2, Pruning 1. Where do you focus improvement effort?
Correct. Fix the lowest score first. A Pruning score of 1 means no-ops, sediment, or duplication are actively diluting the skill's signal. Cleaning those out is the highest-leverage improvement — it makes the steering rules more visible by removing noise around them.
Always fix the lowest score first. Pruning at 1 means the skill has dead weight — no-ops, sediment, or duplication — that actively dilutes everything else. Fixing pruning also improves perceived steering quality because load-bearing instructions become more prominent once noise is removed.
You run the maintenance calendar for a quarter. Your portfolio grew from 28 to 35 skills. Context budget went from ~1,200 to ~1,800 words. What's the right action?
Correct. Growth isn't inherently bad, but 50% context budget growth in one quarter demands scrutiny. Audit each new skill's invocation mode. Also revisit existing skills — some that were worth the context rent at 1,200 words might not be at 1,800. The goal is intentional budget allocation, not arbitrary caps.
A 50% context budget increase in one quarter needs investigation, not acceptance or blanket rules. Audit each new skill individually — is model-invocation justified? Then revisit existing ones. Some skills that earned their context rent at lower portfolio sizes may not justify it as the total grows.

Your Exercise

  1. Pick your 3 most-used skills (the ones you invoke most often).
  2. Run the full Two-Minute Audit on each. Score all four axes 1–3.
  3. For each, identify the single highest-leverage improvement (lowest score → highest impact fix).
  4. Implement at least one improvement. Re-score after.
  5. Run the context budget calculation for your full portfolio. Is it sustainable?
The final insight

You now have a methodology. It's not a one-time exercise — it's a maintenance practice. Skills that are audited regularly stay sharp. Skills that aren't drift toward sediment, duplication, and stale invocation modes. The calendar is the mechanism. Publish it, teach it, refine it.

📖 Primary Source

Agents at Scale — Lesson 10: Anti-Patterns — The enterprise governance angle: what happens when skill portfolios are managed across teams, not just individuals. The audit discipline scales from 30 skills to 300.

Questions? Send me your 3 audit scorecards and I'll validate your ratings, challenge anything that looks generous, and help you prioritise the fixes.
← Prev