Testing a Leading Word
Ian, a leading word is a hypothesis about behaviour. Matt's guide says there is no automated eval and to check by running the document. Promptfoo can automate the running part. It has a guide for testing skills, which is what this lesson is built on.
The setup
Promptfoo compares two skill versions by holding the model, task files and permissions constant and swapping only the SKILL.md; any difference should then come from the skill text (source). That is the shape of a leading-word test: one document edited, everything else frozen.
The guide's three questions map onto leading words like this:
| Question | Assertion | What it tests for you |
|---|---|---|
| Does the agent use the skill? | skill-used | Pointer wording (the description) |
| Does it avoid a neighbour? | not-skill-used on a near-miss prompt | Trigger boundary |
| Does the work improve? | JavaScript or rubric on output and tool calls | Body wording (execution) |
Route A: Improving's promptfoo demo (use this first)
improving/promptfoo is a demo of Evaluation Driven Development. Its pattern is an exec: provider that shells out to a CLI, plus the same script acting as the rubric judge. It ships kiro.js, which wraps kiro-cli chat --no-interactive, so your own agent is the system under test. It pins promptfoo 0.121.17 and says to always use --no-cache, because promptfoo's cache cannot see changes inside external prompt files.
My adaptation is in eval-improving/, with the repo's kiro.js copied unchanged. It has three arms, one task, one model:
| Arm | Prompt contents |
|---|---|
none | No skill. The baseline. |
triad | Your diagnose body as written: "fast, deterministic, runnable pass/fail signal". |
tight | The same body with tight and red on lines 16, 27 and 42. |
The skill text is pasted into the prompt and Kiro is told it cannot run commands, so it writes the commands it would run. Three llm-rubric assertions follow Improving's one-dimension-per-assertion style: loop-first, red-capable and raises-repro-rate, plus a cheap keyword check.
cd eval-improving
npm install
npx promptfoo eval -c promptfooconfig.yaml --no-cache --repeat 3
npx promptfoo view
Config validates, and headless Kiro works with your existing login (the demo's docs ask for KIRO_API_KEY; I did not need it). I ran one pass, repeat 1: none 0.00, triad 0.78, tight 0.96. That is one sample per arm and proves nothing. Treat it only as evidence the pipeline runs.
On the none arm, the judge said the output was "empty". The raw output was 556 characters of a correct, short answer. So the judge is itself a source of error, here a false negative. Always read a sample of judge reasons before trusting a score. Also, both skill arms ended with ", Ian.": your global Kiro rules leak into the provider and judge. Equal across arms, but it means this is not a clean-room test.
Route B: Claude Agent SDK (tests the trigger)
Route A pastes the skill into the prompt, so it cannot tell you whether the agent chooses the skill. For that you need a harness with real skill discovery. eval/ uses promptfoo's Claude Agent SDK provider with skill-used, not-skill-used on a near-miss prompt, and a Bash-before-Edit check. I validated this config (promptfoo@0.125.1) but have not run it, so I do not know whether its tool-call names behave as intended. It gives the agent Bash, Edit and Write, so keep fixtures free of secrets (Promptfoo warns Agent SDK runs can execute arbitrary code: source).
cd eval
npx promptfoo@0.125.1 eval -c promptfooconfig.yaml --repeat 5 --no-cache
Rough cost: Route A with 3 repeats is 9 generations plus 27 judge calls. Route B with 5 repeats is 20 agent sessions.
What this eval cannot prove
These are the reasons a green result can mislead you. They are my analysis unless a link is given.
- Confound. The
tightvariant changes three lines and drops the "faster? sharper? more deterministic?" questions. A win is not a win for the word alone. A cleaner test swaps one site at a time. - Proxy. In Route A the agent writes a plan, not actions, so you measure what it says it would do. In Route B, "Bash before Edit" stands in for "built a red loop before fixing", and an agent can run
lsfirst and pass. This is the Explain the Number principle from pstack: rule out that you measured something other than the work you meant (poteto-mode). Inspect the traces of a few passes. - Ceiling. If the task is easy, both versions pass and you learn nothing. Another skills repo notes that evals written from the skill itself score near 100% and are not evidence it changes behaviour (source). Write tasks from real failures you have seen.
- Judge error. The rubric judge is an LLM too. See the false "empty output" above. Spot-check its reasons, and keep deterministic assertions alongside it.
- Noise. Agent runs vary. Five repeats can separate 5/5 from 1/5, not 5/5 from 4/5. Promptfoo's guidance is to rerun with repeats and inspect failures before choosing a version (source).
- One model. A word that leads one model may do nothing on another. The same config can target other harnesses (Codex, OpenCode) per the guide.
Other options
| Tool | What I know | Status |
|---|---|---|
Promptfoo with Improving's kiro.js | Text-in, text-out eval against Kiro CLI, rubric judge via the same script | Smoke-run once, 3 arms |
| Promptfoo, Claude Agent SDK provider | Skill-aware: skill-used / not-skill-used, tool-call checks | Docs read, config validated, not run |
| aws-samples/sample-agent-skill-eval | Describes with-skill vs without-skill, trigger precision, cost | Search snippet only |
| tardigrde/agent-skill-eval | Runs through the full harness; catches never-triggers | Search snippet only |
| LLM-as-judge (llm-rubric) | Grades open-ended output when exact checks are too brittle | Snippet only; next step if the JS proxy is too crude |
Gap that remains: nothing here measures whether Kiro triggers your skill from its description. Over exec: you get text only. That needs a skill-aware harness, which is Route B on Claude Code or a future Kiro-native one.
Retrieval check
Which assertion tells you the skill stays out of the way on a nearby task?
Yes. The promptfoo guide tests routing with near-miss prompts and asserts the skill you do not want is absent.
Those only measure the happy path. To test a boundary you need a prompt that belongs elsewhere and an assertion that the skill did not fire.
Tight passes 5 of 5 runs and triad passes 4 of 5. What follows?
Right. One miss in five is within run-to-run noise. Add repeats and look at why the failure failed before deciding.
That draws a conclusion from one lucky or unlucky run. The guide recommends rerunning with repeats and inspecting failures first.
Practice
- Read
eval-improving/promptfooconfig.yaml. Predict, before running: the order of the three arms by score, and whethernoneortriadis closer totight. - Run Route A with
--repeat 3or more (or ask me to). Note the score per arm and per metric. - Open the web view. Read the judge reasons for two passes and two failures. Does the judge agree with what you would conclude by reading the output?
- Write down one reason the result could be wrong (confound, proxy, ceiling, noise).
- Report back. I will write your learning record from your prediction versus the result.