Testing a skill with planted errors and a run without the skill

I asked Claude Code to make a skill that checks the units on a calculation sheet. A skill is a file of written instructions, a SKILL.md, that Claude uses when the task calls for it or when you type its name as a command.11Anthropic (2026), “Extend Claude with skills”, Claude Code documentation. It is like an SOP, but for the assistant.

Along with the instructions it wrote a script, check_units.py, that does the conversions, with a test where 15 of 15 values agree when they are worked out by two routes. It wrote two reference sheets, one of unit factors and one of unit traps. And it checked its sources first: the conversion factors in NIST SP 811,22Thompson and Taylor (2008), NIST Special Publication 811, Appendix B, “Conversion Factors”. the licence of pint, which is a Python library for units, and the NASA report on the Mars Climate Orbiter. That is the spacecraft that was lost because thruster data came in pound-force seconds where the specification asked for newton seconds, so the numbers were off by a factor of 4.45.33Mars Climate Orbiter Mishap Investigation Board (1999), Phase I Report, section 4, “MCO Root Cause”, p. 16.

Then it made the test, a reference problem, which is a short calculation sheet with four errors planted in it, and five pass criteria written down with it. The four errors were a gallon conversion, a temperature difference, a site atmospheric pressure, and a missing gc. If you have not worked in US customary units, gc is the constant that connects pound-mass and pound-force in an equation, 32.174 lbm ft per lbf s squared, and if you leave it out your answer is wrong by a factor of 32.17.

To me this is like testing a method in the lab, where you make a sample that you already know the answer for and give it to somebody who does not know the answer. Here the somebody was a fresh assistant each time, one that had seen nothing of the work. Some trials had the skill and some did not, and some were allowed to run the script while the others had to do the arithmetic by hand. It used two Claude models, Haiku, which is the small one, and Fable 5.1, which is the strong one.

In the first round, Haiku with no skill caught three of the four errors, the gallon conversion, the temperature difference and the atmospheric pressure, and missed the missing gc. Fable 5.1 with no skill found all four. So the strong model passes this problem without the skill. Haiku with the skill, working by hand, caught the missing gc and named the 32.17 ratio as the sign of it. Fable 5.1 with the skill and the script passed every criterion and also returned ten rough edges in the skill text.

So Claude revised the skill and ran the trials again. Haiku by hand had two defects in round one: it had not answered all four conversions, and its values did not all carry both the sheet unit and the SI unit. Both were fixed after the revision, and it found all four errors again, but it still failed. Fable 5.1 passed by hand and with the script. The Haiku run with the script ended in an API error, and the partial report it left had missed gc again. So on the small model the catch did not hold every time.

After that the skill text was frozen at 169 lines and a final round was run on the frozen text. Fable 5.1 passed by hand and with the script. Haiku 4.5 passed with the script. Haiku by hand found all the errors and failed one criterion, because a single SI value was 1.9 percent off.

Before trusting the record Claude audited the transcripts of the runs, because a stray file from one of the earlier runs had landed in the skill folder itself, so a trial could have seen the expected answers.

So the strong model did not need the skill on this problem, and the small one gained from it, though not reliably. All of that is one problem and two models from one company, and the skill still needs a run on another vendor’s assistant and my own review.

Footnotes

1. Anthropic (2026), “Extend Claude with skills”, Claude Code documentation. ↩

2. Thompson and Taylor (2008), NIST Special Publication 811, Appendix B, “Conversion Factors”. ↩

3. Mars Climate Orbiter Mishap Investigation Board (1999), Phase I Report, section 4, “MCO Root Cause”, p. 16. ↩