18 of 20 with our skills. 3 of 20 without.
Same low-cost model, a quarter of the cost per solved task. Below is every result from 40 pre-registered blind trials, with sample sizes and method notes.
Pass rate and cost per solved task
In October 2026 we ran a pre-registered study: four hard Nim tasks, five blind runs per arm, a low-cost model (DeepSeek V4.1 Flash, maximum thinking) with and without the verified skill. Tasks, arms, metrics and scorers were fixed before the first run. Costs are recomputed from logged tokens at the provider's off-peak API rates.
| Task (5 runs per arm) | Verified skill | No skill | Fisher p |
|---|---|---|---|
| Bug report with a reproducer | 5/5 | 0/5 | 0.008 |
| C-to-Nim port of a tool with a latent bug | 5/5 | 0/5 | 0.008 |
| Error handling across modules | 3/5 | 0/5 | 0.17 |
| Manually managed container with lifecycle hooks | 5/5 | 3/5 | 0.44 |
| Total | 18/20 | 3/20 |
- Per attempt, the skill-guided agent used about twice the tokens (3.1M vs 1.6M) and cost about $0.048 vs $0.028.
- Per solved task, it cost about $0.05 vs $0.21: four times cheaper, because the unguided agent rarely finishes.
- A larger model (DeepSeek V4 Pro) without skills passed 0 of 6 on the bug-report and port tasks, at the same or up to four times the cost per attempt.
- On the container task the model already copes; there the skill adds about 30% cost for a gain that is not statistically significant.
Read the full write-up of the cost study.
Verified skill vs its unverified draft
The verified skill passed every run; its unverified draft passed one in three. Agents solved each task in a sandbox with the verified skill, the unverified draft of the same skill, or no skill, and deterministic checks scored the result. Recent runs used a low-cost, fast model in the DeepSeek-V4-Flash class.
| Task (3 runs per arm) | Verified skill | Unverified draft | No skill |
|---|---|---|---|
| Bug-report drafting | 3/3 | 1/3 | 0/3 |
| C-to-Nim port of a tool with a latent bug | 3/3 | 1/3 | 0/3 |
| Error-handling discipline across modules | 3/3 | 1/3 | 0/3 |
Probes across the catalog. On 12 of 17 skills, the agent without the skill failed a check that the skill-guided agent passed (one run per arm). The failures included a use-after-free under self-assignment aliasing, C-library build configuration that only works on one machine, fuzz harnesses missing required flags, and test runners that silently skip new tests.
Effort. Trimming one skill cut mean agent time per task by 44% (about 1,480 s to 824 s) at the same 3/3 pass rate. Maintenance pays off even when a task already passes.
What the compiler audit found
This evidence does not depend on agent behavior at all.
- 912 claims across 21 skills, checked against Nim 2.3.1 or Nimony 0.6.3. 783 are testable by a program; the rest are process advice.
- 690 compiler-run test programs back the testable claims.
- 127 statements corrected from the first drafts, and 104 given a precise caveat: right in the common case, wrong at an edge an agent will reach.
- 7 Nimony compiler bugs in error handling, #2674–#2680, filed 2026-10-15 and fixed upstream within two days. Four lost errors silently or crashed.
- Auditing the Nimony assembler skill found further toolchain bugs, including a silent miscompile. Some are public as #2686 and #2687.
Two corrections in detail: why a moved-from object segfaults on reuse and the Nimony release build that swallowed the error.
Scope and method notes
- Earlier three-arm trials are n=3 per arm; the cost study is n=5 per arm; catalog probes are n=1 per arm.
- Cost-study tasks were chosen because they had separated before, so its pass-rate gaps are an upper bound. One model family, one weekend of runs. Two verdicts hit by a scoring race were re-scored with the unchanged scorers.
- The C-to-Nim port result is after one skill revision (2/3 before); the other arms were not re-run.
- The error-handling result is after a documented scorer revision; the first scoring gave 2/3, 0/3, 0/3.
- Some probe separations measure whether the agent adopted a convention the skill teaches, such as evidence labels in a report.
- In probes, five skills had no separating task because every arm solved it; one of those later separated in the full trial above. On one doc-comment task the verified skill scored 2/3 against 3/3. Tasks that every arm solves are kept as regression checks.
- Run counts trace to our notes and datasets; raw run logs were not retained.
Stronger models will solve more tasks unaided. We track that and prune guidance that stops helping, which keeps the skills small.
What comes next
A larger published benchmark: Nim and Nimony tasks across several models and compiler versions, with and without the skills, with every model, version and sample size stated. Task contents stay private so agents cannot train to the test.