Cost per solved task: what a cheap coding agent really costs on Nim

18 of 20 with the skill, 3 of 20 without

We gave a low-cost model, DeepSeek V4.1 Flash at maximum thinking, four hard Nim tasks and ran each one five times with our verified skill and five times without. Every run was blind and sandboxed, scored by deterministic checks, with the plan fixed before the first run.

Why cost per attempt misleads

With the skill loaded, the agent read more, checked its own output more often and made more tool calls. That doubled the tokens per attempt. Compare agents per attempt and the unguided one wins, but only because most of its attempts stopped early with a wrong answer.

Look at the totals instead. Twenty skill-guided attempts cost $0.97 and produced 18 solutions. Twenty unguided attempts cost $0.62 and produced 3. On three of the four tasks the unguided agent never passed in five tries, so its cost per solved task there is not higher; it is undefined. You can keep paying for attempts and still have nothing to merge.

The API bill is a few cents either way. The expensive part is an engineer reviewing a confident, plausible result that is wrong. The skill cut that from 17 results in 20 to 2 in 20.

The four tasks

Each task is work where an agent can produce something that compiles and looks finished but fails a check that matters in production. Task contents stay private so that models cannot be trained to the test.

Cost per solve is skill / no skill; a dash means no run solved the task. The bug report and the port are five passes against zero, each significant on its own. Error handling, 3/5 against 0/5, is a large gap that five runs cannot confirm alone.

“Just use a bigger model?”

On the bug-report and port tasks we also ran DeepSeek V4 Pro, a larger and pricier model, without skills.

The larger model passed 0 of 6. On the port it cost about four times as much per attempt, took about twice as long and failed the same checks as the cheap unguided model. A bigger model knows more in general. It did not discover the Nim-specific conventions and pitfalls that decide these tasks. The skill puts them in front of the agent, so a cheap model gets them right.

Where the skill didn’t pay

On the container task the model already copes: 3 of 5 without the skill, 5 of 5 with it. That gain is not significant, and each skill-guided attempt cost about 30% more. Cost per solved task was still lower, but we would not sell a skill on that margin. Every line of guidance costs tokens on every run, so when a model starts solving a task unaided, we prune the guidance for it.

Method

For the kind of Nim code that compiles, passes a quick test and is still wrong, see why a moved-from object segfaults on reuse.