Utkrushta · Engineering Experiment Report
Verifying task-solvability on prod: what we tested, found, and should build
A hands-on measurement study — 2 agentic harnesses, 11 models, 3 real tasks, 60+ solve runs on real E2B sandboxes with real charges — to make task verification cheap and reliable at prod scale.
TL;DR
- The interface, not the model, is the bottleneck. With structured tool-calling, cheap open-weight models solve the same tasks a frontier model does — including infra tasks with live services.
- A real agentic solve costs ~$0.50 on Sonnet (infra), ~$0.02–0.07 (non-infra) — but the same solve is $0 marginal on an Ollama Cloud flat plan, and ~$0.01 on cheap API models. Roughly 10–50× cheaper than Sonnet.
- Best model for us:
kimi-k2.7-code(Ollama Cloud) — solved every task we threw at it, fast and token-efficient.glm-5.2is the efficiency runner-up;nemotron-3-superfails infra; Devstral isn't on Ollama. - Solver runtime: Pi headless. Model-agnostic, tool-calling, driven from the backend; provisions an E2B sandbox for infra tasks and calls Ollama for the model.
- Biggest cost lever is architectural: don't re-solve. Cache the verdict on the task; re-run only when the task's
updatedfield changes. Position creation reads the cache. - Governance is fine: the strong coders are Chinese-origin but US-hosted by Ollama with zero retention — data never leaves to China. Origin ≠ hosting.
Core finding
Interface > model
Tool-calling decides whether cheap models can solve — proven on infra.
Best model
kimi-k2.7-code
Solved all 3 tasks; $0 marginal on Ollama Pro.
Cost lever
Cache verdict
Re-solve only when the task's
updated field changes.Solver runtime
Pi headless
Model-agnostic tool-calling + E2B for infra.
01The question & hypothesis
PR #37's task-audit bundle verifies the end-to-end task flow (task-repo-check → task-solvability → task-verify). It works from local Claude Code; we want it on prod, triggered by a backend DAG, on the tasks of every new position — without a runaway bill.
Only one of the four skills spends real money: task-solvability runs a full coding agent that solves each task RED→GREEN. task-audit itself is a zero-LLM Python linter. So "audit every task" means "run a coding agent to solve every task" — and that's what we measured. (The original idea of running it on a Claude subscription is a non-starter — Anthropic's 2026 ToS bans subscription tokens for automated/headless prod use.)
02What we tested
Two agentic harnesses, three real tasks, three providers, 60+ solve runs — each graded by the task's own test suite with real charges.
| Harness | What it does | Edit interface |
|---|---|---|
| solve_bench_tools.py | agentic loop, local OR E2B sandbox | structured tool-calling |
| pi_bench.py + Pi | drives the real Pi coding-agent harness headless | Pi's native tool-calling |
| Dimension | Coverage |
|---|---|
| Tasks (3, real, ship RED) | live-watch-stats-aggregator & summarize-viewer-engagement (non-infra Rust, cargo test) · streamflix-watchlist-mutation-repair (infra: FastAPI + Strawberry GraphQL + live Postgres, pytest in a real E2B sandbox) |
| Providers (3) | OpenRouter (pay-per-token: Gemini, DeepSeek, Devstral, MiniMax, GLM) · Anthropic (Sonnet-5) · Ollama Cloud (flat plan: kimi, glm-5.2, qwen3.5, gpt-oss, nemotron, gemma4) |
| Models (11) | Sonnet-5, Gemini-2.5-flash-lite, DeepSeek-V4-Pro, GLM-5.2, Kimi-K2.7-Code, Qwen3.5, gpt-oss:120b, MiniMax-M3, Nemotron-3-Super, Devstral-2, Gemma 4 |
| Solve runs | 60+ across text-protocol and tool-calling, non-infra and infra |
03Findings
1 The edit interface — not the model — is the bottleneck the big one
With a naïve text-delimiter protocol, cheap models fail non-deterministically (same task, temperature 0, solved one run, unparseable the next), worse as context grows — on the infra task, only Sonnet completed it. Swap to structured tool-calling and all of them solve. Devstral (72% SWE-bench) "failed" purely because its edit format couldn't be parsed; via tool-calling it solves cleanly. This is why the prod runtime must be a real tool-calling agent (Pi), not a hand-rolled text loop.2 A real agent loop costs ~4–7× a one-shot rewrite cost
Tool-calling reads files, edits, runs tests, iterates over 20–40 calls with growing context. Sonnet's infra solve: $0.496 via tool-calling vs $0.114 one-shot. Budget ~$0.50/infra-task on Sonnet — it's how a real agent actually works.3 Cheap open-weight coders match Sonnet's correctness at a fraction of the cost headline
On tool-calling, the strong Ollama coders solved every non-infra task (2/2) and 4 of 5 infra runs — at $0 marginal on the flat plan. Cheap API models (DeepSeek, MiniMax, Gemini) solved for ~$0.01. Sonnet remains the most consistent, but at 10–50× the price.4 Origin ≠ hosting governance
GLM/Kimi/Qwen are Chinese-origin but open-weight; run via Ollama Cloud they execute on US infra with zero data retention — data never touches China. The rule is "no Chinese hosting," not "no Chinese model."5 Reliability still varies → keep a reference gate reliability
Lite models (Gemini-2.5-flash-lite) sometimes decline to use tools and defer to the human; even strong ones flip run-to-run. A single cheap "unsolvable" isn't trustworthy — validate against a Sonnet reference before relying on it.04The measured data — deeper
Interface proof — infra task, same task, two ways
| Model (provider) | Text protocol | Tool-calling | $ tool-calling |
|---|---|---|---|
| gemini-2.5-flash-lite OpenRouter | ✘ unparseable | ✔ solved | 0.010 |
| devstral-2 OpenRouter | ✘ unparseable | ✔ solved | 0.057 |
| claude-sonnet-5 Anthropic | ✔ 0.114 | ✔ solved | 0.496 |
Non-infra Rust tasks — cost per solve (tool-calling, real charges)
| Model | Provider | Solved | $ / solve |
|---|---|---|---|
| claude-sonnet-5 | Anthropic | ✔ reliable | 0.02–0.07 |
| deepseek-v4-pro | OpenRouter | ✔ | ~0.011 |
| minimax-m3 | OpenRouter | ✔ | ~0.005 |
| devstral-2 | OpenRouter | ✔ | ~0.005 |
| gemini-2.5-flash-lite | OpenRouter | flaky | ~0.001 |
| kimi / glm / qwen / gpt-oss | Ollama | ✔ 2/2 each (via Pi) | flat ($0) |
Ollama Cloud coding models — non-infra (via Pi) & infra (tool-calling)
| Model | Origin | Non-infra | Infra | Infra speed | Infra tokens |
|---|---|---|---|---|---|
| kimi-k2.7-code | CN·🇺🇸-host | 2/2 | ✔ | 102s | 52.7k |
| glm-5.2 | CN·🇺🇸-host | 2/2 | ✔ | 128s | 31.5k |
| gpt-oss:120b | 🇺🇸 | 2/2 | ✔ | 98s | 70k |
| qwen3.5 | CN·🇺🇸-host | 2/2 | ✔ | 163s | 123k |
| nemotron-3-super | 🇺🇸 | — | ✘ gave up | 33s | 17k |
All Ollama runs = $0 marginal on the $20/mo Pro plan. Sonnet reference (Anthropic): ~$0.50/infra-solve, ~$0.47/non-infra via Pi. Devstral is not on Ollama Cloud (local-only). Lower tokens = less GPU-time = stretches the flat plan further, so glm-5.2 (31.5k) and kimi (52.7k) are the most plan-efficient.
05Provider landscape
| Provider | Access | Pricing | Governance |
|---|---|---|---|
| Ollama Cloud | open-weight coders (kimi, glm, qwen, gpt-oss) | flat $20/$100 mo, $0 marginal | 🇺🇸, zero-retention |
| OpenRouter | everything (Gemini, DeepSeek, Devstral, Sonnet) | pay-per-token (~$0.001–0.06) | routes to US hosts |
| Anthropic | Claude (the reliability reference) | API per-token (~$0.5/infra solve) | 🇺🇸 |
06Production architecture — the DAG
A backend DAG fires with a batch of task IDs (e.g. 10 when a position is created) and must return a per-task shippable/not verdict. Skills are interactive runbooks — on prod you implement their logic as a service: the deterministic linter (a direct port of task_audit.py) plus the agentic solver.
DAG trigger
[10 task ids]→ cache gate
updated?→ queue
(only updated)→ Pi solver worker
cheap-first + gate→ write verdict
to task row
[10 task ids]→ cache gate
updated?→ queue
(only updated)→ Pi solver worker
cheap-first + gate→ write verdict
to task row
- Cache gate — check the
updatedfield. For each id, compare the task'supdatedfield to the stored solvability verdict. Unchanged → return the cached verdict instantly. Only tasks whoseupdatedfield moved (regenerated / edited) get re-solved. Position creation reads this cache and never triggers a fresh solve on its own. - Enqueue the updated ones. Each becomes one solve job on a queue, decoupling the trigger from the (minutes-long) solve. With the cache gate, this is usually a small number.
- Pi solver worker (headless). Per job, a containerized worker runs Pi headless — model-agnostic, tool-calling. It provisions an E2B sandbox for infra tasks (via the E2B key) and drives an Ollama Cloud model (via the Ollama key) to solve RED→GREEN. It grades against the task's tests and also runs the deterministic audit linter, emitting a combined verdict.
- Escalation ladder for cost. Cheap model first (kimi-k2.7-code, $0 marginal). A PASS is self-validating — the tests really went green, so it's SOLVABLE regardless of model. Only a FAIL escalates to Sonnet to separate "genuinely unsolvable" from "cheap model too weak." Frontier price only on the minority that fail the cheap pass.
- Write & cache the verdict. Store
{verdict, grade_signal, model, ts}on the task row, stamped against theupdatedvalue it was verified at — so the next trigger's cache gate is exact.
Handling the 10-at-once trigger
Don't block the DAG on 10 minutes-long solves. The service returns immediately with {cached:[...], queued:[...]}; solves run in workers and write results to the task rows; the DAG polls the rows or takes a webhook. Because the cache gate filters everything whose updated field hasn't moved, a typical batch resolves mostly from cache with only the changed tasks actually solving.07Recommendation
- Cache the verdict; re-solve only on
updated. The single biggest lever — turns "10 solves per position × M positions" into "each task solved once until it changes." - Solver = Pi headless, driving an Ollama Cloud model, provisioning an E2B sandbox for infra tasks. Real tool-calling is what makes cheap models work.
- Escalation ladder: kimi-k2.7-code first ($0 marginal) → Sonnet only on failure. Reference-gate cheap verdicts before trusting an "unsolvable."
- Model pick: kimi-k2.7-code (solved all 3 tasks, fast, efficient). glm-5.2 for max plan-efficiency; gpt-oss:120b for US-origin + speed. Drop nemotron (fails infra); Devstral isn't on Ollama.
08Caveats & sources
Read the numbers honestly
Reliability is measured on 1–2 runs per model per task — indicative, not conclusive; a harder task or more runs could still surface variance (hence the reference gate). Ollama token counts are high via agentic loops (GPU-time toward plan caps, though not per-token billed). Speeds are end-to-end wall (include tool exec + test runs). One integration step remains for a fully-Pi infra path: wiring Pi's Ollama-cloud provider inside the sandbox — the model ranking is unaffected.Anthropic Agent SDK + subscription policy — support.claude.com/en/articles/15036540
OpenRouter live pricing — openrouter.ai/api/v1/models · Ollama pricing — ollama.com/pricing
Devstral (72% SWE-bench) — tessl.io/blog/devstral · Pi harness — github.com/earendil-works/pi
Harnesses: solve_bench_tools.py · pi_bench.py + Pi · 11 models · 3 tasks · 60+ solve runs · real E2B · real OpenRouter / Anthropic / Ollama charges