Utkrushta · Engineering Experiment Report

Verifying task-solvability on prod: what we tested, found, and should build

A hands-on measurement study — 2 agentic harnesses, 11 models, 3 real tasks, 60+ solve runs on real E2B sandboxes with real charges — to make task verification cheap and reliable at prod scale.

date 2026-07-07 providers tested OpenRouter · Anthropic · Ollama Cloud grading real cargo test / pytest

TL;DR

The interface, not the model, is the bottleneck. With structured tool-calling, cheap open-weight models solve the same tasks a frontier model does — including infra tasks with live services.
A real agentic solve costs ~$0.50 on Sonnet (infra), ~$0.02–0.07 (non-infra) — but the same solve is $0 marginal on an Ollama Cloud flat plan, and ~$0.01 on cheap API models. Roughly 10–50× cheaper than Sonnet.
Best model for us: kimi-k2.7-code (Ollama Cloud) — solved every task we threw at it, fast and token-efficient. glm-5.2 is the efficiency runner-up; nemotron-3-super fails infra; Devstral isn't on Ollama.
Solver runtime: Pi headless. Model-agnostic, tool-calling, driven from the backend; provisions an E2B sandbox for infra tasks and calls Ollama for the model.
Biggest cost lever is architectural: don't re-solve. Cache the verdict on the task; re-run only when the task's updated field changes. Position creation reads the cache.
Governance is fine: the strong coders are Chinese-origin but US-hosted by Ollama with zero retention — data never leaves to China. Origin ≠ hosting.

Core finding

Interface > model

Tool-calling decides whether cheap models can solve — proven on infra.

Best model

kimi-k2.7-code

Solved all 3 tasks; $0 marginal on Ollama Pro.

Cost lever

Cache verdict

Re-solve only when the task's updated field changes.

Solver runtime

Pi headless

Model-agnostic tool-calling + E2B for infra.

01The question & hypothesis

PR #37's task-audit bundle verifies the end-to-end task flow (task-repo-check → task-solvability → task-verify). It works from local Claude Code; we want it on prod, triggered by a backend DAG, on the tasks of every new position — without a runaway bill.

Only one of the four skills spends real money: task-solvability runs a full coding agent that solves each task RED→GREEN. task-audit itself is a zero-LLM Python linter. So "audit every task" means "run a coding agent to solve every task" — and that's what we measured. (The original idea of running it on a Claude subscription is a non-starter — Anthropic's 2026 ToS bans subscription tokens for automated/headless prod use.)

02What we tested

Two agentic harnesses, three real tasks, three providers, 60+ solve runs — each graded by the task's own test suite with real charges.

Harness	What it does	Edit interface
solve_bench_tools.py	agentic loop, local OR E2B sandbox	structured tool-calling
pi_bench.py + Pi	drives the real Pi coding-agent harness headless	Pi's native tool-calling

Dimension	Coverage
Tasks (3, real, ship RED)	live-watch-stats-aggregator & summarize-viewer-engagement (non-infra Rust, cargo test) · streamflix-watchlist-mutation-repair (infra: FastAPI + Strawberry GraphQL + live Postgres, pytest in a real E2B sandbox)
Providers (3)	OpenRouter (pay-per-token: Gemini, DeepSeek, Devstral, MiniMax, GLM) · Anthropic (Sonnet-5) · Ollama Cloud (flat plan: kimi, glm-5.2, qwen3.5, gpt-oss, nemotron, gemma4)
Models (11)	Sonnet-5, Gemini-2.5-flash-lite, DeepSeek-V4-Pro, GLM-5.2, Kimi-K2.7-Code, Qwen3.5, gpt-oss:120b, MiniMax-M3, Nemotron-3-Super, Devstral-2, Gemma 4
Solve runs	60+ across text-protocol and tool-calling, non-infra and infra

03Findings

1 The edit interface — not the model — is the bottleneck the big one

With a naïve text-delimiter protocol, cheap models fail non-deterministically (same task, temperature 0, solved one run, unparseable the next), worse as context grows — on the infra task, only Sonnet completed it. Swap to structured tool-calling and all of them solve. Devstral (72% SWE-bench) "failed" purely because its edit format couldn't be parsed; via tool-calling it solves cleanly. This is why the prod runtime must be a real tool-calling agent (Pi), not a hand-rolled text loop.

2 A real agent loop costs ~4–7× a one-shot rewrite cost

Tool-calling reads files, edits, runs tests, iterates over 20–40 calls with growing context. Sonnet's infra solve: $0.496 via tool-calling vs $0.114 one-shot. Budget ~$0.50/infra-task on Sonnet — it's how a real agent actually works.

3 Cheap open-weight coders match Sonnet's correctness at a fraction of the cost headline

On tool-calling, the strong Ollama coders solved every non-infra task (2/2) and 4 of 5 infra runs — at $0 marginal on the flat plan. Cheap API models (DeepSeek, MiniMax, Gemini) solved for ~$0.01. Sonnet remains the most consistent, but at 10–50× the price.

4 Origin ≠ hosting governance

GLM/Kimi/Qwen are Chinese-origin but open-weight; run via Ollama Cloud they execute on US infra with zero data retention — data never touches China. The rule is "no Chinese hosting," not "no Chinese model."

5 Reliability still varies → keep a reference gate reliability

Lite models (Gemini-2.5-flash-lite) sometimes decline to use tools and defer to the human; even strong ones flip run-to-run. A single cheap "unsolvable" isn't trustworthy — validate against a Sonnet reference before relying on it.

04The measured data — deeper

Interface proof — infra task, same task, two ways

Model (provider)	Text protocol	Tool-calling	$ tool-calling
gemini-2.5-flash-lite OpenRouter	✘ unparseable	✔ solved	0.010
devstral-2 OpenRouter	✘ unparseable	✔ solved	0.057
claude-sonnet-5 Anthropic	✔ 0.114	✔ solved	0.496

Non-infra Rust tasks — cost per solve (tool-calling, real charges)

Model	Provider	Solved	$ / solve
claude-sonnet-5	Anthropic	✔ reliable	0.02–0.07
deepseek-v4-pro	OpenRouter	✔	~0.011
minimax-m3	OpenRouter	✔	~0.005
devstral-2	OpenRouter	✔	~0.005
gemini-2.5-flash-lite	OpenRouter	flaky	~0.001
kimi / glm / qwen / gpt-oss	Ollama	✔ 2/2 each (via Pi)	flat ($0)

Ollama Cloud coding models — non-infra (via Pi) & infra (tool-calling)

Model	Origin	Non-infra	Infra	Infra speed	Infra tokens
kimi-k2.7-code	CN·🇺🇸-host	2/2	✔	102s	52.7k
glm-5.2	CN·🇺🇸-host	2/2	✔	128s	31.5k
gpt-oss:120b	🇺🇸	2/2	✔	98s	70k
qwen3.5	CN·🇺🇸-host	2/2	✔	163s	123k
nemotron-3-super	🇺🇸	—	✘ gave up	33s	17k

All Ollama runs = $0 marginal on the $20/mo Pro plan. Sonnet reference (Anthropic): ~$0.50/infra-solve, ~$0.47/non-infra via Pi. Devstral is not on Ollama Cloud (local-only). Lower tokens = less GPU-time = stretches the flat plan further, so glm-5.2 (31.5k) and kimi (52.7k) are the most plan-efficient.

05Provider landscape

Provider	Access	Pricing	Governance
Ollama Cloud	open-weight coders (kimi, glm, qwen, gpt-oss)	flat $20/$100 mo, $0 marginal	🇺🇸, zero-retention
OpenRouter	everything (Gemini, DeepSeek, Devstral, Sonnet)	pay-per-token (~$0.001–0.06)	routes to US hosts
Anthropic	Claude (the reliability reference)	API per-token (~$0.5/infra solve)	🇺🇸

06Production architecture — the DAG

A backend DAG fires with a batch of task IDs (e.g. 10 when a position is created) and must return a per-task shippable/not verdict. Skills are interactive runbooks — on prod you implement their logic as a service: the deterministic linter (a direct port of task_audit.py) plus the agentic solver.

DAG trigger
[10 task ids]→ cache gate
updated?→ queue
(only updated)→ Pi solver worker
cheap-first + gate→ write verdict
to task row

Cache gate — check the updated field. For each id, compare the task's updated field to the stored solvability verdict. Unchanged → return the cached verdict instantly. Only tasks whose updated field moved (regenerated / edited) get re-solved. Position creation reads this cache and never triggers a fresh solve on its own.
Enqueue the updated ones. Each becomes one solve job on a queue, decoupling the trigger from the (minutes-long) solve. With the cache gate, this is usually a small number.
Pi solver worker (headless). Per job, a containerized worker runs Pi headless — model-agnostic, tool-calling. It provisions an E2B sandbox for infra tasks (via the E2B key) and drives an Ollama Cloud model (via the Ollama key) to solve RED→GREEN. It grades against the task's tests and also runs the deterministic audit linter, emitting a combined verdict.
Escalation ladder for cost. Cheap model first (kimi-k2.7-code, $0 marginal). A PASS is self-validating — the tests really went green, so it's SOLVABLE regardless of model. Only a FAIL escalates to Sonnet to separate "genuinely unsolvable" from "cheap model too weak." Frontier price only on the minority that fail the cheap pass.
Write & cache the verdict. Store {verdict, grade_signal, model, ts} on the task row, stamped against the updated value it was verified at — so the next trigger's cache gate is exact.

Handling the 10-at-once trigger

Don't block the DAG on 10 minutes-long solves. The service returns immediately with {cached:[...], queued:[...]}; solves run in workers and write results to the task rows; the DAG polls the rows or takes a webhook. Because the cache gate filters everything whose updated field hasn't moved, a typical batch resolves mostly from cache with only the changed tasks actually solving.

07Recommendation

Cache the verdict; re-solve only on updated. The single biggest lever — turns "10 solves per position × M positions" into "each task solved once until it changes."
Solver = Pi headless, driving an Ollama Cloud model, provisioning an E2B sandbox for infra tasks. Real tool-calling is what makes cheap models work.
Escalation ladder: kimi-k2.7-code first ($0 marginal) → Sonnet only on failure. Reference-gate cheap verdicts before trusting an "unsolvable."
Model pick: kimi-k2.7-code (solved all 3 tasks, fast, efficient). glm-5.2 for max plan-efficiency; gpt-oss:120b for US-origin + speed. Drop nemotron (fails infra); Devstral isn't on Ollama.

08Caveats & sources

Read the numbers honestly

Reliability is measured on 1–2 runs per model per task — indicative, not conclusive; a harder task or more runs could still surface variance (hence the reference gate). Ollama token counts are high via agentic loops (GPU-time toward plan caps, though not per-token billed). Speeds are end-to-end wall (include tool exec + test runs). One integration step remains for a fully-Pi infra path: wiring Pi's Ollama-cloud provider inside the sandbox — the model ranking is unaffected.

Anthropic Agent SDK + subscription policy — support.claude.com/en/articles/15036540 OpenRouter live pricing — openrouter.ai/api/v1/models · Ollama pricing — ollama.com/pricing Devstral (72% SWE-bench) — tessl.io/blog/devstral · Pi harness — github.com/earendil-works/pi

Harnesses: solve_bench_tools.py · pi_bench.py + Pi · 11 models · 3 tasks · 60+ solve runs · real E2B · real OpenRouter / Anthropic / Ollama charges