Utkrushta · Engineering Experiment Report

Verifying task-solvability on prod: what we tested, found, and should build

A hands-on measurement study — 2 agentic harnesses, 11 models, 3 real tasks, 60+ solve runs on real E2B sandboxes with real charges — to make task verification cheap and reliable at prod scale.

01The question & hypothesis

PR #37's task-audit bundle verifies the end-to-end task flow (task-repo-check → task-solvability → task-verify). It works from local Claude Code; we want it on prod, triggered by a backend DAG, on the tasks of every new position — without a runaway bill.

Only one of the four skills spends real money: task-solvability runs a full coding agent that solves each task RED→GREEN. task-audit itself is a zero-LLM Python linter. So "audit every task" means "run a coding agent to solve every task" — and that's what we measured. (The original idea of running it on a Claude subscription is a non-starter — Anthropic's 2026 ToS bans subscription tokens for automated/headless prod use.)

02What we tested

Two agentic harnesses, three real tasks, three providers, 60+ solve runs — each graded by the task's own test suite with real charges.

03Findings

04The measured data — deeper

Interface proof — infra task, same task, two ways

Non-infra Rust tasks — cost per solve (tool-calling, real charges)

Ollama Cloud coding models — non-infra (via Pi) & infra (tool-calling)

All Ollama runs = $0 marginal on the $20/mo Pro plan. Sonnet reference (Anthropic): ~$0.50/infra-solve, ~$0.47/non-infra via Pi. Devstral is not on Ollama Cloud (local-only). Lower tokens = less GPU-time = stretches the flat plan further, so glm-5.2 (31.5k) and kimi (52.7k) are the most plan-efficient.

05Provider landscape

06Production architecture — the DAG

A backend DAG fires with a batch of task IDs (e.g. 10 when a position is created) and must return a per-task shippable/not verdict. Skills are interactive runbooks — on prod you implement their logic as a service: the deterministic linter (a direct port of task_audit.py) plus the agentic solver.

1. Cache gate — a content hash. tasks has no updated timestamp, so for each id we compute a content_hash (sha256 of task_blob + criterias + pre_requisites + answer + template_id) and look for a task_audits row with that exact hash. Match → return that row's verdict instantly (no Fargate). Only tasks whose audited content actually changed (or were never audited) get re-audited — more precise than a timestamp (an incidental row touch doesn't trigger a re-run). Position creation reads this cache and never triggers a fresh audit on its own.
2. Enqueue the changed ones. Each becomes one audit job on a queue, decoupling the trigger from the (minutes-long) solve. With the cache gate, this is usually a small number.
3. Pi audit worker (headless). Per job, a containerized worker runs Pi headless — model-agnostic, tool-calling. It provisions an E2B sandbox for infra tasks (via the E2B key) and drives an Ollama Cloud model (via the Ollama key) to solve RED→GREEN. It grades against the task's tests and also runs the deterministic integrity linter, emitting a combined verdict.
4. Escalation ladder for cost. Cheap model first (kimi-k2.7-code, $0 marginal). A PASS is self-validating — the tests really went green, so it's SOLVABLE regardless of model. Only a FAIL escalates to Sonnet to separate "genuinely unsolvable" from "cheap model too weak." Frontier price only on the minority that fail the cheap pass.
5. Write the audit row. INSERT one row into task_audits {verdict, findings, solvable, solved_by, failure_reason, trace_url, detail, content_hash, cost_usd, …} — one row per audit run (full history, "failed → fixed → passed"), stamped with the content_hash it was verified at, so the next trigger's cache gate is an exact hash match.

Deployment topology — where each piece runs

Mirrors the existing v2-grader Fargate pattern: an Airflow DAG launches one ephemeral Fargate task per job via EcsRunTaskOperator. What's new is the sandbox-agent worker (Pi + Ollama Cloud) and a sandbox for isolated test execution. Build it as one generic worker parameterized by job-type (audit / code-grade / auto-fix / regression-sweep).

What to build (reusing the grader scaffolding): a runner package (python runner.py --task-id … --persist db) → a slim arm64 Dockerfile → an ECR CI workflow → an audit_task DAG with EcsRunTaskOperator → the ECS task-def utkrusht-task-audit-{dev,prod} + IAM (E2B/Ollama keys in Secrets Manager, egress to e2b.app + ollama.com). Codify the AWS side in IaC (the grader's is hand-created in the console) and keep one shared core (the grader paid a "lift-duplication tax" copying compute).

Do we need E2B? Yes — and here's the sandbox call

A sandbox is required for infra tasks: ECS Fargate can't run nested containers, so the Postgres/Kafka service stack needs a real VM. The question is which VM — and we have $20k E2B credits.

07Recommendation

1. Cache the verdict; re-audit only when content_hash changes. The single biggest lever — turns "10 audits per position × M positions" into "each task audited once until its content changes."
2. Solver = Pi headless, driving an Ollama Cloud model, provisioning an E2B sandbox for infra tasks. Real tool-calling is what makes cheap models work.
3. Escalation ladder: kimi-k2.7-code first ($0 marginal) → Sonnet only on failure. Reference-gate cheap verdicts before trusting an "unsolvable."
4. Model pick: kimi-k2.7-code (solved all 3 tasks, fast, efficient). glm-5.2 for max plan-efficiency; gpt-oss:120b for US-origin + speed. Drop nemotron (fails infra); Devstral isn't on Ollama.

08Caveats & sources

Harnesses: solve_bench_tools.py · pi_bench.py + Pi · 11 models · 3 tasks · 60+ solve runs · real E2B · real OpenRouter / Anthropic / Ollama charges