Utkrushta · Task Audit Pipeline
What happens from the moment a position is created to the moment every one of its tasks has a verdict, a trace, and a cache entry — one endpoint call, one DAG, one ephemeral container, two audit legs, one solve.
deterministic check LLM / interpretive E2B sandbox persisted artifact
A position is created (or an operator wants tasks verified). The backend calls one endpoint with the position's task ids — a batch, not one call per task.
For each id, the endpoint asks: "have we already audited this exact content?" It never trusts a timestamp
(the tasks table has none) — it fingerprints the content itself.
cached[] ·
No row / negative verdict / edited → queued[] ·
Id not in tasks table → unknown[] (rejected up-front, never queued)If everything was cached, the DAG is never triggered — the endpoint returns immediately with the cached verdicts.
Otherwise the queued ids are comma-joined into ONE DAG run (idempotent run_id, so retries can't double-fire).
The DAG is a thin orchestrator — it owns no audit logic. It launches one throwaway container running the worker image, passes the ids as the container command, streams the container's logs into the Airflow task log, and waits.
EcsRunTaskOperator → AWS Fargate task utkrusht-task-audit-{dev,prod}, deferrable + reattach, CloudWatch logsdocker run --rm utkrusht-task-audit:local via the mounted docker.sock — same image, same command, zero AWSBuilt grader-style: explicit COPYs only — never COPY . ., so secrets can't land in image layers.
All secrets arrive at runtime (-e env locally / AWS Secrets Manager in prod):
OLLAMA_API_KEY, OPENROUTER_API_KEY, E2B_API_KEY, GitHub tokens, Supabase keys, S3 creds.
For each task id (serial loop, crash-isolated per task), the worker re-checks the cache (safety net), then runs two complementary audits:
sbtest.sh) syncs every changed file INTO the sandbox and runs
pytest THERE — where the live services are. Local mode skips the courier: Pi runs the tests directly.
error, the failing-test tail, and its sampling metadata (temperature / resolved model) — so "passed yesterday, failed today" is answerable.The row just written contains the task's content_hash. The next time anyone triggers an audit for this task —
tomorrow, or for a different position sharing the same task — stage 2 recomputes the hash, finds this row, and answers in 2 seconds for $0.
Only a real content edit (new question, new answer, new criteria…) changes the hash and earns a fresh audit.
| task_audit.py — deterministic | llm_audit — interpretive | |
|---|---|---|
| Nature | Pure rules — counting, presence, format, ranges, duplicates, FK existence, live URL reachability | Judgment — prose quality, narrative shape, leaks, "is the starter appropriately incomplete?" |
| Same input → same output? | Always (that's the point) | Near-deterministic (temperature 0) but fundamentally a model's judgment |
| Where the rules live | In the code — each check is an explicit Python rule | In the SKILL.md files, passed verbatim as the system prompt — editing the skill updates the auditor, no code change |
| Input | The Supabase task row (+ reference sets from competencies/templates tables) | task row + README + starter source files + gist files fetched from GitHub |
| Cost / speed | ~free, <1 s (+2 GitHub API calls) | one LLM call (~15–20 s, ~$0 on the Ollama flat plan) |
| Example catch | pre_requisites: 6 items (max 3) | short_overview[1] opens with imperative "Harden" — instruction-stacking |
| Why both | Mechanical rules can't judge prose; an LLM shouldn't be trusted to count. Each does what the other can't — findings from both merge into one findings[]. | |
The tasks table has no updated_at column — so instead of trusting a timestamp, we fingerprint the content itself.
The hash is stored only on audit rows (the report), never on the task (the cake) — the current hash is recomputed fresh each time, which is free and can never go stale.
| Event | Fresh hash | Match in task_audits? | Action |
|---|---|---|---|
| First ever audit | eb2459… | none | audit → store row with eb2459… |
| Re-trigger, task untouched | eb2459… | ✓ matches (PASS-family) | cached — no container, no sandbox, no LLM |
| Someone edits the question | 55de77… | ✗ different | re-audit → store row with 55de77… |
| Task previously FAILED / UNVERIFIED | unchanged | ✗ negative verdicts never cached | re-audit — a repo-side fix always gets a fresh look |
Someone touches status / other non-content field | eb2459… (unchanged) | ✓ matches | cached — incidental touches never waste an audit |
| Layer | What's captured | Where it lives |
|---|---|---|
| Endpoint | cache-gate decision per batch (requested / cached / queued / unknown counts) | structured app logs |
| Airflow | the exact container command (secrets excluded — value-less -e passthrough) + the container's full stdout, streamed live | Airflow task log (CloudWatch in prod) |
| Worker phases | [1/2] AUDIT findings count → [2/2] mode + baseline → per-model attempt: passed?, tool calls, the failing-test tail (the exact RED reason), errors (pi exit codes, timeouts, provider errors) | container stderr → Airflow log |
| Verdict | verdict + findings + escalation ladder + failure_reason + baseline + solve_mode + sampling (temperature / resolved model) + cost + duration + content_hash | task_audits row (queryable) |
| The recording | full Pi transcript per attempt (winner AND failed attempts) — every thinking block, tool call, test output; plus the solution diff | S3 trace*.json (trace_url on the row) + detail.solution_diff |
| Scenario | Wall time | Cost | Outcome |
|---|---|---|---|
| Cache hit (unchanged task) | 2.3 s | $0 | prior verdict returned, no compute |
| Full audit, kimi solves (streamflix infra task) | ~2.5 min | $0 marginal (Ollama flat) | PASS_WITH_FINDINGS · 17–27 tool calls · trace + diff persisted |
| Full audit, escalates to Sonnet | ~6 min | $0.31 | PASS_WITH_FINDINGS · 33 calls · reference gate did its job |
| Broken task (no tests) — BEFORE the baseline check | ~17 min | $1.09 | FAIL after burning the full ladder — the motivation for the UNVERIFIED short-circuit |
| Same broken task — AFTER the baseline check | ~2 min | $0 | UNVERIFIED — baseline detected "no tests" and skipped all three models entirely |
| Non-infra task (LocalExec, no sandbox) | ~4 min | $0 marginal | PASS_WITH_FINDINGS · solved by kimi in-container — unlocks the 341 template-less tasks (69% of catalog) |
| Mixed batch (cached + unknown + new) | — | — | cached answered instantly · bogus id rejected as unknown · only the new id triggered the DAG |
Stack: FastAPI (cache-gated endpoint + DAOs) · Airflow audit_task DAG (EcsRunTaskOperator / local docker) · Dockerfile.verifier (python + node + pi, 1.16 GB) · runner.py (task_audit.py + llm_audit + baseline + Pi solve, E2B or local) · task_audits (Supabase) + S3 traces · 19 unit tests incl. cross-repo hash-parity lock. All timings/costs measured on real dev runs, 2026-07-10.