Engineering Doc

Task Audit —
running our skills in the cloud

We already have SKILL.md files that check whether a generated task is any good. Today someone runs them by hand, one task at a time. This pipeline runs the same skills automatically, in a throwaway cloud container, for every task in a position.

01What this is

Our tasks are generated by AI, so some of them are broken. The question might be too short, a link might be dead, or the task might simply be impossible to do. We only find out when a candidate complains.

We already know how to check this by hand: open a coding agent on your laptop, point it at a SKILL.md file, and let it go through the task. It works, but it takes about twenty minutes per task and nobody has time to do it for every task.

This pipeline is that same check, done by a machine. When a recruiter creates a position, every task in it is sent off to a small cloud container. Inside, an AI agent reads the task, tries to actually do it, and something independent checks whether the work is right. The team gets one Discord message saying which tasks are fine and which need fixing.

Before — by hand

Someone opens a coding agent on their laptop

Points it at the SKILL.md file

Waits ~20 minutes

Does this for one task, when they remember

Now — automatic

A position is created

The same SKILL.md runs inside a container

All tasks in the position, in one go

Results land in the database and Discord

02The flow

Four steps from "position created" to "results in Discord".

Start to finish — scroll to zoom, drag to move, or open it full screen
scroll to zoom · drag to move · double-click to zoom in
A position is created it has a list of tasks Our API is called POST /v2/tasks/audit skips tasks already checked Airflow is told to run the audit_task job it only starts things, no logic A container starts on AWS Fargate — a fresh machine, deleted afterwards INSIDE THE CONTAINER — it goes through the tasks one by one 1 · Check the details links work? question long enough? an AI reads our SKILL.md rules and looks for problems 2 · Set up the task download the starter code, start the same environment a candidate would get 3 · The agent does the task headless Pi reads the code and writes the fix, on its own (the thinking happens at Ollama) 4 · Check the work we check it ourselves — the agent never marks its own homework OLLAMA CLOUD the AI models the agent talks to over the internet — flat plan, so no per-task cost asks answers A row in the database pass or fail, what's wrong, and the fix the agent wrote Files in S3 the full record of what the agent did, if you need to dig in One Discord message per position — which tasks are fine, which ones need fixing

A bit more on each step

  • The API call. We send it the list of task IDs. Before doing anything expensive, it checks which tasks have already been checked and haven't changed since — those are skipped. If a task was edited (even just its README on GitHub), it gets checked again.
  • Airflow. Think of it as the scheduler. It doesn't do any of the checking itself — its only job is to start the container, watch it, show us the logs, and start it again if the machine crashes. We cap it at two containers at a time so we don't flood anything.
  • Fargate. AWS gives us a fresh machine, runs our container on it, and deletes it when the container exits. Nothing is left behind, so every run starts clean.

03Inside the container

The Dockerfile is just a list of what to install so the machine can do the job on its own. Five things matter:

# 1. Python, git and curl — the basics
FROM python:3.12-slim

# 2. Node, then the "pi" coding agent — this is the thing that will do the task
RUN install node ... && npm i -g pi-coding-agent

# 3. Tell pi where to get its AI from: Ollama Cloud, over the internet.
#    (On a laptop this points at localhost — there's no such thing in a container,
#     so we point it straight at ollama.com. This one line trips people up.)
RUN write /root/.pi/agent/models.json  →  baseUrl: https://ollama.com/v1

# 4. Our own code and our SKILL.md files
COPY runner.py  scripts/  rules/   /app/

# 5. When the container starts, run our script
ENTRYPOINT ["python", "runner.py"]

Notice what is not in there: no passwords, no API keys. Those are handed to the container by AWS when it starts. The image itself is safe to store.

The last line is the important one. Airflow doesn't send code to the container — the code is already inside. Airflow only passes the arguments: which tasks to check, and whether this is dev or prod.

The file that actually runs — runner.py

Think of it as the foreman. The Dockerfile builds the workshop and hires the workers; runner.py is the one holding the clipboard, walking the job from start to finish and writing down the result. It's the file named in that last line, so it starts when the container starts — and when it's done, the container dies with it.

It reads the arguments it was given (which tasks, dev or prod, where to save results), then works through the tasks one at a time. If one task blows up, the rest still get done.

For each task, in order:

  1. Check whether anything about the task has changed since last time. If not, reuse the old result and skip the rest.
  2. Run the plain checks — fields filled in, links working, the environment it points to actually exists. Ordinary code, no AI: these are facts, and an AI would just make them up.
  3. Send the task's content to an AI along with our SKILL.md rules, for the judgement calls — is the answer leaked in the starter code, does the question read badly.
  4. Decide where the work happens: a full cloud sandbox for tasks that need live services, or right inside the container for simpler ones.
  5. Run the task's tests on the untouched starter first. That shows whether there's genuinely something to solve, and how the work will be judged later.
  6. Start the Pi agent and let it attempt the task — trying a second model if the first genuinely fails, and handling the awkward cases (the agent did nothing at all, or it hung).
  7. Grade the result, after putting the original tests back in case the agent touched them.
  8. Walk the candidate's guided tour to check nothing is broken, then save the row, upload the files, and move on.

Once the whole batch is done, it posts the single Discord message and exits.

What it deliberately does not do is any of the specialist work — it only decides what happens next, and in what order:

JobWho does it
Talking to the sandboxcourier.py
Running the agentpi_solve.py
The plain field checkstask_audit.py
Saving to the database and S3persist.py
Writing the Discord messagenotify.py
A detail worth knowing Whether runner.py reports success is about us, not about the tasks. A task that fails its check is still a successful run — the container did its job and recorded a real answer. It only reports failure when something on our side broke, which is the signal for Airflow to try again. That's what stops a genuinely bad task from being retried forever.

04How it actually solves the task

This is the part people usually want to see. Our script starts the pi agent as a normal command, the same way you would in a terminal:

pi -p "<the task, written out>" --provider ollama --model kimi-k2.7-code --api-key ...

After that we don't drive it. Pi runs on its own in a loop:

  • It sends the task and the files it has read to Ollama Cloud.
  • The AI answers with something to do — read this file, change that line, run this command.
  • Pi does it, locally, in the copy of the code we downloaded.
  • It sends the result back and asks what's next. Repeat until it thinks it's done.

Two details worth mentioning:

  • The agent gets the same instructions a candidate gets — the actual task description, not a vague "go fix this". We also tell it to run things and look at the real output before changing anything, so we know its findings are real and not guessed.
  • We check the work, not the agent. When it finishes, our script takes over: it puts the original tests back (in case the agent touched them) and checks the result itself. If the task has no tests, a separate AI reads the change against what the task asked for and the answer we have on file. Either way, the agent doing the work never decides whether it passed.
On cost We use Ollama Cloud, which is a flat monthly plan — running a thousand tasks costs the same as running one. We only fall back to a paid model (Claude) if Ollama actually refuses to answer, and when that happens Discord gets an alert, because that's the moment things start costing money.

05How a SKILL.md file runs in the cloud

This is the whole point, so it's worth being clear about it.

A SKILL.md file is just instructions written in plain English — "check the title is properly written", "make sure the answer isn't leaked in the starter code", "the question should be between 120 and 1500 characters". When you run a skill on your laptop, your coding agent reads that file and follows it.

In the container it's the same idea, with one difference: instead of a person opening the agent, our script does it. The SKILL.md files are copied into the image (rules/ in the Dockerfile above), and at the right moment the script reads the file and hands it to an AI as its instructions — word for word, nothing rewritten.

On your laptop

You open the coding agent

You point it at SKILL.md

It follows the instructions on one task

You read the result on screen

In the container

The script starts automatically

It reads the same SKILL.md from rules/

It follows the instructions on every task

The result is saved and posted to Discord

Which skills go where

We have three SKILL.md files in the image, and they're used at different moments:

Skill fileWhen it runs
task-auditWhile checking the task's details — the field-by-field rules
task-repo-checkSame moment — the rules for the starter code and the gist
task-summaryAt the end — how to write the report a human will read
One thing to be clear about The agent that does the task never sees these files. It only gets the task description, the same as a candidate. The SKILL.md files go to the AI that checks things. Rules for the examiner, not the student — otherwise we'd be handing out the marking scheme.

Where these three came from

Upstream we don't have three separate skills like this. We have one bigger skill, task-verify, with task-solvability underneath it. That didn't get copied across as it was — it got split up, because different parts of it needed to become different kinds of thing:

Part of the original skillWhat it became here
The step-by-step procedure — download the code, run it, let an agent try, check the result Code. It's the eight steps under Inside the container (section 03) — that's runner.py
How to write up the result for a human A rule file — task-summary. Still a judgement call, so it stays as instructions for an AI
Its sandbox helper Code — courier.py

task-audit and task-repo-check were already skills in their own right, and they were copied in unchanged.

The rule of thumb behind the split: a fixed sequence of steps belongs in code; a judgement call belongs in a SKILL.md. Steps don't need an AI to follow them, and code can't make a judgement call.

06What you get back

Every task ends with a verdict. These are the exact values saved in the verdict column and shown in the Discord message:

VerdictWhat it means
PASSGood to give to a candidate.
PASS_WITH_FINDINGSIt works, but some details need tidying. The findings list says exactly what.
FAILThe agent genuinely tried and the task couldn't be done — or the guided walkthrough is broken.
INVALIDThe starter code already passes, so there's nothing to actually do.
ERROROur own machine broke. Not a verdict on the task — it just runs again.

Only PASS and PASS_WITH_FINDINGS are remembered. A task that failed always gets checked again next time, so a fix is never hidden behind an old result.

All of it lands in a task_audits row (kept forever, so you can see a task go from failing to fixed), the full record goes to S3, and the team gets one Discord message per position listing what needs attention and how to fix each thing.

07What's left to set up

The code is done and has been tested end to end on dev by running the container locally. What's missing is the AWS side — nothing has been created there yet.

  • Create the AWS pieces — somewhere to store the image, the container definition for dev and prod, and permission for our CI to push images.
  • Add the keys to that container definition: Ollama, Anthropic (the fallback), GitHub, E2B, AWS, Supabase, and the new Discord webhook.
  • Copy the database changes to prod. They're all applied on dev. One of them matters more than the rest: without it the container can't read the template list, and the check that catches broken tasks stays switched off.
  • Call the endpoint when a position is created.
  • Fix six tasks already live in prod that this audit found — they point at environments that don't exist, so they won't even start for a real candidate.

One thing to expect on the first run: everything gets checked once, because none of it has been checked before. After that, only tasks that actually changed are re-checked.

Utkrushta · Task Audit · PR #635