Engineering Doc

Task Audit —
running our skills in the cloud

We already have SKILL.md files that check whether a generated task is any good. Today someone runs them by hand, one task at a time. This pipeline runs the same skills automatically, in a throwaway cloud container, for every task in a position.

01What this is

Our tasks are generated by AI, so some of them are broken. The question might be too short, a link might be dead, or the task might simply be impossible to do. We only find out when a candidate complains.

We already know how to check this by hand: open a coding agent on your laptop, point it at a SKILL.md file, and let it go through the task. It works, but it takes about twenty minutes per task and nobody has time to do it for every task.

This pipeline is that same check, done by a machine. When a recruiter creates a position, every task in it is sent off to a small cloud container. Inside, an AI agent reads the task, tries to actually do it, and something independent checks whether the work is right. The team gets one Discord message saying which tasks are fine and which need fixing.

Before — by hand

Someone opens a coding agent on their laptop

Points it at the SKILL.md file

Waits ~20 minutes

Does this for one task, when they remember

Now — automatic

A position is created

The same SKILL.md runs inside a container

All tasks in the position, in one go

Results land in the database and Discord

02The flow

Four steps from "position created" to "results in Discord".

Start to finish — scroll to zoom, drag to move, or open it full screen
scroll to zoom · drag to move · double-click to zoom in
A position is created it has a list of tasks Our API is called POST /v2/tasks/audit skips tasks already checked Airflow is told to run the audit_task job it only starts things, no logic A container starts on AWS Fargate — a fresh machine, deleted afterwards INSIDE THE CONTAINER — it goes through the tasks one by one 1 · Check the details links work? question long enough? an AI reads our SKILL.md rules and looks for problems 2 · Set up the task download the starter code, start the same environment a candidate would get 3 · The agent does the task headless Pi reads the code and writes the fix, on its own (the thinking happens at Ollama) 4 · Check the work we check it ourselves — the agent never marks its own homework OLLAMA CLOUD the AI models the agent talks to over the internet — flat plan, so no per-task cost asks answers A row in the database pass or fail, what's wrong, and the fix the agent wrote Files in S3 the full record of what the agent did, if you need to dig in One Discord message per position — which tasks are fine, which ones need fixing

A bit more on each step

  • The API call. We send it the list of task IDs. Before doing anything expensive, it checks which tasks have already been checked and haven't changed since — those are skipped. If a task was edited (even just its README on GitHub), it gets checked again.
  • Airflow. Think of it as the scheduler. It doesn't do any of the checking itself — its only job is to start the container, watch it, show us the logs, and start it again if the machine crashes. We cap it at two containers at a time so we don't flood anything.
  • Fargate. AWS gives us a fresh machine, runs our container on it, and deletes it when the container exits. Nothing is left behind, so every run starts clean.

03Inside the container

The Dockerfile is just a list of what to install so the machine can do the job on its own. Five things matter:

# 1. Python, git and curl — the basics
FROM python:3.12-slim

# 2. Node, then the "pi" coding agent — this is the thing that will do the task
RUN install node ... && npm i -g pi-coding-agent

# 3. Tell pi where to get its AI from: Ollama Cloud, over the internet.
#    (On a laptop this points at localhost — there's no such thing in a container,
#     so we point it straight at ollama.com. This one line trips people up.)
RUN write /root/.pi/agent/models.json  →  baseUrl: https://ollama.com/v1

# 4. Our own code and our SKILL.md files
COPY runner.py  scripts/  rules/   /app/

# 5. When the container starts, run our script
ENTRYPOINT ["python", "runner.py"]

Notice what is not in there: no passwords, no API keys. Those are handed to the container by AWS when it starts. The image itself is safe to store.

The last line is the important one. Airflow doesn't send code to the container — the code is already inside. Airflow only passes the arguments: which tasks to check, and whether this is dev or prod.

04How it actually solves the task

This is the part people usually want to see. Our script starts the pi agent as a normal command, the same way you would in a terminal:

pi -p "<the task, written out>" --provider ollama --model kimi-k2.7-code --api-key ...

After that we don't drive it. Pi runs on its own in a loop:

  • It sends the task and the files it has read to Ollama Cloud.
  • The AI answers with something to do — read this file, change that line, run this command.
  • Pi does it, locally, in the copy of the code we downloaded.
  • It sends the result back and asks what's next. Repeat until it thinks it's done.

Two details worth mentioning:

  • The agent gets the same instructions a candidate gets — the actual task description, not a vague "go fix this". We also tell it to run things and look at the real output before changing anything, so we know its findings are real and not guessed.
  • We check the work, not the agent. When it finishes, our script takes over: it puts the original tests back (in case the agent touched them) and checks the result itself. If the task has no tests, a separate AI reads the change against what the task asked for and the answer we have on file. Either way, the agent doing the work never decides whether it passed.
On cost We use Ollama Cloud, which is a flat monthly plan — running a thousand tasks costs the same as running one. We only fall back to a paid model (Claude) if Ollama actually refuses to answer, and when that happens Discord gets an alert, because that's the moment things start costing money.

05How a SKILL.md file runs in the cloud

This is the whole point, so it's worth being clear about it.

A SKILL.md file is just instructions written in plain English — "check the title is properly written", "make sure the answer isn't leaked in the starter code", "the question should be between 120 and 1500 characters". When you run a skill on your laptop, your coding agent reads that file and follows it.

In the container it's the same idea, with one difference: instead of a person opening the agent, our script does it. The SKILL.md files are copied into the image (rules/ in the Dockerfile above), and at the right moment the script reads the file and hands it to an AI as its instructions — word for word, nothing rewritten.

On your laptop

You open the coding agent

You point it at SKILL.md

It follows the instructions on one task

You read the result on screen

In the container

The script starts automatically

It reads the same SKILL.md from rules/

It follows the instructions on every task

The result is saved and posted to Discord

Which skills go where

We have three SKILL.md files in the image, and they're used at different moments:

Skill fileWhen it runs
task-auditWhile checking the task's details — the field-by-field rules
task-repo-checkSame moment — the rules for the starter code and the gist
task-summaryAt the end — how to write the report a human will read
One thing to be clear about The agent that does the task never sees these files. It only gets the task description, the same as a candidate. The SKILL.md files go to the AI that checks things. Rules for the examiner, not the student — otherwise we'd be handing out the marking scheme.

There was also a fourth skill — the one describing how to check whether a task can be solved at all. That one wasn't about judgement, it was a series of steps: download the code, run the tests, let an agent try, check the result. Steps like that are better as code, so it became runner.py rather than a file we hand to an AI.

06What you get back

Every task ends with one of these:

ResultWhat it means
PassGood to give to a candidate.
Pass, with notesIt works, but the details need tidying. The notes say exactly what.
FailThe agent genuinely tried and the task couldn't be done — or the guided walkthrough is broken.
Couldn't checkWe had no way to judge the work. That's our gap, not the task's fault.
Nothing to doThe starter code already passes. The task needs to actually be incomplete.
ErrorOur own machine broke. Not a verdict on the task — it just runs again.

All of it lands in a task_audits row (kept forever, so you can see a task go from failing to fixed), the full record goes to S3, and the team gets one Discord message per position listing what needs attention and how to fix each thing.

07What's left to set up

The code is done and has been tested end to end on dev by running the container locally. What's missing is the AWS side — nothing has been created there yet.

  • Create the AWS pieces — somewhere to store the image, the container definition for dev and prod, and permission for our CI to push images.
  • Add the keys to that container definition: Ollama, Anthropic (the fallback), GitHub, E2B, AWS, Supabase, and the new Discord webhook.
  • Copy the database changes to prod. They're all applied on dev. One of them matters more than the rest: without it the container can't read the template list, and the check that catches broken tasks stays switched off.
  • Call the endpoint when a position is created.
  • Fix six tasks already live in prod that this audit found — they point at environments that don't exist, so they won't even start for a real candidate.

One thing to expect on the first run: everything gets checked once, because none of it has been checked before. After that, only tasks that actually changed are re-checked.

Utkrushta · Task Audit · PR #635