UtkrushtDesign Document

Design-Review Whiteboard Task Analysis

How the platform analyzes a candidate's system-design whiteboard session — what we built, how the flow works, exactly what the AI grader receives, and the prompts it runs.

TL;DR

A DESIGN_REVIEW task gives the candidate an Excalidraw whiteboard with a system-design scenario: fixed Functional Requirements and a pre-drawn, deliberately flawed starter design. They redesign the architecture and fill in answer sections (Non-Functional Requirements, Core Entities, API Routes, …) while their screen is recorded.

The analysis pipeline now understands this task type natively. It fetches the candidate's board, converts it into a readable text summary, and sends exactly four things to Gemini: the video, the task description, the competencies, and the board summary. Nothing else — no code diff, no README, no answer key, no PR comments. A dedicated design-review prompt package grades design-decision quality against the given requirements, producing the same result shape as every other task type, so reports and leaderboards needed zero changes.

Why this exists

Every other task type produces code as its work product — a git diff, a ZIP of files, or PR review comments. A whiteboard task produces none of those. Its work product is a drawing plus text: boxes, arrows, and filled-in answer sections stored as an Excalidraw scene in the database.

So the pipeline needed two new abilities:

Read the board — turn the scene JSON into text a grader can use, while cleanly separating what we gave the candidate from what they produced.
Grade design thinking, not code — a prompt set where the persona is a senior architect judging component breakdown, data models, API contracts, trade-off reasoning, and failure handling.

The candidate flow (context for the analysis)

Start — POST /v2/sessions/start detects a DESIGN_REVIEW task and short-circuits the entire sandbox path: no E2B, no repo, no template. It returns an already-active session whose single interaction surface is the design-studio whiteboard URL (DESIGN_STUDIO_BASE/design/{session}/{task} — the server fails loudly if that env var is unset rather than hand out a broken link).
Work — the whiteboard app autosaves the scene through PUT /v2/task-sessions/{id}/whiteboard. On every save the server re-asserts the starter locks: any locked (given) element a tampered client deleted is restored, and any it unlocked is re-locked. This protects the analysis contract below. GET .../whiteboard serves the saved scene, falling back to the task's starter scene on first open.
Submit — the completion endpoint triggers the analysis DAG, same as every other task type.

The analysis flow, step by step

submit ──► tasksession_completion DAG (or Fargate runner)
              │
              ├─ 1. list + download screenshare chunks from S3
              ├─ 2. ffmpeg-stitch into one combined MP4, validate, upload
              ├─ 3. work-product fetchers run (each self-gates on task type):
              │       github diff ──► None      (not a GITHUB task)
              │       zip text    ──► None      (not a ZIP task)
              │       PR comments ──► None      (not a PR_REVIEW task)
              │       whiteboard  ──► BOARD SUMMARY   ◄── the one that fires
              ├─ 4. variant selection: DESIGN_REVIEW ► design_review prompts
              ├─ 5. Gemini: video cached once, multi-turn analysis
              │       ≤ 20 min video ► single pass (all steps + overview)
              │       > 20 min video ► map per 20-min segment, then reduce
              └─ 6. result_analysis written ► score ► ANALYSIS_DONE ► cert,
                    leaderboard, notification, mixed-session finalizer

The board summary rides the pipeline's existing single "work product" slot (the variable other task types use for the code diff) — for a whiteboard task that slot would otherwise be empty, so no new plumbing was needed through the four layers between the runner and the prompt.

Exactly what the grader receives

#	Input	Source	Why
1	Screen recording (combined MP4)	S3 chunks → ffmpeg	Primary evidence — how they explored the flaw and reworked the board over time
2	Task description	`tasks.task_blob`	The scenario + the fixed Functional Requirements they designed against
3	Competencies	`tasks.criterias`	The weighted skills each rating must be broken down by
4	Board summary	`task_submissions[task_id].whiteboard_scene` → walker	The candidate's actual output, in text form

Deliberately NOT sent (each was removed or never wired in):

❌ solutions (answer key) — grading is against the given Functional Requirements only; prompts were rewritten so they never reference a key the model doesn't have.
❌ readme_content — repo audit context; meaningless for a board task.
❌ review_comments — PR-review-only input; its fetcher self-gates and the design prompt never renders the field.
❌ Any code — the work-product slot is board-only by contract (see Separation guarantees).

The board summary (the scene → text walker)

airflow/dags/task_analysis_runner/whiteboard.py::summarize_scene() walks the Excalidraw scene. The locked flag is the contract, set by the board builder at generation time:

locked: true → given content: the Functional Requirements section and the flawed starter design. Provided to the candidate — not their work.
locked: false → candidate work: the answer-zone sections they fill in and any shapes they draw in the design area.

Stable element-id conventions (sec-<zone>-*, node-*, edge-*, starter-*, design-area-*) let the walker identify template scaffolding; candidate-drawn shapes carry Excalidraw-generated ids with no known prefix. Arrows are resolved into readable connections via their endpoint bindings. Output shape:

# Candidate's whiteboard submission

## Candidate's redesign (High-Level Design area)
Components the candidate added:
- API Gateway
- Ingestion Queue
Connections the candidate drew:
- API Gateway -> Ingestion Queue
- Ingestion Queue -> Worker Pool

## Answer sections (candidate-written)
- Non-Functional Requirements: 99.9% availability, p99 < 200ms …
- Core Entities: (unfilled)

## Given context (provided to the candidate — NOT their work)
Functional Requirements (given): …
Current design components (given, locked): API; DB
Known flaw in the current design: single write path saturates the DB

Malformed or empty input degrades to a short marker string — the walker never raises. The video remains the primary signal; the summary is structured corroboration.

Separation guarantees (design review touches nothing else)

Every design hook self-gates. The board fetcher returns None for any non-DESIGN_REVIEW task, so GitHub/ZIP/PR analyses are byte-for-byte unchanged (verified by sentinel render tests: their contexts still contain solutions/README/diff/comments).
Board-only work product. For a confirmed design task the fetcher always returns a board-derived string — even with no scene at all it returns the (no whiteboard submission) marker, never None — so the slot can never fall back to github/zip content.
Dual-typing is impossible. A task carrying both PR_REVIEW and DESIGN_REVIEW is rejected at the session-creation door with HTTP 400 CONFLICTING_TASK_TYPES (the array field has no DB constraint, so the route enforces it). A defense-in-depth guard in both fetchers additionally resolves any legacy dual-typed row to the PR pipeline with a warning — one clean pipeline, never a mix.
No infrastructure. The /start short-circuit means a design task never provisions a sandbox or repo; there is nothing to leak from.
Scoring version. Design review runs on scoring v1. The v2 registry short-circuits before the task-type check and has no design catalog yet — design-review positions must keep positions.scoring_version = 'v1'.

The prompts

The full texts live in airflow/dags/video_analysis/prompts/v1_design_review/ (each with version + changelog metadata). Below are the load-bearing parts. Every map prompt ends with the same shared timestamped-sentences boilerplate (exact-excerpt quotes, mm:ss format, video-verified, no overlaps) — shown once at the end rather than repeated.

Base context (sent first, with the cached video)

# Assessment Context
You are a wise and fair senior technical architect and expert talent evaluator
with experience from 500+ candidate interviews. You are evaluating a candidate's
video-based **system-design whiteboard assessment**.

In this assessment, the candidate was given a system-design scenario with fixed
Functional Requirements and a pre-drawn current design (often deliberately
flawed or incomplete). Working on an Excalidraw whiteboard, they redesign or
extend the architecture and fill in the answer sections (Non-Functional
Requirements, Core Entities, API Routes, ...). You are evaluating the QUALITY
of their design and their design reasoning.

## Task Description:
{question_blob}

## Competencies to check for:
{competency_array}

## Candidate's Whiteboard Submission (board summary — components, connections,
and answer-section text they produced; the given/locked content is separated
out as context, not their work):
{board summary — or "(No whiteboard submission available)"}

## Instructions:
Analyze the candidate's screen recording video together with the board summary
above to evaluate their design performance. Focus on:
- Design-decision quality against the given Functional Requirements (component
  breakdown, data model, API contract, trade-off reasoning, failure handling)
- How the candidate explored the given/flawed current design and reworked the
  board over time
- The clarity and completeness of the answer sections they filled in
Judge the candidate on their system-design skills and reasoning, not the tools
they use.

Please watch the entire video carefully and base your analysis on what you
observe in the recording together with the board summary.

## IMPORTANT - Video Duration Confirmation:
Before you begin your analysis, please confirm: What is the total duration
(in minutes and seconds) of the video you received?

`skill` — per-competency proficiency (v1.3.0)

We want to evaluate any evidence of skill and proficiency from the candidate's
system-design work shown in the video and on the design board. Evaluate each
competency -- the evidence of skill against the proficiency level expected for
each skill required. The CHOICE of whether to use AI is not relevant. HOW they
direct AI IS evidence of skill.

Grade design-decision quality against the task's GIVEN functional requirements.
The signals of a good design are: sound component breakdown, an appropriate
data model, a coherent API contract, explicit trade-off reasoning, and failure
handling.

# HOW TO IDENTIFY SKILL AND PROFICIENCY IN SYSTEM DESIGN:
- How well they decompose the system into components ...
- How appropriate their data model is (fits access patterns and scale targets)
- How coherent their API contract is (interfaces actually carry the required
  data flows)
- The quality of their trade-off reasoning (queue vs direct call, SQL vs NoSQL,
  sync vs async — justified against the requirements)
- How they handle failure and scale (bottlenecks, single points of failure,
  stated scale targets)
- Directing/steering AI conversations about the design IS skill evidence;
  recognizing an off-track AI suggestion is a proficiency signal
- Can they identify the root cause of the flaw in the starter design quickly?
- Verification as skill evidence: sanity-checking decisions against the given
  requirements; knowing WHICH parts of the design to scrutinize
- The specificity of reasoning written into the answer zones reflects
  understanding

# CRITICAL -- Per-Skill Isolation in Reasoning:
While reasoning about a particular skill, do NOT mention points about another
skill. Each competency's reasoning must be self-contained. [+ per-competency
0-5 rating, gap description, and 1-3 timestamped sentences each]

`problem_solving` — design process over time (v1.0.0)

Synthesizing your previous analyses (overview, AI usage, skills), provide a
problem-solving summary that ties together the patterns you observed in the
design session.

Use the video timeline: this task is about how the candidate EXPLORED the
flawed/starter design and REWORKED the board over time, not just the final
board state.

Provide a 3-line summary of their design approach:
- Understanding the problem: reading the brief and given functional
  requirements, understanding the flawed starter design and WHY it is flawed
- Exploring and reworking: how they diagnosed the starter design's weaknesses,
  iterated on the board, and converged on a redesign over the video
- Producing a sound design: quality of the final component breakdown, data
  model, API contract, trade-off reasoning, and failure handling

### Design Completeness (IMPORTANT):
If requirements went unaddressed, differentiate between:
1. Ran out of time — still actively designing when the video ended
2. Skipped — committed to a final board leaving requirements unaddressed

`overview` — 5-dimension overview (v1.1.0)

Based on the video recording, provide a comprehensive overview analysis of the
candidate's system-design performance. The candidate's work product is the
design board together with the recording of them building it. Evaluate against
the task's given functional requirements.

### Requirement Coverage — how many given requirements the design satisfies;
    what critical ones were missed; precise vs hand-wavy coverage
### Design Depth — superficial ("add a database") vs reasoned ("a partitioned
    KV store keyed by user id because reads are point-lookups at this scale");
    trade-offs explained; failure/bottleneck/scale addressed concretely
### Architectural Soundness — do the components, data model and API contract
    actually carry the given requirements; components that map to no
    requirement; soundness for the stated workload, scale, failure expectations
### Diagnosis & Verdict Correctness — did they correctly identify the flaw in
    the starter design and rework it appropriately
### Approach to the Design — systematic (brief → starter design → answer
    zones) vs scattered; core decisions prioritized over cosmetics

`ai_usage` — how they direct AI (v1.0.0, core)

AI-assisted work is the norm and expected. We evaluate how effectively the
candidate leverages AI to enhance their design work -- NOT whether they use AI.

## Prompt Quality:
High: detailed, specific prompts showing understanding of the system under
design — "Given a write-heavy ingestion path, should I put a queue between the
API and the workers, and how does that affect delivery guarantees?"
Low: generic prompts pasted without context; accepting AI output wholesale
without evaluating it against the requirements.

`insights`, `highlights`, `chapters` (v1.0.0, purpose)

insights — 4–10 atomic, timestamp-backed behavioral observations, each tagged strength / concern / notable; only what was observed, never a checklist of what's missing.
highlights — the extraordinary moments only, positive or negative ("immediately diagnosed the ingestion bottleneck and explained why a queue was needed"; "identified a subtle single-point-of-failure and reworked the board to add redundancy").
chapters — a continuous 1–3-minute-per-chapter narrative of the whole session, readable as a short story of the design work.

`synthesis` (reduce phase, > 20-minute videos)

One reduce prompt per map dimension (synthesis_skill, synthesis_problem_solving, synthesis_ai_usage, synthesis_insights, synthesis_chapters, synthesis_highlights) merges the per-segment results into one, plus synthesis_recommendation for the final hire-signal narrative. Same schemas as the pr_review variant — which is what keeps result_analysis identical across task types.

Shared timestamp boilerplate (appended to every map prompt)

Timestamped sentences must reference specific moments in the video:
1. Exact Text Match — verbatim, continuous excerpts from your reasoning
2. No Overlaps between sentences
3. Video Evidence Required — omit any timestamp you cannot verify
4. Coverage — each sentence covers ~80% of the content it references
5. Format — always "mm:ss"
6. Validation — range within video duration; content actually demonstrates
   the sentence; text is a verbatim copy

What still needs to be done

Real end-to-end run — a live candidate session with an actual autosaved board and screenshare, submitted through the local stack (backend + local Airflow are prepared for this; watch the run's logs for Whiteboard summary built … and Using design_review analysis pipeline).
Generation-side validation — the design-studio/task-generation repo should also refuse to author a task typed with more than one flow marker (this backend already rejects it at session creation).
v2 scoring catalog — has no design-review dimension; design positions stay on scoring_version='v1' until one is added deliberately.
Optional: a unit test pinning the CONFLICTING_TASK_TYPES rejection in CI.