A two-sided, multimodal system that automatically runs a candidate through an AI-conducted interview, evaluates them, and moves them through stages — serving both candidates practicing and companies screening. Builds directly on the voice stack, multimodal fusion, and LLM-serving work in the companion docs.
An AI interview platform is three systems glued by a state machine: (1) a
real-time multimodal session (conduct the interview), (2) an evaluation
engine (score it fairly and defensibly), and (3) a content engine
(generate/adapt the questions). The "automatically moves you through" part is a
pipeline/state machine gating stage→stage. The model calls are the easy 20%;
the hard parts are fair evaluation, integrity, and the live
multimodal capture.
Hard problem 1 — Fair, defensible evaluation. An LLM judge is noisy, biased, and gameable. Turning a transcript + signals into a score a company can hire on (and a candidate can trust) is the core difficulty — rubrics, calibration, multi-judge, human-in-the-loop.
Hard problem 2 — The live multimodal session. Voice + code/screen + maybe video, on an event-time timeline, low latency — this is the entire voice-stack + fusion problem (companion docs), now with an evaluator listening.
Hard problem 3 — Integrity. Two-sided raises the stakes: candidates will try to cheat (LLM assistance, coached answers); companies need auditable, bias-controlled results. Anti-cheat + fairness are product-critical, not afterthoughts.
What it reuses from the companion docs
The live session is the voice stack + event-time fusion.
The evaluator + content engine are LLM-serving (inference server / router). The pipeline
is a state machine over a persistent session record. The scale story is the
resource scheduler. This doc is mostly composition of pieces already designed.
1 · System architecture
blue = candidate-side · green = company-side / serving · purple = the orchestrator (state machine) · teal = stores
The shape: a gateway (auth/RBAC — candidate vs company) → an
orchestrator that runs the interview state machine and drives three engines
(session, eval, content) → all sharing one LLM-serving
layer and writing one session record. Same three-tier instinct as the
LLM router: stateless control plane (orchestrator) over stateful engines/stores.
2 · The interview as a state machine — the "moves you through" core
"Automatically moves you through" = a state machine per session. Each stage
has an entry gate, a live phase, and an evaluation gate that decides
advance / branch / end. The orchestrator owns it; it's the spine of the product.
purple = live interview stages · green = terminal gate · orange = the adaptive loop-back
Each stage is {entry gate, live phase, eval gate}. Entry checks prerequisites (consent, prior score); live phase runs the session engine; eval gate scores and decides the transition.
The gate decides: advance / branch / loop-back / end. Weak-but-ambiguous signal → an adaptive probe (deepen) rather than a pass/fail — the thing a good human interviewer does and a fixed script can't.
Durable + resumable. The state machine persists per-session (candidate drops connection → resume mid-stage). Same "don't finalize in-flight" discipline as the crawler: a stage isn't committed until its eval gate fires.
Config-driven, not hard-coded. A company defines the stage graph (rounds, gates, thresholds); the orchestrator executes it. That's the product's flexibility surface.
3 · The live session — multimodal capture (reuses the voice stack)
The live phase of each stage is a real-time multimodal session — this is the
voice-stack + fusion problem, with an evaluator subscribed to the timeline.
Transport: WebRTC for browser audio/video, WebSocket for code/screen + control. (Voice-stack §2.)
Capture streams: audio (STT + prosody), video (presence, but be careful — see fairness), code/screen editor events, typed chat. Each is a specialist emitting annotations.
Event-time fusion: merge streams on event-time (audio ~80ms, vision ~200ms, code events ~0) so "they said X while typing Y" is captured correctly. (Fusion §9.)
The AI interviewer is the voice loop (STT→LLM→TTS) with turn-taking + barge-in — but its LLM is prompted as an interviewer (ask, follow up, hint-budget, stay on rubric), not a general assistant.
The evaluator subscribes to the same fused timeline — it doesn't re-listen, it reads the committed enriched transcript (words + prosody + code state), so eval and conduct share one source of truth.
The interviewer/evaluator split — keep them separate
Two different LLM roles on one session: the interviewer (real-time,
latency-critical, drives the conversation) and the evaluator (can be async,
deliberate, multi-pass, sees the whole transcript). Don't fuse them — the
interviewer optimizes for a natural low-latency exchange; the evaluator optimizes
for a fair, defensible score. Different prompts, models, latency budgets, even.
4 · Evaluation — the LLM-as-judge core — the hard problem
An LLM judge is noisy, position-biased, verbosity-biased, and gameable. A raw
"score this answer 1–10" is not hire-grade. The engine's job is to make the
judgment reliable, evidence-linked, and calibrated.
The techniques that make it defensible
Technique
What it fixes
Rubric-anchored scoring
score against explicit per-dimension criteria with worked anchors ("a 3 looks like…"), not a vibe. Cuts variance hugely.
Evidence-linked verdicts
every score must cite transcript spans ("claimed X at 04:12"). No score without evidence → auditable + harder to hallucinate.
Multi-judge + aggregate
N independent judges (diverse prompts/models); take median / require agreement. Adversarial verify for high-stakes — the review-workflow pattern.
Structured output
force a schema (dimension → score → evidence → confidence), not prose — parse-safe, comparable across candidates.
Decouple dimensions
score correctness, communication, depth separately — don't let a well-spoken wrong answer score high (or vice-versa).
Human-in-the-loop for the margin
auto-advance the clear pass/fail; route the ambiguous band + all final decisions to a human. AI triages, humans decide the close calls.
The framing that matters
The AI's output is evidence for a decision, not the decision — especially
company-side, where a wrong reject is costly and possibly unlawful. Position the eval
engine as "structured, evidence-linked assessment that a human ratifies," not
"the AI decides who gets hired." That's both better product and better risk posture.
5 · Content — adaptive question generation
Bank + generation hybrid. A curated, calibrated question bank (known difficulty, vetted rubrics) is the backbone; LLM generation fills gaps and personalizes — but generated questions are riskier (unknown difficulty, possible leakage) so gate them.
Adaptive difficulty. The state machine's probe stage picks the next question from current signal — deepen where weak, move on where strong. This is the same "adaptive threshold" idea as the endpointing knob: match challenge to the candidate.
Anti-leakage / freshness. Public banks get memorized (and now LLM-trained-on). Rotate, parameterize (same shape, different numbers), and track exposure per question — a freshness problem like the crawler's dedup.
Rubric co-generation. A generated question is useless without a rubric — generate them together, and have the eval engine sanity-check the rubric against sample answers before the question goes live.
6 · Two-sided — candidate vs company, one engine
Same session/eval/content engines; the two sides differ in what they get out
and what they configure.
The flywheel (why two-sided is strategically strong)
Candidate practice sessions generate labeled data (questions, answers,
outcomes) that calibrates the company-side evaluator; company rubrics define
what candidates should practice. Each side improves the other — the same
engine, two funnels, a compounding data advantage. (Mirrors the model-improvement
loop, but for interview signal.)
7 · Rubrics, scoring & calibration
A rubric is the contract. Dimensions + weights + per-level anchors. It's what makes two candidates comparable and a score explainable. Company-configurable, versioned (a score is tied to the rubric version that produced it).
Calibration against ground truth. Periodically score sessions that also have human/known outcomes; measure the judge's agreement and drift. This is a continuous eval problem — the judge is a model you must monitor like any other.
Normalize across questions. A "3" on a hard question ≠ a "3" on an easy one. Track per-question difficulty (from the candidate pool's performance) and normalize — item-response-theory territory.
Confidence + abstention. The judge should be able to say "insufficient signal" → triggers a probe or human review, rather than guessing a number.
8 · Integrity, fairness & anti-cheat — product-critical, not afterthought
Anti-cheat (candidate gaming the eval)
The obvious threat: LLM assistance. Candidate reads answers off another model. Mitigations: live spoken follow-ups ("walk me through line 12"), latency/behavioral signals (long pauses then fluent paste), screen/focus signals, and questions that require reasoning aloud not just a final answer.
Coached/memorized answers → adaptive probing (the state machine's job) surfaces shallow understanding; parameterized questions defeat memorization.
Identity → verification at intake for high-stakes company use.
Honesty: perfect anti-cheat is impossible; aim to raise the cost and flag for human review, not to "catch" deterministically. Over-aggressive detection false-positives real candidates — a fairness harm.
Fairness (the eval not harming candidates)
Bias in LLM judges is real and measurable — accent, name, verbosity, dialect, non-native phrasing. Test for it: run matched transcripts varying only the protected signal, measure score deltas.
Prefer content over delivery signals. Video/appearance and even accent-heavy prosody are fairness landmines — weight what was said over how it sounded/looked. Consider audio-only or transcript-only scoring for the graded path.
Auditability + candidate recourse. Every score is evidence-linked and reviewable; a candidate can contest, a company can inspect. Regulated hiring (EEOC, EU AI Act "high-risk") demands this — build it in, don't bolt it on.
This is where the product lives or dies
A coding platform is a commodity; a fair, defensible, auditable evaluation is
the hard, valuable part. The integrity/fairness layer isn't compliance overhead —
it's the moat and the thing that makes companies trust the score.
9 · Data model & the session record
One session record is the source of truth (same instinct as the voice-stack
session store — transport-agnostic, event-time, durable):
Append-only + event-time → replayable and auditable (compliance, contest, calibration all need exact reconstruction).
Full-fidelity vs the LLM's view: store everything; the evaluator/interviewer consume a compacted view (same two-representations point as the voice-stack context section).
Scores are tied to rubric_version + judge_id — so you can re-score, audit drift, and defend a decision after the fact.
10 · Scale, cost & the scheduler
Two workload shapes (the inference-vs-training split, again): the live session is latency-critical realtime (GPU-pinned, like a voice call); the evaluation is throughput batch (can queue, run deliberate multi-pass). Serve them at different tiers.
Model tiering (the router): the interviewer needs a fast model (low TTFT for natural turns); the evaluator can use a bigger, slower model (quality over latency). Content gen is offline/cached. Route by role.
Cost driver: a live session pins realtime capacity for its whole duration; evaluation is amortizable. Batch evals, cache content, reserve realtime capacity for concurrent live sessions. (KV-cache budget admission — the §infer scheduler.)
Self-host vs API per role: interviewer STT/TTS self-hostable (voice-stack economics); frontier judge likely API for quality. Same dial as the voice stack.
11 · Build order — thin slice first
Text mock interview, one stage, candidate-side. Typed Q&A + LLM interviewer + rubric-anchored eval + feedback. No voice, no state machine, no company side. Proves the eval core (the hard part) cheaply.
The state machine. Multi-stage with gates/advance, durable + resumable. Now it "moves you through."
Adaptive content. Bank + probe stage picking next question by signal.
Company side. Rubric config, results/replay console, human-in-the-loop ratification, audit trail.
Calibration + fairness harness. Ground-truth agreement, bias testing, drift monitoring — before any real hiring use.
Voice. Add the live spoken session (voice stack) — biggest lift, so last; text validated everything else first.
The evaluation engine is the hard, valuable part and it's fully testable in
text. Voice is the biggest engineering lift and adds latency/turn-taking complexity
that's orthogonal to whether your scoring is fair and calibrated. Prove the
score, then make it spoken. (The opposite order — voice first — is the classic trap:
a slick spoken demo over a scoring engine you can't defend.)
12 · Risks & open questions
Legal/regulatory (biggest): automated hiring is "high-risk" under the EU AI Act and scrutinized by EEOC/NYC Local Law 144 (bias audits mandatory). Company-side eval may legally require human ratification + published bias audits. This shapes the architecture, not just policy.
Can the judge be fair enough to hire on? Open empirical question — needs the calibration/bias harness answering it continuously, not a one-time check.
Anti-cheat is an arms race — candidates + their LLMs adapt. Design for "raise cost + flag," not "solve."
Candidate trust: people distrust "an AI judged me." Transparency (show the rubric, the evidence, the recourse) is a product requirement, not a nicety.
Cold-start content: the bank needs enough calibrated questions before company use is credible — the flywheel helps but you have to prime it.
Scope: two-sided + multimodal + fair-eval is a lot. The thin-slice order exists so you can find out if the eval core works before building the rest.