A design doc for a messaging service that handles text, voice, and video at once and can escalate between them, where the streams carry complementary information (words, tone, what's being shown) that must be fused on a shared event-time timeline before reaching the LLM. Centerpiece: multimodal fusion + co-sequencing. Escalation tiers and batch transcription are supporting sections.
0 · The thesis (say this first)
Everything becomes tokens at the end — but the modalities are not redundant
transcripts to dedup. They are complementary signals to fuse. Each
specialist model compresses its channel into structured annotations on a
shared event-time timeline; a sequencer aligns and bundles co-temporal
annotations into enriched tokens; the LLM consumes one fused stream. The two
hard parts are (a) aligning streams that have different latencies and rates on
event-time and (b) choosing what survives each channel's compression —
because the whole point is to keep the signal that lives in the audio/video
but not the words.
Two reframes that drive everything below:
Co-sequencing is a stream-join on event-time, not processing-time. (Flink/Dataflow watermark model.)
Escalation is a resource-tier upgrade of one persistent session. Text = cheap async tier; voice/video = expensive real-time tier. Same brain, different transport + latency budget + cost.
"this is fine" + angry tone + frustrated face is a complaint. The
transcript alone reads as approval. For a bank, that's the difference
between catching a churning customer and missing them. The non-verbal signal is
exactly the thing absent from the words — so you cannot flatten to text and
order by arrival.
2 · The core problem: event-time vs processing-time
Streams arrive on separate async paths with different, structural latencies:
purple text (instant) · green speech (+80ms) · orange gesture (+200ms) — the three fire at one real moment (amber line) but reach the sequencer spread across processing-time
Order by arrival and the instant text "actually no—" lands before the
slow-transcribed speech it was correcting → the LLM sees the conversation
backwards. Order by event-time and it's correct.
The concrete timing model (the crux)
Processing latency is per-modality and roughly constant:
audio integration ≈ 80 ms
visual integration ≈ 200 ms
So a sound and a sight that were simultaneous at the edge arrive at the
sequencer ~120 ms apart. This is a known structural offset, not random
jitter — which makes it correctable by construction (same idea as A/V lip-sync
in a video player).
Two consequences that define the design:
Latency calibration → recover event-time. Stamp at capture, then subtract the channel's known latency:
Now a co-temporal sound + gesture land on the same event-time axis and fuse correctly.
Reorder window ≥ max inter-stream skew. You must hold the timeline open at least ~120 ms (visual − audio, + jitter margin) before committing, or the slower visual event arrives after you flushed the audio it belonged with. → The commit delay is set by your SLOWEST modality (visual, 200ms), not the fastest. Your fusion window can never be tighter than your slowest channel.
This is the latency-vs-correctness knob, made physical.
3 · Architecture
edge → specialists (each compresses its channel to tags) → sequencer (event-time merge) → fusion → shared batched LLM, bound two-way to the transport-agnostic session store
4 · The Sequencer (the heart)
A k-way merge keyed on event-time, with a watermark commit. Structurally the
same "don't finalize while work is in flight" as the web-crawler's Queue.join()
— here the in-flight thing is a stream that hasn't reported up to time T yet.
on event e from stream s:
e.event_time = e.arrival_time - LATENCY[s] # calibrate
buffer.push(e) # min-heap on event_time
watermark[s] = e.event_time # this stream is current to here
W = min(watermark over all active streams) # global safe time
while buffer.peek().event_time <= W: # everything ≤ W has arrived
emit(buffer.pop()) # commit in event-time order
Watermark W = min over streams — you can only safely commit up to the point your slowest-reporting stream has reached. One quiet stream holds the line (same straggler dynamic as the training all-reduce barrier).
Idle-stream timeout. If a stream goes silent (user stops gesturing), its watermark would freeze the timeline forever. Advance it with a heartbeat / max-wait so a silent modality can't stall the conversation. (Mirrors the batcher's size-OR-timer: commit on watermark OR deadline.)
Co-temporal bundling. Events within a small ε window at the same event-time are fused into one enriched token rather than emitted separately (the prosody spike belongs to that word).
5 · Fusion: early vs late
early (left, amber): raw features → one coupled model · late (right, green): per-channel specialists emit tags, then fuse — the practical choice
Late fusion is the practical choice. Each specialist emits typed annotations;
the token becomes a multi-channel record:
[t=4.10 words="this is fine"]
[t=4.10 prosody={anger:0.8, certainty:low}]
[t=4.10 visual={ref:"fee_line_item", affect:"frustrated"}]
→ fused LLM context:
User said "this is fine" but tone=angry and pointing at the fee
→ likely sarcastic; treat as a complaint about the fee.
The two hard parts late fusion exposes
Alignment across rates. Prosody is continuous (per-frame), words are discrete (per-utterance), gestures are sparse (per-event). Binding "this anger spike → that word" is a windowed join on event-time within a tolerance — the sequencer joining features to tokens, not just ordering tokens.
Compression: what survives. You can't send raw audio/video to the LLM — it's tokens at the end. Each specialist compresses its channel to a few tags, and the design question is which signal survives. The emotion/referent is precisely what's in the media but not the transcript — lose it in compression and you've discarded the reason the channel exists. Design the tag schema around the signal that's unique to each channel.
6 · Escalation: tiers of one persistent session
The session store is the source of truth and is transport-agnostic — so
moving text→voice→video is binding a new transport to existing state, not a
rewrite. This is the load-bearing decision (make it in step 2 below).
Tier
Transport
Latency budget
Cost
Concurrency/GPU
Text
WS/HTTP
seconds
cheap
many
Voice
WebRTC: VAD→STT→LLM→TTS
sub-second (mouth-to-mouth)
GPU slot
few
Video
+ vision specialist
sub-second + 200ms vision
GPU slot ++
fewer
Escalation triggers: explicit ask · detected frustration (now measurable from the prosody/visual tags!) · task complexity · identity verification · human handoff.
Handoff: provision the tier's GPU slot, attach the existing session (history + intent + fused affect state) — the voice tier starts already knowing the user is frustrated about the fee. Downgrade path symmetric.
Barge-in re-sequences, not just cancels. When the human interrupts the bot, you (1) cancel TTS instantly (10_audio_pipeline_interruptible — single Event, single-chunk latency) AND (2) treat the interruption as a new high-priority event that may reorder/discard buffered tokens in the sequencer.
7 · Build order (the "layer a constraint" rhythm)
Each step runs before the next exists — present it this way; let the interviewer
"add a requirement" and each requirement is the next step.
Text messaging — session store + stateless handlers + LLM. Baseline.
Persistent, transport-agnostic session — {id, identity, history, state}; workers stateless. ← the decision that makes everything else cheap.
Voice channel — WebRTC + VAD→STT→LLM→TTS, reusing the same store + LLM.
Sequencer — event-time calibration + watermark commit; needed the moment two streams coexist.
Fusion — specialists emit tags; bundle co-temporal annotations; design the tag schema around channel-unique signal.
Video channel — add the 200ms vision specialist; the sequencer's window widens to its latency; nothing else changes (that's the payoff of step 4).
Scale & economics — scheduler (text tier vs GPU voice/video slots), batch the shared LLM, persist KV-cache across escalation (~7× headroom from KV-cache compression — see voice_agents_notes.md), cost per session per tier.
"This is a stream-join on event-time — the modalities are sources; the hard part is the reorder window and the watermark that decides when the timeline is safe to commit."
"Audio is ~80ms, visual ~200ms, so simultaneous edge events arrive ~120ms apart. I calibrate by subtracting per-stream latency and size the reorder window to the slowest modality — correctness is bounded by your slowest channel."
"I use late fusion — specialists emit structured tags — so models are swappable and it degrades gracefully. The schema is designed around the signal unique to each channel: tone from audio, referents from video."
"Escalation is a tier upgrade of one persistent session, not a new conversation — which is why I make the session store transport-agnostic on day one."
"Barge-in doesn't just cancel TTS — it re-sequences, because the interruption is a new high-priority event in the timeline."
Open questions for the next pass
Mouth-to-mouth latency target (banks usually want <800ms) → budget breakdown across VAD/STT/LLM/TTS + network + the 200ms vision tax.
Self-host vs API per hop (see voice_agents_notes.md cost tables).
Tag schema: exact fields per modality; how much affect detail the LLM can actually use.
Failure modes: one specialist down (degrade to remaining channels), watermark starvation, clock skew across edge devices.