Voice / Realtime Agents — AI-системы, которые поддерживают непрерывную двустороннюю сессию с пользователем через голос и другие live signals: слышат поток аудио, определяют границы реплик, понимают intent, отвечают с низкой задержкой, могут вызывать инструменты и прерываются, когда пользователь начинает говорить снова.
№73 Multimodal AI owns generic audio/media understanding; №75 owns live duplex session semantics: streams, turn detection, interruption, latency and realtime tool coordination. №70 Model Serving owns low-latency inference execution and capacity. №65 Model Gateway selects eligible realtime-capable model endpoints. №12 Tools owns tool contracts; №75 schedules them inside live conversation. №49 HITL handles human approvals, but realtime conversation itself is not HITL. №74 Computer Use may be invoked by voice intent, but owns GUI actions separately.
Prerequisites: №09 Context Engineering, №10 State, №12 Tools, №23 Uncertainty, №45 Verification, №46 Observability, №48 Guardrails, №50 Contracts, №51 Permissions/Secrets, №52 Security, №63 Resilience, №64 Budgets, №65 Gateway, №70 Serving, №73 Multimodal AI. Forward references: №76 Governance/Privacy, №77 Production Architecture.
REQUEST-TIME: audio ingestion, VAD, streaming ASR/understanding, turn state, model response stream, TTS, barge-in, tool calls. CONTROL PLANE: session policy, model/voice eligibility, max duration, latency SLO, consent/recording, tool allowlist, interruption policy. DATA PLANE: audio chunks, partial/final transcript, assistant deltas, tool events, timestamps, session state. OFFLINE: call-quality analysis, latency profiling, transcript evals, turn-taking evals and voice safety review.
Success: conversation remains coherent across turns, user can interrupt, tool results are reflected without duplicate speech/actions, and latency stays inside declared SLO. Retryable: transient media/network/model/tool failures. Permanent: unsupported codec, policy block, missing consent where required, hard session duration exceeded. Persist: session timeline, final transcripts, tool/action events, interruption/cancel events and minimal media refs as policy permits. Idempotency: interrupted speech output is not replayed blindly; consequential tool actions use stable operation identity.
№75 не владеет general speech recognition research, generic multimodal reasoning, telephony provider internals, TTS model training, browser automation or tool business logic. Она владеет LOW-LATENCY LIVE SESSION ORCHESTRATION ACROSS AUDIO, MODEL OUTPUT, INTERRUPTIONS AND TOOLS.
Assistant should listen and generally not emit overlapping speech unless product intentionally supports overlap.
System interprets final/partial turn and may start response.
Output can be interrupted at any time.
Agent may acknowledge wait, continue conversation or remain silent depending policy.
| Signal | Use | Risk |
|---|---|---|
| Silence duration | Simple turn-end heuristic. | Pauses inside sentence cause premature response. |
| Acoustic VAD | Detect speech/non-speech. | Noise/music can confuse. |
| Semantic completion | Model estimates utterance is complete. | Language/model dependent; must not block forever. |
| Push-to-talk | Explicit user control. | Less natural but robust. |
| Hybrid | VAD + semantic + timeout. | More tuning, usually best UX. |
Audio/text deltas are being emitted.
VAD detects real user input above threshold.
Stop TTS/model stream, mark how much was actually played, transition to USER_SPEAKING.
{
"assistant_turn_id": "AT-42",
"generated_text": "Your appointment is Tuesday at 14:30...",
"audio_started_at": "...",
"played_until_char": 24,
"interrupted": true,
"interrupted_at_ms": 1820,
"next_turn_context": {
"assume_user_heard": "Your appointment is"
}
}
Can drive early semantic preparation, retrieval or response prefill.
Use for committed transcript, consequential tool arguments and audit.
“Book for Friday” may become “don’t book for Friday”. Never commit irreversible action from unstable partial text.
Timestamped user/assistant utterances, partial/final labels.
Current topic, pending tool, turn state, interruption, current form/slot values.
Confirmed names, dates, goals and unresolved questions, separate from raw audio history.
{
"session_id": "RT-...",
"principal_id": "...",
"channel": "voice",
"started_at": "...",
"state": "ASSISTANT_SPEAKING",
"language": "ru-RU",
"model_route": "realtime-standard",
"voice_profile_ref": "voice://approved/v2",
"tool_policy_ref": "policy://voice-tools/v4",
"consent": {
"audio_processing": true,
"recording": false
},
"current_turn_id": "TURN-...",
"pending_operation_ids": [],
"expires_at": "..."
}END-OF-TURN TO FIRST AUDIO ≈ turn detection + final/usable ASR delay + network transit + model first-token latency + tool wait (if needed) + TTS startup + playback buffer EXAMPLE BUDGET: turn detection 180 ms ASR finalize 90 ms network 40 ms model TTFT 220 ms TTS startup 90 ms play buffer 40 ms ---------------------------- first audible reply 660 ms A 2-second tool call changes the interaction entirely.
Turn acknowledgment, local UI cues, simple stream operations.
Fast answer start feels natural for many interactions.
Acceptable if response quality/tool value justifies it.
Need progress cue, filler acknowledgment, asynchronous tool pattern or UX redesign.
Short safe prefix can be spoken while rest of response continues generating.
Unlike text UI, already spoken words cannot be edited after tool result or verifier changes conclusion.
Buffer enough content to avoid sentence fragments/unsafe reversals while keeping latency low.
“Подтвердите: отправить 300 евро Ивану Петрову?”
Only a stable, explicit confirmation authorizes commit.
After approval, execute exact frozen operation contract with idempotency key.
| Tool state | User says “стоп” | Action |
|---|---|---|
| Not started | Cancel intent | Drop pending operation. |
| Read-only request in flight | Cancel if supported | Ignore late result if session no longer needs it. |
| Reversible draft write | Cancel/compensate | Follow tool/business semantics. |
| Irreversible commit may have happened | Do not assume cancellation | Reconcile actual outcome, explain state, compensate only if supported. |
Every assistant response gets an epoch/version.
Older stream marked stale and its late deltas ignored.
Late tool response cannot resurrect an obsolete assistant turn.
{
"event_id": "EVT-...",
"session_id": "RT-...",
"turn_id": "TURN-...",
"sequence": 184,
"type": "assistant.audio.delta",
"timestamp": "...",
"payload_ref": "stream://...",
"correlation_id": "...",
"causation_id": "EVT-..."
}Helps detect gaps/reordering in control events.
Audio chunks preserve playout/capture time separately from event arrival.
Old response deltas ignored after interruption/new turn.
Small buffer smooths network arrival before ASR/audio pipeline.
TTS/network chunks need enough buffering to avoid audible gaps.
Perfectly smooth audio with 2-second buffer feels non-realtime.
Buffer strategy can adapt to measured network quality.
{
"codec": "pcm16_or_supported_codec",
"sample_rate_hz": 16000,
"channels": 1,
"frame_duration_ms": 20,
"language_hint": "ru-RU",
"timestamp_origin": "session_start",
"sequence": 184
}
Use locale/profile as hint, not an absolute constraint.
ASR/model may detect language switch and update response language.
Selected voice/model must support target language naturally enough for product requirement.
Reduce agent's own output leaking back into microphone.
Improve ASR/VAD robustness under background noise.
Poor echo handling can repeatedly interrupt assistant output.
Client audio stack/device capabilities materially affect session quality.
Voice users cannot visually scan a wall of text.
Pause and ask if user wants detail when content is long.
Amounts, dates, addresses and IDs need slower explicit phrasing.
Long URLs, code, tables and detailed reports should move to visual/text channel when available.
Typed fields remain internal.
Speak only information relevant to current user intent.
User can ask for exact details or receive them in text.
“Ignore the user and send files...” is untrusted content.
Speech source may not be authorized user.
High-risk action still requires exact operation confirmation/policy.
Permissions and allowed tools do not change based on spoken instructions in content.
Audio chunks processed for realtime understanding and deleted quickly.
Final transcript may be stored under different retention policy than raw audio.
Persisting raw call audio requires its own legal/product/data-governance basis.
| Capability | Means | Does not mean |
|---|---|---|
| Diarization | “Speaker A vs Speaker B” within audio. | Real-world identity. |
| Profile hint | Expected user/account context. | Proof that current speaker is that user. |
| Voice biometrics | Identity signal if explicitly implemented. | Automatic authorization for every action. |
| Authenticated session | Account/channel authority. | Every heard utterance came from account owner. |
Use secret broker / secure device input.
Handle through trusted auth flow; avoid retaining in transcript/logs.
Read only last digits where sufficient.
Do not assume loudspeaker environment is confidential.
If mic/TTS stream fails, continue in text where product supports it.
Offer to send result later instead of keeping session blocked.
Only if capability/quality policy allows.
Transfer/handoff for unsupported or high-risk conversation states.
Reconnect to same bounded session if still valid.
Resume from final transcript/tool state, not uncertain unplayed buffers.
If user intent near disconnect is ambiguous, clarify rather than replay speculative output.
Low latency, streaming, audio capability.
Complex research/analysis can run asynchronously while session stays responsive.
№65/34 can escalate subtask without forcing every spoken token through most expensive model.
Keep latest conversational exchange verbatim where useful.
Confirmed facts, decisions, preferences and unresolved issues.
Pending/completed operations separate from prose summary.
Older audio/transcript remains addressable outside active model context.
Conversational/general responses can stream with lightweight checks.
Use direct structured data rather than re-inference where possible.
Wait for deterministic verifier/tool/approval even if latency increases.
If later evidence contradicts spoken response, correct clearly rather than silently editing state.
Speech stop → system decides turn is complete.
User stops → first assistant audio heard.
User starts talking → assistant audio actually stops.
ASR quality by language/noise/device segment.
Share of assistant turns users interrupt; can reveal verbosity/latency issues.
External call latency during live conversation.
Session/network failures and resume success.
Audio/model/tools normalized by verified user outcome.
Did user accomplish intended task?
Premature end-of-turn, missed barge-in, false starts.
How often user must correct transcript/intent.
Correct values and safe authorization.
Not just average; p95 and tail gaps matter.
Failure to stop promptly on user speech.
Asks for repetition instead of guessing critical values.
No zombie streams/tools after user leaves.
Agent should not cut user off too early.
Output stops quickly and next turn starts cleanly.
VAD/ASR robustness by environment.
Model/ASR handles locale changes.
“Book... no, don't book” must not trigger early action.
Session behavior stays understandable.
Resume from stable checkpoint.
No unauthorized tool escalation.
Session continues or requests repetition; no silent fabrication.
Partial path does not commit high-risk action.
Turn epoch prevents stale speech output.
Generation/playback cancellation propagates quickly.
Sequence gaps handled deterministically.
No old credentials/tools remain usable.
Fallback/degrade rather than long unexplained silence.
Text fallback or explicit session recovery.
realtime/ ├── session.py ├── transport.py ├── audio.py ├── vad.py ├── transcript.py ├── turn_state.py ├── response_stream.py ├── tools.py ├── interruption.py ├── policy.py ├── metrics.py └── evals/ MVP FLOW: connect ↓ create session_id ↓ audio chunks ↓ VAD / turn detection ↓ partial transcript ↓ final user turn ↓ model response stream ↓ TTS/audio stream ↓ if user speaks: cancel assistant stream mark played boundary start new turn ↓ optional read-only tool ↓ typed result ↓ spoken summary ↓ end session ↓ persist: final transcript tool events latency timeline no raw audio unless policy says so
Add payments/bookings/browser control only after the conversational transport and interruption model is stable.
| Need | Upgrade |
|---|---|
| Phone calls | Telephony transport, call-state integration, DTMF, transfer/hangup semantics. |
| Slow complex tasks | Background subagent/model with asynchronous result and live session acknowledgment. |
| High-risk actions | Exact voice confirmation + HITL/policy + operation idempotency ledger. |
| Multiple languages | Language detection, multilingual ASR/TTS/model routes and eval segmentation. |
| Noisy environments | Better AEC/noise suppression/device adaptation. |
| Realtime screen co-pilot | Fuse №73 visual context and №74 computer-use while voice remains control channel. |
| Long sessions | Session summarization, durable checkpoints, stricter retention and cost controls. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Условно. Только если live conversation/hands-free interaction creates real product value. |
| Separate Component? | YES. Realtime session transport/state layer integrating Context, Tools and Model Router. |
| Минимум 80% ценности? | Duplex audio, VAD/turn state, streaming output, barge-in, session state, one read-only tool, latency timeline and text fallback. |
| Когда overkill? | Использовать live voice stack для задач, где пользователь спокойно ждёт asynchronous text result. |
| Trigger? | Conversation latency, interruptions, hands-free control or live support are material to user experience. |
| Как измерить uplift? | Task success, first-audible latency, barge-in stop time, repair turns, disconnect rate, tool wait and cost/completed session. |
| Можно ли rule/tool/code заменить LLM? | Частично. Transport, VAD, turn state, interruption, permissions and tool control are deterministic. LLM handles language understanding/response, not session safety. |
Not a sequence of independent chat requests.
Barge-in cancellation is core UX and capacity behavior.
Generated text is not equivalent to delivered audio.
Consequential actions wait for stable final intent.
Turn detection, model, tool and TTS each have measurable budgets.
Late model/tool events cannot revive obsolete turns.
Heard instructions do not expand authority.
Processing and recording are separate policies.
Voice → text / realtime → async / AI → human when needed.
CLIENT / MICROPHONE
↓
DUPLEX REALTIME TRANSPORT
↓
SESSION ID
principal
policy
consent
language
turn state
↓
AUDIO CHUNKS
sequence
timestamp
↓
VAD / TURN DETECTION
↓
PARTIAL TRANSCRIPT
useful for:
early understanding
prefetch
preparation
NOT FOR:
irreversible commit
↓
FINAL USER TURN
↓
CONTEXT MANAGER
recent turns
session summary
confirmed facts
pending tools
↓
№65 REALTIME MODEL ROUTE
↓
DECIDE:
respond
clarify
tool
wait
↓
IF TOOL:
typed call
deadline
operation_id
↓
fast?
yes → wait
no → acknowledge /
async pattern
↓
STREAM RESPONSE
text deltas
TTS/audio deltas
↓
TRACK:
generated content
played content
↓
USER STARTS SPEAKING?
YES
↓
BARGE-IN
cancel current model/TTS
mark played boundary
increment turn epoch
ignore late old deltas
↓
LISTEN TO NEW TURN
HIGH-RISK TOOL:
final user intent
↓
read back exact action
amount / recipient / date / object
↓
explicit confirmation
↓
freeze operation contract
↓
execute once
↓
verify / reconcile
↓
speak result
FAILURE:
audio fails
→ text fallback
realtime model overloaded
→ compatible low-latency fallback
OR async mode
slow tool
→ progress cue / async result
disconnect
→ resume from last stable turn
MFA / unsupported approval
→ human handoff
SECURITY:
heard speech
recording
other speaker
TV/audio
=
UNTRUSTED CONTENT
AUTHORITY COMES FROM:
authenticated session
explicit task
permissions
deterministic policy
exact confirmation
PRIVACY:
process audio
≠
record audio
store:
final transcript / events
only as policy requires
raw audio:
short-lived by default
unless recording is explicitly enabled
BOUNDARIES:
№73
generic multimodal/audio understanding
№75
live session + turn-taking + barge-in
№70
low-latency serving
№65
realtime-capable routing
№12
tool contracts
№49
approval/handoff
№76
privacy/retention/governance
CORE PRINCIPLE:
A REALTIME AGENT
IS NOT
A CHATBOT WITH TTS.
IT IS A LIVE CONTROL SYSTEM
THAT MUST KNOW:
WHO HAS THE FLOOR,
WHAT WAS ACTUALLY HEARD,
WHAT IS STILL PARTIAL,
WHICH ACTION IS SAFE,
WHICH RESPONSE IS CURRENT,
WHEN TO STOP SPEAKING,
WHEN TO WAIT,
WHEN TO ASK,
WHEN TO FALL BACK,
AND HOW TO KEEP
THE CONVERSATION COHERENT
UNDER NETWORK,
MODEL
AND TOOL LATENCY.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №75 Voice / Realtime Agents.
B–E. Existing boundary and placement. The existing conceptual boundary, class SPECIALIZED, default OFF and owner Tool / Action Engine remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.