Multimodal AI — способность системы принимать, связывать и интерпретировать информацию из нескольких модальностей: текста, изображений, документов, аудио, видео, интерфейсных кадров, таблиц и других media representations — и использовать их в одном reasoning/action loop.
№56 Document Parsing / OCR / Multimodal Extraction owns deterministic/structured conversion of raw files into Document IR, text, tables, bounding boxes and extracted media observations. №73 owns model-level understanding that depends on native modality or cross-modal relations. №09 Context Engineering owns general context construction; №73 adds modality-aware budgeting/selection. №65 Model Gateway owns selecting a model endpoint with needed vision/audio/video capabilities. №70 Serving owns execution mechanics and modality-related compute capacity. №74 Computer Use owns observe→act loops over UI; it may consume screenshots through №73. №75 Voice / Realtime owns live session/streaming semantics, not generic audio understanding.
Prerequisites: №09 Context Engineering, №12 Tools, №25 Evidence-First, №45 Verification, №50 Contracts, №52 Agent Security, №56 Parsing/OCR, №60 Artifact Store, №61 Provenance, №65 Model Gateway, №70 Serving. Forward references: №74 Computer Use, №75 Voice/Realtime, №76 Data Governance, №77 Production Architecture.
REQUEST-TIME: media selection, preprocessing, modality route, native model call, grounding and result normalization. CONTROL PLANE: supported modalities, image/frame/audio limits, model capability map, privacy rules, extraction-vs-native strategy. DATA PLANE: media refs, crops/frames/time segments, transcripts, spatial selectors, model observations. OFFLINE: media indexing, thumbnails, transcripts, frame/keyframe extraction, eval sets and cost/quality benchmarks.
Success: selected model receives only authorized, supported media within budget and output remains grounded to exact asset/region/time range. Retryable: media decode/transcode/provider/runtime transient failure. Permanent: unsupported/corrupt media, exceeded hard size/duration, forbidden data class. Persist: asset version/hash, media transform, selector, model/capability revision, output refs. Security: media is untrusted content; text rendered inside image/document/audio does not gain instruction authority.
№73 не владеет OCR/parser pipelines, image generation/editing, computer-use action policy, live voice session orchestration, general RAG or model serving. Она владеет MODALITY-AWARE CONTEXT AND MODEL-LEVEL CROSS-MODAL PERCEPTION/REASONING.
Meaning, instructions, exact strings, code, structured claims.
Objects, layout, charts, screenshots, colors, geometry, handwriting.
Speech, prosody, pauses, sound events, speaker turns.
Actions, state changes, motion, sequence, scene transitions.
Pages, tables, forms, diagrams, annotations and reading order.
Controls, labels, focus, dialogs and coordinates used by №74.
Rows/columns/cells; often best kept structured rather than rasterized.
Prefer typed numeric/time-series tools unless native modality model adds value.
{
"asset_ref": "artifact://tenantA/media/IMG-42/v3",
"media_type": "image/png",
"sha256": "...",
"bytes": 4182301,
"width": 3024,
"height": 4032,
"duration_ms": null,
"source_ref": "source://...",
"classification": "INTERNAL",
"created_at": "...",
"metadata": {
"orientation": 1
}
}Downscaled overview preserves composition and object relationships.
Crop high-resolution region when task depends on fine visual details.
Split into overlapping tiles when model limits would destroy detail.
Each additional image/tile consumes model-specific visual tokens/features.
Normalized or pixel coordinates for an object/region.
Use when shape/segmentation matters and tool/model supports it.
“Top-right chart legend” plus exact asset/region selector.
{
"asset_ref": "artifact://.../image-v3",
"selector": {
"type": "bbox",
"x": 0.61,
"y": 0.08,
"w": 0.24,
"h": 0.19,
"coordinate_space": "normalized"
},
"observation": "..."
}
Prefer native text extraction when present.
Columns, forms, captions, headers and proximity convey meaning.
Rows and columns are better represented structurally where possible.
Charts, diagrams, maps and screenshots may require native vision reasoning.
Axis names, legend, data labels and captions become explicit context.
Line slope, bar height, node connections, spatial grouping and arrows remain visible.
When precise numeric claims matter, prefer structured source data/table or deterministic chart extraction over visual estimation.
Buttons, dialogs, selected tabs, error banners and layout.
Structured UI metadata can improve exact element identification.
Multimodal perception may identify target; computer-use policy decides and executes click/type/navigation.
Most semantic tasks should start with timestamped transcription.
Diarization or speaker labels preserve conversation structure.
Pauses/emphasis/tone can matter in limited tasks but are easy to overinterpret.
Alarm, applause, engine, impact, music etc. may require native audio classifier/model.
{
"asset_ref": "artifact://meeting/audio-v1",
"segments": [
{
"start_ms": 84210,
"end_ms": 91760,
"speaker": "S2",
"transcript": "...",
"confidence": 0.94
}
],
"native_audio_needed": false
}
| Strategy | Good for | Failure |
|---|---|---|
| UNIFORM SAMPLE | Coarse scene overview. | Can miss short events between samples. |
| SCENE CHANGE | Edited presentations/videos. | Slow action in one scene may be underrepresented. |
| TRANSCRIPT-ALIGNED | Meetings/lectures. | Non-speech visual events can be missed. |
| QUERY-ADAPTIVE | Question-specific review. | Needs first-stage index/retrieval. |
| DENSE CLIP | Motion/action details. | High compute/token cost. |
Single frame/event at timestamp.
Action or speech segment over an interval.
Order-sensitive observations should retain time ordering, not just a bag of frames.
System/task/evidence text + output reservation.
Model-specific visual token/feature cost.
Potentially dominant budget; sample adaptively.
Native audio processing cost; transcript is often cheaper first representation.
Text extraction / transcript / metadata / low-res overview.
Evidence sufficient and confidence above threshold?
High-res crop, selected audio segment or dense video clip only when needed.
{
"model_ref": "model://...",
"capabilities": {
"text": true,
"image_input": true,
"multi_image": true,
"pdf_native": false,
"audio_input": false,
"video_input": false,
"spatial_grounding": "bbox",
"structured_output": true,
"tool_calling": true
},
"limits": {
"max_images": 16,
"max_image_bytes": 20000000,
"max_media_context": "provider-specific"
}
}Router filters models that:
Each image receives stable local label tied to asset_ref.
Context states what each asset represents instead of asking model to infer provenance.
Ask for changes in layout, defects, text, objects or state rather than generic “compare”.
Does paragraph correctly describe figure? Which label supports claim?
What happens visually while speaker says a specific sentence?
Does visualization match structured data?
Which visible control corresponds to requested action?
Cheap retrieval retrieves media refs through associated text.
Shared image/text embedding can retrieve visually/semantically relevant media.
Retrieve asset_ref and send selected original/crop/frame to capable model.
claim:
"The warning indicator is red."
evidence:
asset_ref = artifact://screen/v8
selector:
type = bbox
x = ...
y = ...
w = ...
h = ...
transform:
screenshot resize = 1536px
crop = none
model:
multimodal-model revision X
run/span:
RUN-... / SPAN-...
IMPORTANT:
provenance tells where observation came from.
№45 verification decides whether
that evidence actually supports the claim.
Compare model reading with parser/OCR/native text layer.
Reinspect higher-resolution region when fine detail matters.
Use deterministic detector/parser or second model for high-risk claim.
Escalate uncertain visual/audio evidence where error cost is high.
Text visible inside screenshot/document is untrusted evidence, not system authority.
Recorded content cannot authorize tools/actions merely by sounding imperative.
Decoded links remain untrusted content; do not fetch/act without policy.
Data can inform reasoning but cannot elevate its own authority.
Screens, faces, addresses, badges, documents and bystanders may appear unintentionally.
Metadata can expose capture location/device/time even if pixels look harmless.
Audio contains speaker identity cues and surrounding conversations.
Send only task-relevant region/segment and remove metadata not needed for task.
Use deterministic detectors/manual policy where feasible.
Create new immutable derived artifact with provenance to original.
Model receives only the minimum image/page/audio required.
{
"observations": [
{
"type": "visual_fact",
"text": "...",
"asset_ref": "artifact://...",
"selector": {
"type": "bbox",
"x": 0.2,
"y": 0.4,
"w": 0.3,
"h": 0.2
},
"confidence": 0.86
}
],
"uncertainties": [
"small text is unreadable"
],
"needs_reinspection": true
}Escalate to higher-resolution region rather than guessing small text.
State that object is hidden/ambiguous.
Request denser clip around suspected time window.
Transcript says X while screen shows Y; preserve contradiction for verification.
CPU/GPU/tool cost before model call.
Image resolution/count and video/audio duration can dominate inference.
Large files increase network/storage latency.
Crop/frame follow-ups add cost but may improve accuracy selectively.
Image/audio/video encoder may add significant prefill-like compute.
Different resolutions/frame counts complicate batching and capacity estimation.
№70 should benchmark text-only and multimodal workloads separately.
Object/text/event recognition accuracy.
BBox/timecode overlap or exact evidence selector correctness.
Correct relation between image/text/audio/video evidence.
Final answer/action decision passes domain verifier.
Model avoids guesses on insufficient resolution/evidence.
Embedded malicious instructions do not change tool authority.
Native calls/crops/frames per successful task.
Decode/preprocess/model/reinspection timing.
Production media is imperfect.
Test unreadable vs readable boundaries.
Evidence can be distributed across pages/figures.
Video tests reveal sampling misses.
Model should surface conflict, not blend sources silently.
Visible/spoken malicious instructions.
Charts, forms, screens and media from actual application.
Measure false positive/hallucination rate.
Share of tasks escalating beyond text/extraction.
Count and resolution buckets sent to model.
Sampling density and selected clip duration.
How much audio bypasses transcript-only path.
Valid page/bbox/time evidence selectors.
Unreadable/uncertain media handled without hallucination.
Total extraction + model cost per verified multimodal task.
Preprocess + upload + inference + reinspection.
Typed failure, no model call with undefined bytes.
Verify from content; reject mismatch where unsafe.
Resize/reject before memory blow-up.
Eval reveals whether adaptive denser inspection triggers.
Tool policy remains unchanged.
Result marked incomplete rather than silently citing nonexistent evidence.
№65 rejects route before execution.
Derived temporary assets/jobs are released according to lifecycle policy.
multimodal/
├── media_contracts.py
├── normalize.py
├── image.py
├── selectors.py
├── context.py
├── capability.py
├── grounding.py
├── security.py
├── metrics.py
└── evals/
MVP FLOW:
asset_ref
↓
verify mime / size / policy
↓
№56 extraction if document
↓
cheap text path first
↓
task needs visual evidence?
├─ no → text model path
└─ yes
↓
resize / crop
↓
store derived ref
↓
№65 select vision-capable model
↓
native image + task
↓
structured observation
↓
bbox/page selector
↓
№45 verify if material
↓
answer / downstream actionAdd audio/video only when a real workload requires information that transcript/keyframes cannot capture.
| Need | Upgrade |
|---|---|
| Small text / dense pages | Adaptive high-res crops/tiling + stronger OCR cross-check. |
| Many figures/charts | Figure-level indexing and multimodal retrieval. |
| Meeting audio | Timestamped ASR/diarization first; native audio only for acoustic questions. |
| Long video archives | Transcript + scene/keyframe index + query-adaptive clip inspection. |
| UI automation | №74 computer-use loop with screenshot/DOM/accessibility fusion. |
| Live speech | №75 realtime session architecture; streaming audio perception/output. |
| Privacy-sensitive media | Pre-model redaction, private/local multimodal targets and stricter retention. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Условно. Если task genuinely depends on non-text evidence. Не включать vision/audio/video автоматически. |
| Separate Component? | NO. Manifest правильно помещает capability в Context Manager + Tool / Action Engine. |
| Минимум 80% ценности? | Native image/document inspection, media refs, capability routing, crop/resize, grounding, security and modality-specific evals. |
| Когда overkill? | Видео/audio processing stack для приложения, где все реальные задачи решаются text/document extraction. |
| Trigger? | Text extraction loses material spatial, visual, acoustic or temporal evidence and evals show native modality uplift. |
| Как измерить uplift? | Perception/grounding/task accuracy, abstention, native-media escalation rate, cost per grounded success and latency. |
| Можно ли rule/tool/code вместо separate LLM agent? | Да. Media normalization, selection, transforms, routing and grounding contracts are deterministic. A multimodal model is invoked as a capability only where required. |
Preserve native media when spatial/acoustic/temporal signal matters.
Extraction/transcript/index is the cheap first path.
Asset hash/version and derived transforms are traceable.
Page, bbox, crop, frame or time range for material claims.
Resolution/frame count/duration are compute inputs.
Do not discover at runtime that selected model cannot process needed modality.
Visible/spoken instructions never gain system authority.
Crop, redact and strip metadata before model boundary when possible.
Keep multimodality as a capability inside R04/R07 unless a real product boundary justifies more.
USER / TASK
↓
INPUT ASSETS
text
image
document
audio
video
UI screenshot
↓
VERIFY / REGISTER
asset_ref
MIME
hash
data class
dimensions / duration
↓
CHEAP REPRESENTATION FIRST
document:
№56 native text / OCR / tables / layout IR
audio:
transcript + timestamps
video:
transcript + scene/keyframe index
image:
metadata + low-res overview if enough
↓
CONTEXT MANAGER
what evidence is actually needed?
↓
DOES TEXT/EXTRACTION PRESERVE
THE MATERIAL SIGNAL?
YES
↓
use cheap representation
NO / UNCERTAIN
↓
NATIVE MODALITY ESCALATION
image:
overview → crop / tile
audio:
selected native segment
video:
selected clip / denser frames
document:
selected page / figure / region
↓
№65 CAPABILITY FILTER
image?
audio?
video?
multi-image?
grounding?
structured output?
policy?
↓
MULTIMODAL MODEL
↓
STRUCTURED OBSERVATIONS
text
uncertainty
asset_ref
page
bbox
time range
↓
№45 VERIFICATION
↓
ANSWER / TOOL INPUT
SECURITY:
text in image
spoken command
PDF instruction
QR / link
=
UNTRUSTED CONTENT
it may inform the task
but cannot grant authority
PRIVACY:
crop only needed region
strip irrelevant metadata
redact unnecessary PII
select approved model boundary
VIDEO RULE:
DO NOT:
send every frame
DO:
index
retrieve time range
inspect native clip
ground to timestamp
DOCUMENT RULE:
DO NOT:
OCR everything blindly
DO:
native text first
OCR where needed
preserve layout
inspect figures visually
verify exact numbers structurally
BOUNDARIES:
№56
EXTRACTS structured content
№73
UNDERSTANDS native/cross-modal evidence
№74
ACTS on visual computer state
№75
RUNS realtime voice/media sessions
№65
ROUTES to capable model
№70
SERVES that multimodal model
CORE PRINCIPLE:
MULTIMODAL AI
IS NOT
"PUT AN IMAGE INTO THE PROMPT."
IT IS A SYSTEM THAT KNOWS:
WHAT SIGNAL THE TASK NEEDS,
WHICH REPRESENTATION PRESERVES IT,
HOW MUCH MEDIA TO SEND,
WHICH MODEL CAN HANDLE IT,
WHERE THE EVIDENCE IS,
HOW TO VERIFY IT,
AND WHEN TEXT ALONE
IS ALREADY ENOUGH.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №73 Multimodal AI.
B–E. Existing boundary and placement. The existing conceptual boundary, class SPECIALIZED, default CONDITIONAL and owner Context Manager + Tool / Action Engine remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.