Document Parsing / OCR / Multimodal Extraction — слой, который превращает PDF, DOCX, HTML, сканы, изображения, таблицы и сложные документы в структурированное представление, пригодное для поиска, RAG, evidence mapping, downstream extraction и повторной обработки.
№55 Data Ingestion & Sync доставляет source object/blob и отслеживает его версию; №56 извлекает структуру/контент из конкретной версии документа. №08 RAG ищет по уже подготовленным chunks/index; №56 создаёт material для indexing. №25 Evidence-First использует page/block refs как evidence; №56 обеспечивает адресуемость. №60 Artifact Store хранит raw/derived blobs; №56 создаёт derived artifacts. №61 Provenance углубляет lineage. №73 Multimodal AI отвечает за model-level reasoning по нескольким модальностям во время task; №56 — преимущественно ingestion-time extraction/normalization.
Prerequisites: №08 RAG, №25 Evidence-First, №46 Observability, №50 Contracts, №53 Sandbox при недоверенных converters, №55 Ingestion & Sync. Forward references: №60 Artifact Store, №61 Provenance, №66 Vector DB/Embeddings, №67 GraphRAG, №73 Multimodal AI.
REQUEST-TIME: иногда — ad hoc uploaded document. CONTROL PLANE: parser profiles, supported types, model/runtime versions, thresholds. DATA PLANE: bytes/pages/blocks/tables/images/Document IR. OFFLINE: основной режим — ingestion, reprocessing, quality benchmarks, regression corpus.
Success: raw document → versioned structured IR with stable page/block refs and acceptable quality. Retryable: transient worker/model/converter failure. Permanent: corrupt/unsupported/encrypted-without-key file, deterministic parser failure. Partial: document may be usable with failed pages/tables marked. Idempotency: same source hash + parser profile/version → reproducible derived version. Persist: source hash, parser/OCR/model version, block refs, page image refs, quality flags, failures. Trace: route → stages → output.
№56 не владеет recurring sync, vector search, embedding storage, downstream question answering, general vision reasoning, source permissions или artifact storage engine. Она владеет RAW DOCUMENT → STRUCTURED, ADDRESSABLE, QUALITY-ANNOTATED DOCUMENT REPRESENTATION.
Фактический текстовый слой.
Page, columns, boxes, reading order, coordinates.
Headings, sections, lists, footnotes, captions.
Rows, columns, merged cells, headers, units.
Charts, diagrams, screenshots, photos, formulas.
Где именно находится evidence.
Title, author, timestamps, file properties, language.
Source object, version, parser profile, transformation chain.
Не доверять только filename extension. Проверять MIME/signature/container.
Bytes, page count, dimensions, archive expansion, object count.
Encrypted/password-protected files получают explicit status.
Есть ли извлекаемый текст и насколько он правдоподобен.
Language/script metadata помогает OCR и downstream normalization.
Converters/parsers для недоверенных файлов желательно запускать в ограниченном execution environment.
Извлечь glyph/text blocks, coordinates, fonts/size hints, page number.
Paragraphs, heading styles, tables, lists, relationships, images.
Semantic elements, headings, lists, tables, links, alt/caption metadata.
Image-only page, scan, photographed page, rasterized text region.
Rotation/orientation, crop, deskew, contrast/resolution where helpful.
Хранить recognized text вместе с bounding boxes/page refs/confidence.
Numbers, formulas, names, punctuation, columns and tables особенно чувствительны.
У страницы нормальный native text layer?
Если да — использовать его.
Есть raster regions: scan, embedded image, figure labels, stamp?
OCR сохраняет region/page coordinates и добавляется в IR как отдельный extracted block.
Paragraph, title, header, footer, list, caption, table, figure.
Logical order independent of raw extraction order.
Page + bounding box lets downstream cite/highlight exact region.
Heading levels and parent-child section relations improve chunking.
{
"block_id": "tbl-7",
"type": "TABLE",
"page": 12,
"bbox": [72, 188, 520, 640],
"caption": "Operating metrics",
"header_rows": 1,
"columns": [
{"name": "Metric"},
{"name": "2025"},
{"name": "2026"}
],
"rows": [
["Revenue", "120", "145"],
["Margin", "21%", "24%"]
],
"source_ref": "doc://...#page=12",
"quality": {
"structure_confidence": 0.93
}
}Для critical tables полезно хранить и structured cells, и original page/image reference.
Сохранить page/bbox/image artifact ref, caption и surrounding text.
Извлечь axis labels, legend, annotations, screenshot text where useful.
При необходимости мультимодальная модель формирует structured description/data claims — обязательно с figure ref.
| Case | Deterministic first | Multimodal escalation |
|---|---|---|
| Simple digital text | Native parser. | Usually unnecessary. |
| Scanned paragraph | OCR. | Only if OCR/layout insufficient. |
| Complex chart | Extract caption/labels/image ref. | Describe axes/trends/relations into schema. |
| UI screenshot | OCR text + regions. | Infer component relationships/state if needed. |
| Diagram | Image region + text labels. | Extract nodes/edges/semantic relations. |
| Form | Layout/OCR fields. | Resolve ambiguous field-value associations. |
{
"contract": "document.ir.v2",
"document_ref": "doc://tenant_A/123",
"source_version": "v42",
"parser_profile": "documents.v2",
"pages": [
{
"page": 1,
"width": 595,
"height": 842,
"blocks": [
{
"block_id": "b1",
"type": "HEADING",
"text": "...",
"bbox": [72, 80, 520, 120],
"reading_order": 1,
"parent_section": "s1",
"extraction": "NATIVE",
"quality": 0.99
}
],
"figures": [],
"tables": []
}
],
"quality": {
"text_coverage": 0.98,
"failed_pages": []
}
}Stable canonical document reference tied to tenant/source object.
Page number/index + page image reference when applicable.
block ID scoped by parsed document version; do not assume block IDs survive parser reprocessing unchanged.
Создаёт sections, blocks, tables, captions, reading order, coordinates.
Использует структуру и retrieval requirements, чтобы собрать chunks.
Embeddings / search index хранит chunks + metadata + source refs.
Какая доля pages/regions имеет usable text.
Насколько logical order соответствует документу.
Confidence + suspicious characters/low-quality regions.
Header/cell completeness, row/column consistency.
Failed pages/blocks tracked explicitly.
Native + OCR duplicates detected.
Detected language/script inconsistent with OCR profile.
Extreme block count, empty headings, broken hierarchy flags.
| Status | Meaning | Downstream handling |
|---|---|---|
| SUCCESS | Required extraction stages completed. | Normal indexing. |
| PARTIAL | Usable IR exists, but pages/blocks/tables failed. | Index valid regions; expose quality flags; retry failed regions. |
| NEEDS_OCR | Native text unavailable/insufficient. | Route selected pages to OCR. |
| NEEDS_MULTIMODAL | Critical information is visual/layout-dependent. | Escalate selected regions, not entire corpus blindly. |
| UNSUPPORTED | No supported extraction path. | Quarantine / converter / manual handling. |
| ENCRYPTED | Cannot access content with current authorization. | Do not brute-force; request proper access. |
| CORRUPT | Artifact cannot be parsed reliably. | Retain source ref and error; no fake extraction. |
Какая exact source version обработана.
Native parser/layout pipeline profile.
OCR engine/model/language/preprocess settings.
Model/prompt/schema version for selected visual regions.
Сложные formats/converters могут иметь vulnerabilities; недоверенные преобразования лучше изолировать.
Text inside document remains untrusted evidence/content and не получает instruction authority.
Parsed blocks/chunks inherit tenant/access metadata from source document.
native / OCR / layout / multimodal / hybrid.
total, parsed, OCR, failed, multimodal pages/regions.
per stage and per page/document.
OCR/model calls/compute where relevant.
coverage/table/OCR/structure flags.
source/parser/OCR/MM versions.
page images, tables, IR, logs, extracted figures.
unsupported/encrypted/corrupt/page failure/error class.
Representative passages/names/numbers preserved.
Multi-column/list/footnote ordering.
Headers, numeric values, merged cells, units.
Required chart/diagram facts mapped to source region.
Names, dates, numbers, codes on scan corpus.
Downstream answer can point to correct page/block.
Does new parser improve answer/evidence retrieval on eval tasks?
OCR/MM escalation rate and cost per usable page.
% pages/regions with usable extracted text.
% pages requiring OCR rather than native parse.
% pages/regions sent to vision/MM extraction.
Failed or partial pages per document/source type.
Critical table extraction pass rate.
Correct page/block source mapping downstream.
Time per document/page by route.
Downstream retrieval/evidence improvement vs parser baseline.
parsing/ ├── detect.py ├── router.py ├── document_ir.py ├── native/ │ ├── pdf.py │ ├── docx.py │ └── html.py ├── ocr/ │ ├── page.py │ └── quality.py ├── tables.py ├── figures.py ├── validate.py └── tests/ parse(source_ref, profile): inspect file choose native route assess page coverage OCR only missing pages/regions preserve tables/figures build Document IR validate quality persist derived version
Добавлять sophisticated layout/MM extraction только по failure clusters и eval uplift.
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да, если система работает с raw documents/scans/images. Для чистых API/JSON/text sources — нет. |
| Separate Component? | YES. Parser pipeline имеет собственные profiles, versions, artifacts, quality metrics и reprocessing lifecycle. |
| Минимум 80% ценности? | Native parse, page/block refs, tables, selective OCR, Document IR, quality flags, versioning. |
| Когда overkill? | Прогонять каждый простой DOCX через OCR + vision model + complex layout engine без доказанного downstream улучшения. |
| Trigger? | Raw files whose useful information is not already available as clean structured text/data. |
| Как измерить uplift? | Text/table/citation accuracy, RAG retrieval/evidence success, OCR/MM escalation cost, regression failures. |
| Можно ли rule/tool/code вместо LLM-agent? | Да, в большинстве pipeline. Native parser/OCR/layout tools first; multimodal model only for semantic visual extraction where needed. |
Если есть хороший text/structure layer — использовать его до OCR.
Pages, blocks, hierarchy, tables, figures и coordinates — first-class.
Pixels-to-text только там, где это реально нужно.
Vision/MM включается для visual semantics, а не по умолчанию на весь corpus.
Downstream должен уметь вернуться к page/region/source.
Не выдавать неполный extraction за SUCCESS.
Parser/OCR/MM/profile versions сохраняются рядом с IR.
OCR/vision не повышают authority источника.
Оптимизировать extraction по retrieval/evidence/task success, не только локальной accuracy.
№55 INGESTION / SOURCE OBJECT
↓
RAW DOCUMENT / ARTIFACT REF
↓
PRE-FLIGHT
type
size
encryption
text availability
language/script
↓
ROUTE EXTRACTION
├─ NATIVE PARSER
├─ LAYOUT EXTRACTION
├─ SELECTIVE OCR
└─ SELECTIVE MULTIMODAL EXTRACTION
↓
NORMALIZE INTO DOCUMENT IR
document
pages
sections
blocks
tables
figures
coordinates
reading order
provenance
↓
QUALITY CHECK
coverage
failed pages
OCR flags
table sanity
reading order
↓
PERSIST VERSIONED DERIVED ARTIFACT
↓
DOWNSTREAM
chunking
search
embeddings
GraphRAG
evidence-first reasoning
CORE PRINCIPLE:
DO NOT TURN EVERY DOCUMENT
INTO ONE BIG STRING.
DO NOT TURN EVERY PAGE
INTO AN OCR JOB.
DO NOT TURN EVERY IMAGE
INTO A MULTIMODAL MODEL CALL.
USE THE CHEAPEST RELIABLE EXTRACTOR,
PRESERVE STRUCTURE AND SOURCE LOCATION,
AND ESCALATE ONLY WHERE INFORMATION
WOULD OTHERWISE BE LOST.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №56 Document Parsing / OCR / Multimodal Extraction.
B–E. Existing boundary and placement. The existing conceptual boundary, class SPECIALIZED, default CONDITIONAL and owner Knowledge / Research Engine remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.