Provenance отвечает на вопрос: «откуда взялся этот факт, файл, chunk, ответ или решение?» Lineage отвечает: «через какие преобразования и версии компонентов он прошёл?»
№46 Observability & Tracing фиксирует execution telemetry: run/span/tool/model latency/errors; №61 фиксирует происхождение и transformation chain данных/artifacts/claims. №25 Evidence-First использует lineage, чтобы связать material claim с evidence. №45 Verification проверяет конкретный result и может читать provenance; №61 сама не решает PASS/FAIL. №55 Ingestion создаёт source/version records; №56 Parsing создаёт derived blocks/tables; №60 Artifact Store хранит bytes и parent refs; №61 связывает всё в derivation graph. №76 Governance позже использует lineage для deletion/retention/impact analysis.
Prerequisites: №25 Evidence-First, №45 Verification, №46 Observability, №50 Contracts, №55 Ingestion, №56 Parsing, №60 Artifact Store. Forward references: №66 Vector DB, №67 GraphRAG, №69 Distributed Reliability, №76 Data Governance & Privacy.
REQUEST-TIME: YES — capture retrieval/evidence/claim edges while answering. CONTROL PLANE: lineage schemas, transformation types, retention, provenance depth. DATA PLANE: entities, versions, edges, evidence links. OFFLINE: audits, impact analysis, replay/rebuild, quality/eval investigations.
Success: material output can resolve to its declared source/evidence and transformation history. Retryable: transient lineage-write failure if buffered/outboxed. Permanent: invalid/missing referenced entity/version. Idempotency: deterministic edge/event IDs or unique transformation keys. Persist: entity IDs, versions/hashes, edge type, transform version/config refs, timestamps, run/trace IDs, tenant/access metadata. Security: lineage query cannot reveal inaccessible source metadata. Operations: missing lineage for high-risk output may be release-blocking.
№61 не владеет execution tracing, correctness verification, source storage, artifact bytes, vector retrieval, business audit policy или hidden reasoning. Она владеет EXTERNALIZED ORIGIN, DERIVATION AND EVIDENCE RELATIONSHIPS BETWEEN SYSTEM OBJECTS.
Source URL/object, external ID, author/system, fetched version, original artifact, evidence page/block, retrieval result.
Fetched → parsed → table extracted → chunked → embedded → retrieved → summarized → verified → released.
| Question | №46 Observability | №61 Provenance / Lineage |
|---|---|---|
| Что система делала? | Run/span/tool/model events. | References run that produced transformation. |
| Сколько заняло? | Latency/tokens/cost/errors. | Usually not primary concern. |
| Откуда взялся answer? | Can show retrieval/tool spans. | Explicit answer/claim → evidence/source chain. |
| Какой parser/model version? | Span attributes may record it. | Version is part of durable derivation record. |
| Можно ли rebuild affected outputs? | Trace alone often insufficient. | Descendant/ancestor graph enables impact/rebuild analysis. |
run_id / trace_id / span_id, но lineage должна пережить expiration sampled traces, если reproducibility/audit требует более долгого хранения.Drive file, webpage, DB row/snapshot, message, API record, user upload.
PDF/image/audio/report/archive/page image/dataset.
Page, heading, paragraph, table, figure, structured field.
Text/structured chunk built from one or several blocks.
Vector, entity extraction, normalized record, classification.
Query, rank, score, selected chunk/version.
Atomic statement in answer/report that should be supportable.
Report/document/chart/file produced from inputs.
Optional: released/approved route/result tied to evidence and policy.
| Edge | Meaning | Example |
|---|---|---|
| FETCHED_FROM | Artifact/record obtained from external source. | raw PDF → Drive file v42. |
| PARSED_FROM | Structured representation extracted from artifact. | Document IR → raw PDF. |
| DERIVED_FROM | Generic deterministic/semantic derivation. | table CSV → table block. |
| CHUNKED_FROM | Retrieval chunk assembled from blocks. | chunk 17 → blocks 20–25. |
| EMBEDDED_FROM | Vector produced from exact chunk/version. | embedding v3 → chunk hash H. |
| RETRIEVED_FROM | Run selected entity from index/source. | retrieval hit → chunk 17. |
| SUPPORTED_BY | Claim cites/supports evidence entity. | claim C4 → block/page P12. |
| CONTAINS | Output contains claim/artifact component. | answer A → claims C1..C5. |
| TRANSFORMED_BY | Result produced using transformation config/version. | summary → model/prompt/tool ref. |
| SUPERSEDES | New logical version replaces prior version for current use. | doc v43 → doc v42. |
RELATED_TO. Он почти бесполезен для audit, rebuild и impact analysis.{
"lineage_event_id": "LIN-...",
"relation": "PARSED_FROM",
"output": {
"ref": "document-ir://doc123/v3",
"version": "3"
},
"inputs": [
{
"ref": "artifact://tenant_A/raw123",
"sha256": "sha256:..."
}
],
"transform": {
"type": "document.parse",
"implementation": "parser-profile-v3",
"config_hash": "sha256:..."
},
"run_id": "RUN-...",
"trace_id": "TRACE-...",
"tenant_id": "tenant_A",
"created_at": "..."
}Producer of a derived object should emit its lineage record when the object becomes durable.
Не восстанавливать lineage позже из догадок.
updated_at/version/content hash/external revision.
Code/config/profile version or hash.
Provider/model/version where available; routing decision captured.
Prompt/skill/config version or immutable hash.
Tool/connector implementation/version and structured arguments reference.
Relevant policy/approval version for released actions when material.
Embedding/index namespace/version/snapshot.
Artifact hash/answer version/claim IDs.
source → artifact → IR → chunk → embedding → report. Может жить месяцами/годами и использоваться для rebuild/deletion/audit.
Run R7 queried index, retrieved chunks X/Y, called model M, verifier V, released answer A. Связано с traces/spans №46.
{
"claim_id": "CLM-4",
"claim_text_hash": "sha256:...",
"relation": "SUPPORTED_BY",
"evidence": {
"ref": "block://doc123/v3/page12/b7",
"source_ref": "source://drive/file456/v42"
},
"support": {
"type": "DIRECT",
"scope": "claim",
"verified": true
},
"retrieval": {
"run_id": "RUN-...",
"rank": 2,
"query_hash": "sha256:..."
}
}Verification policy determines which relations are sufficient for release.
Revenue 2025, Revenue 2026 from source table.
Code/tool formula computes growth rate; code hash + inputs + result captured.
«Выручка выросла на X%» supported by calculation + source cells.
Source A establishes project/company/object identity.
Source B uses resolved identifier to obtain financial/status data.
Claim records all material evidence parents plus transformation/research run.
Source text remains untrusted instruction-wise.
Official, internal, secondary, user-provided, generated, inferred.
Fetched_at, source version, validity interval when known.
Tenant/data classification/ACL propagate to descendants.
| Derived object | Default access rule |
|---|---|
| Parsed block from private document | Inherit/restrict to source document access. |
| Embedding from confidential chunk | Same tenant/classification boundary; retrieval filters apply. |
| Summary combining public + private sources | At least as restrictive as the most restrictive material input, unless explicit declassification/redaction policy exists. |
| Public report derived from private analysis | Requires deliberate release/redaction/policy decision; not automatic inheritance relaxation. |
Given report/claim/artifact, walk backwards to sources, transformations and versions.
Given source/chunk/model/config version, find all derived indexes/reports/claims that may need rebuild/invalidation.
source://doc/42/v7 ↓ PARSED_FROM document-ir://42/p3 ↓ CHUNKED_FROM chunk://42/17 ↓ EMBEDDED_FROM embedding://42/17/e3 ↓ RETRIEVED_FROM run://R18/hit4 ↓ SUPPORTED_BY claim://C9 ↓ CONTAINS report://Q4 SOURCE v7 RETRACTED → mark descendants potentially stale → invalidate retrieval/index if required → queue rebuild/reverification → notify owners of released artifacts if policy requires
{
"output_ref": "answer://RUN-42/final",
"relation": "GENERATED_FROM",
"inputs": [
"task://...",
"context://RUN-42/v1",
"evidence-set://RUN-42/e3"
],
"transform": {
"model_route": "router-decision://...",
"model": "provider/model-id",
"prompt_ref": "prompt://answer/v7",
"skill_refs": ["skill://research/v4"],
"structured_output": "answer.v3"
},
"run_id": "RUN-42"
}Нужно сохранять:
Не требуется хранить приватное внутреннее рассуждение модели.
Tool name/version, sanitized arguments/ref, tenant, run/span.
Structured result or artifact_ref + provider/source IDs and timestamps.
Claim may be supported by tool result, calculation artifact or fetched external object.
% material claims/derived artifacts with complete provenance.
No dangling source/artifact/block references.
Link points to exact version/hash, not generic latest.
Verification checks that cited evidence actually supports claim.
Evidence current enough for claim's time sensitivity.
Official/primary/secondary/user-generated classification where useful.
Lineage does not expose inaccessible source metadata.
No unexplained gap between source and released output.
Claim C4 declares evidence E2/E5 and transform T1.
Checks evidence exists, supports claim, calculation correct, source acceptable.
PASS / REVISE / FAIL / UNCERTAIN, with verification record linked back to claim.
Correct reasoning from stale/incorrect source.
Corpus contains evidence, retrieval hit missed it.
Model misread/ignored/miscombined correct evidence.
Parsed blocks, embeddings, caches, summaries and artifacts may contain copied data.
Delete, tombstone, de-identify, retain under legal hold, or reprocess.
Lineage can report which descendants were removed/retained and why.
Новая версия/edge добавляется, старые records остаются для audit where retention permits.
Mark previous version superseded/retracted instead of mutating facts invisibly.
Sensitive fields may need removal; preserve non-sensitive audit metadata according to policy.
lineage_entities( entity_ref text primary key, entity_type text, tenant_id text, logical_id text, version text, content_hash text, source_time timestamptz, created_at timestamptz, metadata_json jsonb ) lineage_edges( edge_id uuid primary key, relation text, from_ref text, to_ref text, transform_type text, transform_version text, config_hash text, run_id text, trace_id text, created_at timestamptz, metadata_json jsonb, unique(...) ) indexes: from_ref to_ref tenant_id relation run_id
PostgreSQL recursive CTEs обычно достаточно для:
Graph DB имеет смысл позже при очень больших/deep lineage graphs и graph-heavy interactive analysis.
Derived record + lineage edge commit in same DB transaction when they live together.
Producer commits output + lineage event intent; relay writes central lineage service asynchronously.
Periodic checks detect durable outputs missing required lineage and backfill from deterministic metadata where possible.
Даже факт существования private document/customer can be sensitive.
Ancestor/descendant queries respect current tenant/principal permissions.
Store secret refs or hashes where possible, not credentials/raw confidential payload.
Lineage retention/classification follows governance and audit needs.
% material outputs/claims with required lineage.
% lineage refs resolving to existing authorized entities.
% externally verifiable claims mapped to evidence.
Edges referencing missing entity/version.
% source links pinned to exact version/hash rather than latest.
Time to identify descendants of faulty/retracted source.
% sampled pipelines reconstructable from recorded refs/configs.
Released output where material origin cannot be established.
provenance/
├── entities.py
├── edges.py
├── capture.py
├── queries.py
├── claims.py
├── access.py
├── reconcile.py
└── tests/
MVP capture points:
ingestion:
source → raw artifact
parsing:
raw artifact → document IR/block
chunking:
blocks → chunk
embedding:
chunk → embedding/index record
retrieval:
query/run → retrieval hit → chunk
answer:
answer → claim
claim → evidence
generated files:
inputs → output artifact
Queries:
ancestors(ref)
descendants(ref)
evidence_for(claim_id)
outputs_for_run(run_id)No graph DB, no LLM provenance agent, no enterprise catalog required.
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да. Минимальный provenance слой нужен для production knowledge/quality systems, особенно если используются RAG, external sources, generated artifacts и evals. |
| Separate Component? | YES логически. Может быть набором общих таблиц/library hooks, а не отдельным microservice. |
| Минимум 80% ценности? | Stable refs + versions/hashes + typed edges + claim/evidence links + transform/run metadata + ancestor/descendant queries. |
| Когда overkill? | Enterprise graph/catalog/semantic ontology до того, как базовые source/artifact IDs и derivation capture дисциплинированы. |
| Trigger? | По умолчанию ON; depth increases for external knowledge, high-risk outputs, regulated data, complex transformations and released artifacts. |
| Как измерить uplift? | Time to debug wrong answer, citation/evidence accuracy, impact-analysis time, replay success, lineage gap rate, stale-output incidents. |
| Можно ли rule/tool/code вместо LLM-agent? | Да — и предпочтительно. Capture is deterministic metadata. LLM может помочь semantic claim/evidence mapping, но не должен быть единственным source of provenance truth. |
Producer writes lineage when derived object becomes durable.
Source/ref without version/hash is weak provenance.
PARSED_FROM/CHUNKED_FROM/SUPPORTED_BY beat generic RELATED_TO.
Material assertions should have machine-addressable support relations.
run/trace IDs link execution to durable lineage.
Parser/chunker/model/prompt/tool/config versions explain derivation.
Ancestors and descendants are first-class operations.
Lineage metadata must not leak protected resources.
Record externalized inputs, evidence, actions and outputs — not private CoT.
EXTERNAL SOURCE
source_id
source_version
fetched_at
authority / trust
↓ FETCHED_FROM
RAW ARTIFACT
artifact_ref
sha256
↓ PARSED_FROM
DOCUMENT IR / BLOCKS
parser_version
page / bbox
↓ CHUNKED_FROM
RETRIEVAL CHUNKS
chunker_version
↓ EMBEDDED_FROM
INDEX / EMBEDDING
embedding_model
index_version
↓ RETRIEVED_FROM
RUN-SPECIFIC EVIDENCE SET
query
rank
score
↓ SUPPORTED_BY
CLAIMS
claim_id
evidence relation
↓ CONTAINS
ANSWER / REPORT / ARTIFACT
output version/hash
PARALLEL EXECUTION LINK:
run_id
trace_id
span_id
model route
prompt/skill version
tool result refs
verifier result
↓
DURABLE LINEAGE GRAPH
ASK BACKWARDS:
"Where did this claim come from?"
"Which exact source version?"
"Which parser/model/config?"
"What evidence supported it?"
ASK FORWARDS:
"What depends on this source?"
"What becomes stale if it is retracted?"
"What must be reprocessed if parser v2 is wrong?"
"What outputs contain deleted private data?"
CORE PRINCIPLE:
OBSERVABILITY TELLS US
WHAT THE SYSTEM DID.
PROVENANCE TELLS US
WHAT THE RESULT CAME FROM.
LINEAGE TELLS US
HOW IT WAS TRANSFORMED.
VERIFICATION TELLS US
WHETHER THAT RESULT
SHOULD BE TRUSTED.
DO NOT STORE ONLY THE ANSWER.
STORE ENOUGH EXTERNALIZED
ORIGIN + VERSION + TRANSFORMATION
TO EXPLAIN, REBUILD,
INVALIDATE AND VERIFY IT.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №61 Provenance / Lineage.
B–E. Existing boundary and placement. The existing conceptual boundary, class PRODUCTION, default ON and owner Knowledge / Research Engine + Quality Engine remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.