Embedding — числовое представление объекта в векторном пространстве, где геометрическая близость приблизительно отражает семантическую похожесть. Vector DB / Vector Index — инфраструктура, которая хранит эти векторы и быстро ищет ближайшие объекты по similarity, обычно вместе с metadata filters.
№08 RAG владеет всей retrieval-augmented generation architecture; №66 — только dense embeddings/index/search mechanics. №27 Agentic RAG решает, когда/как adaptive retrieval повторять, переписывать запрос и reroute; №66 выполняет конкретный dense/hybrid search. №55 Ingestion доставляет source versions; №56 Parsing создаёт structured blocks/chunks inputs; №66 индексирует уже подготовленные units. №61 Provenance владеет source/derivation graph; vector index хранит refs/hash/version, но не становится lineage system. №67 Knowledge Graph & GraphRAG владеет explicit entities/relations/graph traversal; vector similarity ≠ graph relation.
Prerequisites: №08 RAG, №25 Evidence-First, №27 Agentic RAG, №46 Observability, №50 Contracts, №55 Ingestion, №56 Parsing, №61 Provenance, №62 Caching. Forward references: №67 Knowledge Graph & GraphRAG, №76 Data Governance & Privacy.
REQUEST-TIME: query embedding + search + filter/rerank. CONTROL PLANE: embedding model/version, metric, index type, namespaces, filter schema, recall/latency targets. DATA PLANE: vectors, chunk refs, metadata, index structures. OFFLINE: embedding generation, index build/rebuild, migration, retrieval evals, compaction and deletion reconciliation.
Success: search returns authorized, version-compatible candidate refs from intended corpus/index. Retryable: transient index/network failure. Permanent: dimension mismatch, unsupported namespace, invalid filters, missing index generation. Idempotency: upsert by stable chunk/vector identity + embedding version. Persist: chunk_ref, source/version/hash, embedding model/version/dimensions, vector, metadata, tenant/access scope, index generation. Trace: query→embedding→filter→search→rank→rerank. Security: ACL/tenant filter enforced at retrieval boundary, not after generation.
№66 не владеет document parsing, chunking semantics as a whole, RAG orchestration, answer generation, evidence verification, memory, graph traversal or source of truth. Она владеет EMBEDDING REPRESENTATION, VECTOR INDEXING AND SIMILARITY-BASED CANDIDATE RETRIEVAL.
Text chunk, title, query, image, audio or multimodal object depending on embedding model.
Например, hundreds/thousands of floating-point dimensions. Coordinates individually usually have no human-readable meaning.
«Условия концессии...»
embedding_model=v3 → vector D
Vectors are comparable under the model's intended similarity metric.
«Какие обязательства у концессионера?»
embedding_model=v3 → vector Q
{
"vector_id": "VEC-...",
"object_ref": "chunk://doc123/v7/ch42",
"tenant_id": "tenant_A",
"content_hash": "sha256:...",
"embedding": {
"model": "embed-v3",
"dimensions": 1536,
"normalization": "L2",
"version": "2026-08"
},
"index": {
"namespace": "knowledge-prod",
"generation": "g17",
"metric": "COSINE"
},
"metadata": {
"document_id": "doc123",
"source_version": "v7",
"language": "ru",
"acl_group": "project-X"
}
}| Metric | Meaning | Notes |
|---|---|---|
| COSINE SIMILARITY | Angle/direction similarity. | Very common for normalized text embeddings. |
| DOT PRODUCT | Vector alignment weighted by magnitude. | Often used when model is trained for it; normalization matters. |
| EUCLIDEAN / L2 | Geometric distance. | Useful depending on model/index assumptions. |
High recall/low latency, memory-heavy, popular general-purpose option.
Search subset of vector space; tunable probes vs recall.
Lower memory/storage, possible recall loss; useful at very large scale.
Lower CPU/latency, potentially misses relevant vectors.
More graph exploration/probes, higher latency.
Larger K improves coverage but adds reranking/context cost.
Retrieve broader candidate set, rerank with stronger lexical/cross-encoder/LLM judge.
Chunk matches keyword/phrase but lacks enough surrounding meaning.
One vector represents many unrelated concepts; retrieval becomes vague.
Headings/paragraphs/tables/sections often beat blind fixed-token slicing.
Retrieve child chunk but expand to parent section/page for answer context.
Good when chunk is self-contained.
Heading/document title can disambiguate short local passages.
Add bounded generated/structured context only if evals show uplift; retain source provenance.
Good for paraphrases, conceptual similarity and natural-language queries.
Good for names, codes, numbers, rare terms, exact legal/technical phrases.
Can leak through scores/logs and destroys recall if most top hits are inaccessible.
Tenant/source/access/language/date filters participate in retrieval execution.
Retrieved ref is re-authorized before content is loaded into model context.
chunk_id, document_id, source_ref, tenant_id.
source_version, content_hash, index_generation.
ACL groups, classification, project/team scope.
page, section, heading path, block type.
created_at, valid_from/to, published_at where meaningful.
Language/locale can improve model/query routing.
internal, official, user-uploaded, web, generated.
Deleted/retracted versions cannot remain retrievable.
| Stage | Purpose |
|---|---|
| ANN / DENSE SEARCH | Cheaply retrieve broad semantically plausible candidates. |
| LEXICAL / FILTER FUSION | Recover exact matches and business constraints. |
| RERANKER | Evaluate query-document relevance more precisely over 20–100 candidates. |
| EVIDENCE CHECK | Determine whether top material actually supports requested fact/claim. |
Embed user/task query as-is after light normalization.
Resolve references, remove conversational noise, add domain terms when needed.
Generate several distinct retrieval queries, union candidates, dedupe, rerank.
Ingestion/governance marks source deleted or access revoked.
Find chunks/embeddings/index entries derived from source version.
Make entries immediately ineligible; physical compaction can happen later.
knowledge_index: generation g17 ← ACTIVE generation g18 ← BUILDING BUILD g18: new embedding model new chunking profile new metadata schema full / incremental load retrieval eval ACL checks count/hash reconciliation PROMOTE: active_generation = g18 ROLLBACK: active_generation = g17 RETIRE: delete g17 later after safety window
Build new vectors/index generation in parallel, evaluate, then switch.
Do not casually combine vectors from model v2 and v3 in one nearest-neighbor space.
Embedding/query/retrieval cache keys include model/index generation.
Detect identical normalized chunk content.
Same press release copied across many pages can dominate semantic results.
Limit per document/source/domain before final rerank where useful.
Dedup physically/logically without losing source authority/version metadata.
Tenant_id mandatory in every row/query; robust filter enforcement required.
Operational overhead grows with tenant count.
Separate DB/cluster/index for strict regulatory or enterprise isolation.
| Layer | What is stored/retrieved | Purpose |
|---|---|---|
| VECTOR INDEX | Vectors + refs + metadata. | Approximate semantic candidate retrieval. |
| RAG KNOWLEDGE | External/source-backed corpus. | Bring relevant knowledge into current context. |
| MEMORY | Retained facts/preferences/episodes/procedures. | Persist experience/user/system knowledge over time. |
Find semantically similar chunks/entities without explicit relation type.
Company A OWNS Project B, person C WORKS_FOR organization D, project CONNECTS city X→Y.
Did top-K contain at least one truly relevant item?
How early does first relevant item appear?
Useful when relevance has levels, not binary label.
How many returned items are actually relevant?
Unauthorized result rate. Target 0.
Retrieval of superseded/deleted versions.
p50/p95 under realistic filters and corpus size.
Embedding + search + rerank + context token cost.
Measure independently with relevance labels.
Reranker/context builder can drop useful candidates.
Answer may hallucinate despite perfect retrieval.
Same content hash and embedding config → reuse vector.
Useful for repetitive workloads; scope by embedding model/version.
Cache only with tenant/filter/index generation in key.
By namespace/tenant/model/generation.
Lag source update → searchable vector.
With realistic filters/top-K.
Offline Recall@K by query segment.
Queries with poor candidate coverage.
Superseded/deleted versions returned.
Unauthorized candidate/result rate.
Embedding failures, dimension mismatch, orphan vectors, generation completeness.
Share of queries where relevant evidence is present in top-K.
How high the first relevant result appears.
Noise level among top results.
Filtered ANN/hybrid latency at production scale.
Source update → searchable new vector delay.
Target: zero.
Superseded/deleted entries still retrievable.
Answer/eval uplift versus lexical-only/no-retrieval baselines.
vector_index/
├── embeddings.py
├── repository.py
├── search.py
├── filters.py
├── hybrid.py
├── rerank.py
├── migrations.py
├── reconcile.py
└── tests/
vector_entries(
vector_id uuid primary key,
chunk_ref text,
tenant_id text,
source_ref text,
source_version text,
content_hash text,
embedding_model text,
embedding_version text,
dimensions int,
index_generation text,
vector vector(...),
metadata_json jsonb,
deleted boolean,
created_at timestamptz,
unique(
chunk_ref,
embedding_version,
index_generation
)
)
search(query, ctx):
embed query with matching model
apply tenant/ACL/source filters
dense top_k = 30
lexical top_k = 30
fuse/dedupe
rerank
return refs + scores + provenanceНе нужен отдельный vector cluster, если corpus/load comfortably fits PostgreSQL and latency target.
| Signal | Potential upgrade |
|---|---|
| Millions/billions of vectors + strict latency | Dedicated ANN engine/vector database. |
| High concurrent filtered search | Distributed/sharded vector index with robust metadata filtering. |
| Memory pressure | Quantized/compressed index or disk-oriented ANN. |
| Complex hybrid retrieval | Dedicated search stack / coordinated lexical+dense engine. |
| Frequent index migrations | Automated generation build/canary/promote/rollback pipeline. |
| Entity/relation reasoning becomes important | Add №67 Knowledge Graph / GraphRAG rather than forcing vectors to represent explicit relationships. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да, если dense semantic retrieval показывает uplift. Не нужен автоматически для любой AI-системы. |
| Separate Component? | YES логически. Физически MVP может жить в PostgreSQL рядом с corpus metadata. |
| Минимум 80% ценности? | Versioned embeddings, stable chunk refs, ACL filters, coherent index generations, hybrid retrieval, reranking, deletion sync, retrieval evals. |
| Когда overkill? | Dedicated vector cluster для тысяч chunks, которые exact/Postgres search находит за milliseconds. |
| Trigger? | Keyword retrieval misses paraphrases/conceptual matches, corpus scale/search latency requires dense index, or evals prove semantic retrieval uplift. |
| Как измерить uplift? | Recall@K, MRR, answer quality uplift, retrieval latency, stale/ACL error rate, end-to-end cost and context efficiency. |
| Можно ли rule/tool/code вместо LLM-agent? | Да. Embedding/index/search are deterministic infrastructure. LLM may rewrite queries or rerank, but Vector DB itself requires no agent. |
Source of truth remains source/artifact/document stores.
Embedding model/version/dimensions/index generation are mandatory.
Tenant/ACL/data scope belongs in retrieval eligibility.
Use lexical + dense when exact identifiers and semantic paraphrases both matter.
ANN is candidate generation, not final evidence judgment.
Garbage units in → faithfully retrieved garbage out.
Revoked/deleted sources become ineligible immediately.
Recall/rank/security/freshness before blaming generation.
Use specialized vector DB only after Postgres/exact baseline stops meeting requirements.
INGESTED SOURCE
↓
PARSED STRUCTURE
↓
CHUNKS / RETRIEVAL UNITS
chunk_ref
source_ref
source_version
content_hash
ACL / tenant
↓
EMBEDDING PIPELINE
exact representation text
embedding model
embedding version
dimensions
↓
VECTOR
↓
VECTOR INDEX
namespace
generation
metric
ANN structure
metadata filters
↓
QUERY
↓
QUERY NORMALIZATION
↓
QUERY EMBEDDING
SAME VECTOR SPACE
↓
FILTER ELIGIBLE CORPUS
tenant
ACL
source
date
language
type
↓
DENSE SEARCH
+
LEXICAL SEARCH
↓
FUSION / DEDUPE
↓
RERANK
↓
TOP EVIDENCE CANDIDATES
refs
scores
provenance
↓
RAG CONTEXT BUILDER
↓
GENERATION
↓
VERIFICATION
UPDATE PATH:
source v8
↓
parse/chunk
↓
embed changed content
↓
upsert new generation/version
↓
retire old entries
DELETE PATH:
source revoked/deleted
↓
lineage descendants
↓
tombstone vector entries
↓
physical compaction later
MIGRATION:
g17 ACTIVE
g18 BUILDING
↓
eval
↓
PROMOTE g18
↓
rollback if needed
BOUNDARIES:
VECTOR DB
= semantic candidate index
RAG
= retrieval-augmented answering pipeline
MEMORY
= retained knowledge/experience semantics
KNOWLEDGE GRAPH
= explicit entities + typed relations
CORE PRINCIPLE:
VECTOR SIMILARITY MEANS:
"THESE OBJECTS LOOK
SEMANTICALLY CLOSE
IN THIS MODEL'S SPACE."
IT DOES NOT MEAN:
"THIS IS TRUE."
"THIS IS AUTHORIZED."
"THIS IS FRESH."
"THIS SUPPORTS THE CLAIM."
"THIS IS THE BEST SOURCE."
THOSE DECISIONS BELONG
TO THE LARGER RETRIEVAL,
POLICY, PROVENANCE
AND VERIFICATION SYSTEM.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №66 Vector DB & Embeddings.
B–E. Existing boundary and placement. The existing conceptual boundary, class SPECIALIZED, default CONDITIONAL and owner Knowledge / Research Engine remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.