Caching — механизм повторного использования ранее вычисленного, загруженного или сгенерированного результата, если новый запрос считается достаточно эквивалентным старому и сохранённый результат всё ещё допустим по freshness, permissions, policy и version constraints.
№60 Artifact Store хранит durable objects, которые нельзя просто выбросить; №62 хранит disposable acceleration copies/results. №07 Memory сохраняет knowledge/experience для будущего reasoning; cache сохраняет result ради reuse и не является learned memory. №08 RAG определяет retrieval pipeline; cache может ускорять query/result/embedding stages, но не владеет retrieval semantics. №61 Provenance говорит, какая source/version produced result; cache использует эти версии для safe key/invalidation. №63 Retry/Fallbacks отвечает за resilience при failure; cache может быть fallback only if policy explicitly allows stale result.
Prerequisites: №08 RAG, №46 Observability, №50 Contracts, №51 Permissions, №60 Artifact Store, №61 Provenance. Forward references: №63 Retry/Circuit Breakers, №64 Quotas/Budgets, №65 Model Gateway, №66 Vector DB & Embeddings, №76 Data Governance.
REQUEST-TIME: YES — lookup/set on hot paths. CONTROL PLANE: TTLs, capacity, namespaces, versions, stale policy, bypass rules. DATA PLANE: cached values/refs and metadata. OFFLINE: warming, invalidation, capacity analysis, cache-key regression tests.
Success: hit returns a result equivalent enough for current request under current policy/version/access context. Failure: cache unavailable should usually degrade to miss, not break correctness. Idempotency: set/refresh safe to repeat. Persist: key, namespace, version, created_at, expires_at, source/version refs, tenant/security scope, value/ref. Trace: HIT/MISS/STALE/BYPASS/REFRESH reason. Security: never share result across principals/tenants unless equivalence and access are proven.
№62 не владеет durable storage, model memory, source-of-truth data, retrieval ranking, retry strategy or state. Она владеет DISPOSABLE REUSE OF PREVIOUSLY COMPUTED OR FETCHED RESULTS UNDER EXPLICIT EQUIVALENCE AND FRESHNESS RULES.
Reuse compatible result.
Lower latency / lower cost.
Compute authoritative path normally.
Then optionally populate cache.
Both paths must satisfy the same output/security/quality requirements.
Key = content hash + embedding model/version + preprocessing version.
Key = raw artifact hash + parser profile/version.
Use TTL if source changes over time and stale window is acceptable.
Key must include corpus/index version, filters, tenant, query normalization.
Exact prompt/context/model/version or carefully bounded semantic cache.
Short TTL or version key for schemas, model metadata, source descriptors.
Dangerous to reuse too long; stale ACL can leak access.
Financial/current inventory should usually read authoritative state or very short TTL.
Approval token tied to exact action and expiry; don't cache broad ALLOW.
Cached response may be wrong if policy/source versions changed.
cache_key = hash({
"namespace": "embedding",
"input_hash": "sha256:...",
"model": "embed-v3",
"preprocess": "norm-v2",
"tenant_scope": "tenant_A",
"policy_version": "p17"
})
safe hit requires:
same namespace
same relevant inputs
same transformation version
same security scope
not expired
not invalidatedВ key/version context часто входят:
parse:, embed:, retrieve:, llm:, toolread:.
embed:v3: makes old v2 cache unreachable without deleting immediately.
Separate tenant/security namespaces or include scope in key deterministically.
function get_or_compute(key):
value = cache.get(key)
if value exists and is_valid(value):
return value # HIT
result = authoritative_compute()
cache.set(
key,
result,
ttl=...
)
return result # MISS → FILL
| Pattern | Meaning | AI system note |
|---|---|---|
| CACHE-ASIDE | Application loads on miss and fills cache. | Best default: explicit and simple. |
| READ-THROUGH | Cache layer itself loads source on miss. | Useful if abstraction stable. |
| WRITE-THROUGH | Writes go authoritative store + cache together. | For current projections, but cache isn't source of truth. |
| WRITE-BEHIND | Cache accepts write, authoritative store updated later. | High risk for critical state; avoid unless carefully designed. |
Return cached value immediately.
Return stale value with age/freshness flag; asynchronously refresh.
Block stale response and perform live compute/read.
Первый caller computes; остальные ждут/shared result.
Refresh key has bounded lock/lease so crash can recover.
Randomized TTL prevents thousands of keys expiring at same second.
Serve acceptable stale value while one worker refreshes.
Reduce repeated expensive lookups when absence is likely stable for a short window.
Access can change; deny cache must be principal-scoped and often short-lived.
Timeout/5xx is not «not found». Resilience belongs to №63.
NOT_FOUND, DENIED and ERROR are not interchangeable.key = hash({
"content_hash": sha256(normalized_text),
"embedding_model": "embed-v3",
"normalizer": "norm-v2",
"dimensions": 1536
})
hit:
return vector
miss:
vector = embed(text)
cache.set(key, vector)
return vectorЕсли exact input и exact embedding configuration одинаковы, повторный результат практически не несёт новой ценности.
Лучше сохранять embedding как durable derived data/index, если он является частью corpus. Cache полезен для transient/repeated calls до durable persistence.
Raw artifact hash identifies exact source bytes.
Parser/OCR/layout profile/version included in key.
Cache can return existing derived artifact_ref instead of duplicating bytes.
retrieval_key = hash({
"query_normalized": "...",
"tenant_id": "tenant_A",
"filters": {...},
"index_version": "knowledge-v17",
"retriever": "hybrid-v4",
"top_k": 12,
"reranker": "rerank-v2",
"policy_scope": "..."
})Index/corpus version in key is often safer than trying to invalidate every possible query.
Same assembled messages/context/model/config → reuse exact prior output if freshness/security allow.
Hash task contract + evidence refs + prompt/skill/model versions.
Use only when «similar question» truly permits same answer and source freshness is bounded.
Если output зависит от user profile/history, key должен отражать relevant personalization version.
Never share private RAG/tool/model outputs across tenants by query text alone.
Global cache allowed only for explicitly public, context-independent results.
GLOBAL_PUBLIC, TENANT, USER, RUN.
Material policy changes create miss/invalidation for affected cache classes.
Even cached generation can pass current output/release guardrail before exposure.
Prefer live/current authorization check even when data payload is cached.
source://doc42/v7 invalid.
Find chunks, embeddings, retrieval caches, reports depending on v7.
Delete cache keys or bump affected generations; queue rebuild.
Хороший общий default для memory caches.
Useful when hot set stable and recency alone insufficient.
Entries naturally disappear based on freshness policy.
Prioritize entries with high recomputation cost × reuse probability.
Popular static FAQ, common embeddings, high-traffic metadata.
Warm a small top set after version change if miss storm is expensive.
Most possible prompts/queries may never be requested.
In-process memory, small hot objects, very low latency. Lost on restart; per-process inconsistency tolerated only for cacheable data.
Redis/DB/object mapping shared by multiple processes. Higher latency but better hit rate.
| Backend | Good for | Notes |
|---|---|---|
| IN-PROCESS MEMORY | Tiny local hot cache. | No shared state, disappears on restart. |
| POSTGRES | Low/medium throughput shared cache, version mappings. | Often enough initially; not ideal for very high QPS hot cache. |
| REDIS-LIKE | Fast shared TTL/LRU cache. | Great when performance justifies extra service. |
| ARTIFACT STORE + INDEX | Large cached outputs. | Cache key maps to artifact_ref; don't put giant blobs in Redis. |
Short timeout, treat as miss if authoritative path healthy.
Usually do not fail user request because cache write failed.
Serve only if stale policy allows; otherwise live path.
Validation/hash/schema mismatch → delete/bypass and recompute.
By namespace/key class/tenant/workload.
not found / expired / version mismatch / bypass / access mismatch.
Estimated compute/tool/model time avoided.
Tokens/API/GPU calls avoided minus cache infra cost.
Count/age by policy class.
Pressure/capacity by namespace.
Concurrent computes for same key / singleflight waits.
Cache hit returned semantically/security-invalid result. Target: zero.
Equivalent exact inputs yield same key; meaningful differences change key.
Prompt/model/parser/index change invalidates expected entries.
Same query across tenants/users cannot leak cached private result.
Near-phrases with different intent must not reuse answer.
Source change/misfire invalidates or ages result according to policy.
System falls back to authoritative path.
100 simultaneous misses create bounded recomputations.
Latency/cost reduction exceeds operational complexity.
Hits / lookups by namespace and workload class.
p95 request/stage reduction due to cache hits.
Model/API/GPU/tool cost avoided minus cache cost.
Semantically/security-invalid hits. Target: effectively zero.
How often stale results are served and average age.
Misses caused by model/prompt/source/index generation changes.
Capacity pressure and churn.
Duplicate recomputations prevented per hot key.
cache/ ├── keys.py ├── service.py ├── policy.py ├── invalidation.py ├── singleflight.py └── tests/ cache_entries( namespace text, cache_key text, tenant_scope text, value_json jsonb, artifact_ref text, created_at timestamptz, expires_at timestamptz, source_version text, config_version text, metadata_json jsonb, primary key(namespace, cache_key) ) get(key, context): if BYPASS: miss entry = load if absent: miss if expired: miss/stale-policy if access/version invalid: miss return hit set: bounded size ttl namespace/version scope metadata
Добавлять Redis или semantic cache только после измеренного bottleneck/ROI.
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да, после появления повторяемой дорогой работы и измеренного benefit. Не обязательный первый компонент. |
| Separate Component? | YES логически. Физически сначала может быть library + local/Postgres cache. |
| Минимум 80% ценности? | Exact key, namespace/version, TTL/freshness, tenant scope, singleflight, invalidation, hit/miss/cost metrics, bypass. |
| Когда overkill? | Semantic cache, distributed Redis cluster и multi-level invalidation для системы с низким повторением запросов. |
| Trigger? | Repeated identical/equivalent expensive transformations or reads create material latency/cost load. |
| Как измерить uplift? | Hit rate, latency/cost saved, false-hit incidents, stale response rate, cache infra cost, duplicate work prevented. |
| Можно ли rule/tool/code вместо LLM-agent? | Да, и нужно. Exact caching is deterministic infrastructure. LLM/embeddings can participate only in optional semantic similarity layer. |
Cache miss/outage must preserve correctness.
Every meaning-changing input/version belongs in key/context.
Start with deterministic reuse; semantic matching needs eval proof.
Model/prompt/parser/index changes should make old results unreachable.
GLOBAL_PUBLIC / TENANT / USER / RUN.
Use source/config versions and events where meaning changes.
Singleflight/lease/jitter for hot expensive keys.
Hit rate without correctness can reward dangerous behavior.
Cache where saved latency/cost exceeds invalidation/infra complexity.
REQUEST / PIPELINE STAGE
↓
BUILD CACHE CONTEXT
namespace
normalized input
tenant/user scope
model/prompt/tool/parser version
corpus/index version
policy version
↓
HASH → CACHE KEY
↓
LOOKUP
├─ MISS
│ ↓
│ AUTHORITATIVE COMPUTE
│ ↓
│ VALIDATE RESULT
│ ↓
│ CACHE SET
│ ↓
│ RETURN
│
├─ HIT
│ ↓
│ CHECK:
│ not expired
│ current access allowed
│ versions compatible
│ source freshness acceptable
│ schema valid
│ ↓
│ RETURN
│
└─ STALE
↓
policy:
serve + refresh
OR miss + recompute
INVALIDATION:
source changed
parser changed
model changed
prompt changed
index rebuilt
policy changed
↓
bump namespace/version
OR targeted invalidation
↓
optional warm
SAFE FIRST CANDIDATES:
content_hash + parser_version
content_hash + embedding_version
stable API read + short TTL
retrieval + exact corpus/index version
exact model request + exact context/config
HIGH-RISK:
permissions
current financial state
approvals
volatile external facts
semantic reuse without evals
CORE PRINCIPLE:
CACHE IS A
DISPOSABLE ACCELERATOR.
THE HARD QUESTION IS NOT
"WHERE DO WE STORE IT?"
THE HARD QUESTION IS:
"WHEN ARE TWO REQUESTS
REALLY EQUIVALENT,
AND FOR HOW LONG
IS THE OLD RESULT
STILL SAFE TO REUSE?"
IF THAT CANNOT BE ANSWERED
PRECISELY—
DO NOT CACHE THE RESULT.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №62 Caching.
B–E. Existing boundary and placement. The existing conceptual boundary, class PRODUCTION, default CONDITIONAL and owner Production Fabric remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.