Data Governance & Privacy — системный слой, который отвечает на вопросы: какие данные у нас есть, откуда они пришли, зачем мы их используем, где они могут храниться и обрабатываться, кто к ним имеет доступ, сколько их хранить, как удалить и какие производные данные тоже затронуты.
№51 Permissions & Secrets owns identity, scopes, credentials and least privilege; №76 defines data-use, retention, residency and privacy constraints that permissions enforce. №52 Agent Security owns injection/exfiltration threat model; governance defines which data is sensitive and where it is allowed to flow. №61 Provenance records lineage; governance uses lineage for deletion/impact analysis but does not replace it. №60 Artifact Store stores bytes under lifecycle policy; №76 defines retention/deletion classes. №46 Observability logs telemetry; governance defines what may appear in logs and for how long. №65 Gateway routes models; governance supplies provider/region/data-class eligibility constraints.
Prerequisites: №07 Memory, №08 RAG, №10 State, №45 Verification, №46 Observability, №48 Policies, №50 Contracts, №51 Permissions, №52 Security, №55 Ingestion, №60 Artifact Store, №61 Provenance, №62 Caching, №65 Gateway, №66 Vector DB, №73 Multimodal, №75 Realtime. Forward: №77 Production AI Architecture synthesizes governance across whole system.
CONTROL PLANE: taxonomy, purposes, retention schedules, residency, provider eligibility, deletion rules, legal/policy mappings, owners/stewards. DATA PLANE: classification/subject/purpose tags propagated with objects, queries, artifacts and model calls. RUNTIME: allow/deny/minimize/redact/route decisions before data crosses boundaries. OFFLINE: inventory scans, retention sweeps, deletion jobs, lineage impact analysis, compliance evidence, stale/orphan data review.
Success: every governed object has enough metadata to determine allowed use, location, retention and deletion path. Fail closed: unknown classification/purpose/provider eligibility on sensitive flows. Retryable: deletion/index cleanup/backups reconciliation jobs. Permanent: policy prohibits requested processing or destination. Evidence: immutable-ish policy version, decision reason, affected refs, deletion status, exceptions and approvals.
№76 не является юридической консультацией и не кодирует одну конкретную юрисдикцию. Она не владеет authentication, encryption algorithms, security monitoring, provenance capture or storage engine internals. Она владеет DATA LIFECYCLE POLICY, PRIVACY CONSTRAINTS AND MACHINE-ENFORCEABLE DATA HANDLING RULES.
User upload, CRM, web, email, sensor, generated artifact, vendor, public source.
Business/domain owner and technical steward.
Support, search, personalization, fraud detection, analytics, evaluation, training candidate etc.
Database, object store, vector index, logs, backup, provider endpoint, local cache.
{
"data_asset_id": "DA-...",
"tenant_id": "T-...",
"data_type": "document",
"source_ref": "source://...",
"owner": "team://support",
"classification": "CONFIDENTIAL",
"subject_refs": ["subject://..."],
"purposes": ["CUSTOMER_SUPPORT"],
"allowed_regions": ["EU"],
"retention_class": "SUPPORT_180D",
"provider_policy": "PRIVATE_OR_APPROVED",
"training_use": "DENY",
"created_at": "...",
"expires_at": "...",
"lineage_root_ref": "prov://..."
}Customer document.
OCR text, chunks, embeddings, summary, extracted fields, cached answer.
Derived objects inherit or recompute classification; never silently downgrade because format changed.
Use ticket/customer context to answer the current support task.
May require separate user/product basis and opt-out controls.
Production samples should enter eval datasets only under explicit governance.
Distinct purpose with higher persistence and downstream propagation; never assume allowed.
Emails, phone numbers, IDs, account patterns, secrets via rules/checksums.
Useful for broader entity detection but can have false positives/negatives.
CRM field semantics may be more reliable than scanning text after extraction.
Unknown sensitive status should not automatically become unrestricted.
Delete/black out fields that have no task value.
“Customer_17” preserves relational identity without exposing real name to a downstream processor.
Trusted host can rehydrate output when authorized; model never sees original identifier.
Age + rare job + location + date can identify a person without explicit name.
Stories, addresses and unique events can re-identify.
Treat “anonymous” as an evaluated state with defined assumptions, not a regex outcome.
Temporary prompts/caches/session audio can expire quickly.
Retention can start from case closure/account deletion rather than creation time.
Explicit hold prevents ordinary deletion under controlled authorization.
No “keep forever because storage is cheap”.
retention_class: id: SUPPORT_180D trigger: CASE_CLOSED active_days: 180 derived_policy: INHERIT_OR_SHORTER cache_ttl: 7d logs_ttl: 30d backup_expiry: 35d delete_mode: TOMBSTONE_THEN_PURGE
{
"deletion_id": "DEL-...",
"subject_ref": "subject://...",
"scope": "ACCOUNT_AND_DERIVATIVES",
"requested_at": "...",
"policy_version": "gov-v12",
"targets": {
"primary": 12,
"artifacts": 34,
"vector_entries": 910,
"memories": 8,
"cache_keys": 61,
"eval_samples": 2
},
"status": "IN_PROGRESS",
"exceptions": [],
"verification_ref": null
}Records that a source/subject/object must remain absent after deletion.
Late connector event cannot re-import deleted object without explicit policy.
Tombstone should retain only what is needed for deletion semantics/audit.
Backups inherit a defined retention window.
After restore, replay tombstones/deletion ledger before serving data.
Backup copies should have stricter operational access, not become analytics datasets.
Track which backup generations can still contain deleted data and when they expire.
Where primary database/object/vector data is physically/logically stored.
Model call may process data elsewhere even when source DB is regional.
Central logging can accidentally move sensitive payloads across boundaries.
Disaster recovery replicas are still data locations.
request_data_policy: classification: CONFIDENTIAL purpose: CUSTOMER_SUPPORT allowed_regions: [EU] external_processing: APPROVED_ONLY provider_training_use: DENY retention_at_provider: ZERO_OR_APPROVED requires_contract_profile: ENTERPRISE_PRIVACY №65 eligible_targets(): 1. capability 2. data policy / region / provider 3. permission / tenant 4. health / capacity 5. quality / cost / latency
A cheaper or smarter model endpoint is irrelevant if the data class cannot be processed there.
Exact fields/media/tool arguments required for service.
Map provider/service configuration into internal policy profile.
Provider region and subprocessor path may matter.
Must align with product contract/policy; default assumptions are unsafe.
Every vector query constrained by trusted tenant/access policy.
Each chunk/vector points back to source/version/subject where applicable.
Deletion workflow removes/rebuilds affected index entries.
Embedding may reveal or enable retrieval of source information; govern it accordingly.
Only approved fact classes/purposes enter durable memory.
Memory must be discoverable for review/delete/correction.
Preferences/facts may expire or require revalidation.
Keep secrets in №51 vault, not semantic memory.
Bound cache lifetime according to data sensitivity.
No cross-tenant cache hits for governed content.
Lineage or subject/source index enables targeted purge.
Request IDs, model, token count, timing and policy decision without raw prompt by default.
Only for approved debugging/eval purpose with short retention and access control.
Before log sink; downstream masking is too late if raw data already left process.
Trace inputs/outputs/decisions without hidden chain-of-thought.
Explicit rule for production→evaluation promotion.
Strip unrelated personal/customer content before durable eval storage.
Convert failure pattern into synthetic case without retaining real identity.
If real sample remains, deletion workflow can find it.
Unlike cache/RAG entry, learned influence is not trivially deleted row-by-row.
Only approved curated datasets enter training/fine-tuning pipelines.
Track exact dataset composition, source eligibility and model versions trained from it.
Send only relevant part of screenshot/photo.
Location/device/time metadata can be more sensitive than visible pixels.
Background speech may contain unrelated personal data.
Derived media text/images inherit governance and retention.
| Artifact | Possible policy | Reason |
|---|---|---|
| Raw audio chunks | Ephemeral / very short TTL | Needed for live inference only. |
| Final transcript | Separate retention class | May support product history/quality. |
| Call recording | Explicitly enabled | Distinct privacy/legal/product purpose. |
| Voice biometrics | Special restricted class | Identity-sensitive data with separate controls. |
Limit unrelated browsing state/accounts.
Capture only needed viewport/window where possible.
Mask/autofill outside model context/screenshots.
Downloaded files enter normal classification/retention pipeline.
Link records/artifacts/memories/eval samples to governed subject where applicable.
Map email/account/customer IDs under controlled identity process.
Subject index should expose only what deletion/access workflows need, not create an unrestricted people graph.
Catalog + lineage + subject index find relevant records.
Generate scoped package/report without leaking other tenants/subjects.
Corrections propagate/stale derived data where relevant.
Temporarily block certain use while preserving required data.
policy input: principal tenant purpose data_class subject_scope source destination_type destination_provider destination_region operation retention_class policy output: ALLOW DENY ALLOW_WITH_REDACTION ALLOW_PRIVATE_ONLY REQUIRE_APPROVAL REQUIRE_SHORTER_RETENTION decision also returns: reason_code policy_version obligations[]
Every runtime policy decision references exact ruleset version.
Rules can change; historical decisions need historical context.
Changed retention/provider eligibility can trigger scans/migrations/rebuilds.
{
"exception_id": "GEX-...",
"policy_rule": "CONFIDENTIAL_PROVIDER_DENY",
"scope": "dataset://legacy/v2",
"approved_by": "...",
"reason": "...",
"conditions": ["private_contract_profile"],
"created_at": "...",
"expires_at": "...",
"review_owner": "team://governance"
}
Schema metadata says how field is classified.
Which products/workflows may consume it.
Field/object-level schedule where needed.
Whether field may appear in model input, logs, analytics or third-party export.
Governance can require encrypted stores for data classes; implementation belongs to infrastructure/security.
Transport encryption protects movement but does not authorize destination.
Encryption is not privacy if every service can decrypt everything.
Queries include trusted tenant key / row policy.
No cross-tenant ANN search before filter/access enforcement.
Stable refs still require tenant-aware authorization.
Cache keys, traces and exports retain tenant context.
What rule/version allowed a data flow.
Reference/hash/classification, not raw sensitive payload where avoidable.
Provider/region/store/model route.
Evidence that retention and deletion jobs executed.
% known production data stores/assets with owner/classification/retention.
% governed objects/calls with resolved data class.
% sensitive external/model flows checked by policy engine.
Request → verified purge/restriction completion.
Stores/artifacts with no owner, retention or source lineage.
Expired objects still accessible beyond policy.
Blocked or detected disallowed region/provider flows.
Share of sensitive traces containing raw content; target minimized.
Find and purge/tombstone data beyond schedule.
Discover unregistered stores/artifacts.
Revalidate external service profiles after configuration/contract change.
Periodically check deletion ledger against lineage/store indexes.
Sensitive external model call is blocked until classification resolved.
Tombstone prevents deleted source from reappearing.
Reconciliation finds stale embedding entry after source deletion.
Deletion ledger reapplied before restored data becomes active.
Sensitive content leak into telemetry triggers control failure.
Router rejects healthy but ineligible provider region.
Old exception no longer bypasses policy.
Cross-tenant retrieval/cache attempts blocked.
governance/ ├── taxonomy.yaml ├── policy.py ├── provider_profiles.yaml ├── retention.py ├── deletion.py ├── subject_index.py ├── redaction.py ├── audit.py ├── reconciler.py └── tests/ PostgreSQL: data_assets( asset_id, tenant_id, source_ref, owner, classification, purposes[], region_policy, retention_class, subject_refs[], lineage_ref, expires_at, status ) policy_decisions( decision_id, policy_version, principal, operation, data_asset_id, destination, result, obligations, reason_code, created_at ) deletion_jobs( deletion_id, subject_ref, scope, status, policy_version, requested_at, completed_at, verification_ref )
Do not begin with a huge enterprise catalog if the system has ten stores. Start with enough metadata to make real runtime and deletion decisions.
| Observed need | Upgrade |
|---|---|
| Many datasets/stores | Dedicated data catalog, automated lineage connectors and stewardship workflows. |
| Multiple regions/providers | Formal residency/processor policy engine integrated with routing/deployment. |
| High deletion volume | Subject index, distributed deletion orchestrator, reconciliation and SLA monitoring. |
| Production data reused for eval/training | Dataset approval pipeline, de-identification review and model lineage. |
| Multiple legal/product regimes | Policy profiles by tenant/product/region with legal mapping outside core runtime taxonomy. |
| Sensitive multimodal/voice | Redaction services, ephemeral media processing, stricter artifact/log controls. |
| Enterprise audit requirements | Evidence packages, policy/version reporting, access review and immutable decision logs. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да. В production governance always-on, даже если первая версия минимальна. |
| Separate Component? | YES. Policy/catalog/lifecycle/deletion layer in Production Fabric with hooks across stores and model/tool routes. |
| Минимум 80% ценности? | Inventory, classification, purpose, retention, provider/region policy, governed logs, deletion ledger + lineage and derived-data cleanup. |
| Когда overkill? | Покупать тяжёлую enterprise governance platform до появления реальной сложности; сами governance invariants при этом не overkill. |
| Trigger? | Production data exists. Depth increases with personal/sensitive data, external processors, multi-tenancy, memory/RAG, regulation and retention obligations. |
| Как измерить uplift? | Coverage, deletion SLA, retention violations, orphan data, policy-enforced flow rate, raw-payload leakage rate and governance incident rate. |
| Можно ли rule/tool/code заменить LLM-agent? | Да, и это предпочтительно. Governance enforcement should be deterministic policy/code. LLM may assist classification/review, but never be sole authority for privacy decisions. |
Primary DB, vectors, caches, logs, artifacts, evals and backups all count.
Derived representations do not silently lose sensitivity.
Data possession does not imply unlimited AI use.
Send only fields/regions needed for task.
Provider/region eligibility precedes quality/cost.
Primary record, vectors, memory, cache, evals and backups all have defined treatment.
Observability cannot become a privacy bypass.
Do not casually turn production data into weights.
LLM can suggest; policy engine decides.
DATA ENTERS SYSTEM
user
connector
upload
web
tool
media
↓
REGISTER
source
owner
tenant
subject refs
classification
purpose
region
retention
↓
INGEST / STORE
↓
DERIVE
parsed text
chunks
embeddings
summaries
memories
artifacts
caches
logs
eval samples
↓
PROPAGATE GOVERNANCE
classification
subject links
purpose constraints
retention
residency
deletion lineage
↓
BEFORE EXTERNAL / MODEL PROCESSING
↓
POLICY ENGINE
INPUT:
principal
tenant
purpose
data class
destination
provider
region
operation
OUTPUT:
ALLOW
DENY
REDACT
PRIVATE_ONLY
SHORTER_TTL
APPROVAL_REQUIRED
↓
MINIMIZE
select fields
crop media
redact identifiers
pseudonymize
↓
№65 MODEL ROUTER
capability
THEN governance eligibility
THEN quality/cost/latency
↓
PROCESS
↓
OBSERVABILITY
refs/metadata first
raw payload only when explicitly governed
↓
RETENTION CLOCK
↓
EXPIRE / DELETE / HOLD
DELETE REQUEST
↓
SUBJECT / SOURCE RESOLUTION
↓
№61 LINEAGE
descendants()
↓
PRIMARY DB
ARTIFACT STORE
RAG CHUNKS
VECTOR ENTRIES
MEMORY
CACHE
EVAL SAMPLES
LOGS
BACKUPS
↓
PURGE / TOMBSTONE / EXCEPTION
↓
RECONCILE
↓
VERIFIED DELETION EVIDENCE
BACKUP RESTORE:
restore
↓
replay tombstones/deletion ledger
↓
only then serve data
TRAINING:
separate eligibility
separate purpose
versioned dataset
model lineage
because learned influence
is harder to reverse
BOUNDARIES:
№51
WHO may access / credentials / scopes
№52
HOW attackers may misuse/exfiltrate
№61
WHERE data came from and where it propagated
№76
WHY data may be used,
WHERE it may go,
HOW LONG it may live,
WHAT must happen on deletion
№77
integrates all production layers
CORE PRINCIPLE:
PRIVACY IS NOT
A REDACTION REGEX
IN FRONT OF THE LLM.
DATA GOVERNANCE IS
A DISTRIBUTED LIFECYCLE CONTRACT.
EVERY DATA OBJECT
SHOULD HAVE ENOUGH CONTEXT
FOR THE SYSTEM TO ANSWER:
WHAT IS THIS?
WHO OWNS IT?
WHO IS IT ABOUT?
WHY DO WE HAVE IT?
WHERE MAY IT GO?
WHO MAY SEE IT?
HOW LONG MAY IT LIVE?
CAN IT TRAIN A MODEL?
WHAT DID WE DERIVE FROM IT?
AND HOW DO WE PROVE
IT IS GONE
WHEN IT MUST BE GONE?
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №76 Data Governance & Privacy.
B–E. Existing boundary and placement. The existing conceptual boundary, class PRODUCTION, default ON and owner Production Fabric remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.