Prompt Injection & Agent Security — архитектура защиты AI-системы от ситуации, когда внешние данные, пользовательский контент, retrieved documents, web pages, emails, tool outputs или другой agent пытаются повлиять на поведение модели так, будто они обладают более высоким authority, чем на самом деле.
№48 Guardrails & Policies определяет общие runtime rules и enforcement; №52 фокусируется на threat model, hostile/untrusted content и misuse paths. №51 Permissions & Secrets ограничивает реальные capabilities/credentials; №52 предполагает, что attacker может пытаться заставить модель злоупотребить доступными capabilities. №50 Contracts уменьшает ambiguity payloads, но valid schema не делает содержание безопасным. №45 Verification проверяет correctness результата; security проверяет допустимость и происхождение действий/данных. №53 Sandbox контейнирует выполнение untrusted code. №76 Governance/Privacy шире — data classification, retention, compliance.
Prerequisites: №9 Context Engineering, №12 Tools, №25 Evidence-First, №42 Events, №43 A2A, №46 Observability, №48 Policies, №50 Contracts, №51 Permissions & Secrets. Forward references: №53 Sandbox, №54 Connectors, №61 Provenance/Lineage, №74 Computer Use, №76 Data Governance & Privacy.
REQUEST-TIME: YES — context, tool/action, output/egress checks. CONTROL PLANE: YES — threat rules, capability inventory, trust policies, incident response. DATA PLANE: YES — retrieved content, tool results, external documents, connector data. OFFLINE: YES — adversarial evals, red-team scenarios, threat reviews, regression suites.
Success: untrusted content cannot silently gain instruction authority or expand capability beyond host policy. Retryable: transient security classifier/check failure only where retry is safe. Permanent: blocked exfiltration, cross-tenant request, forbidden side effect, unsupported trust transition. Idempotency: security retries must not duplicate actions. Persist: source/provenance/trust labels, decisions, blocked sinks, incident refs. Trace: security events without storing secrets or hidden chain-of-thought.
№52 не владеет permission store, secret vault, policy DSL, sandbox implementation, general data governance, malware detection platform или network security stack. Она владеет THREAT MODEL + TRUST BOUNDARIES + DEFENSE-IN-DEPTH FOR AGENTIC AI.
Пользователь пытается изменить behavior/priority system через обычный input. Сам по себе direct input не равен compromise: система должна применять hierarchy/policy.
Web page, email, PDF, retrieved note, ticket или tool result содержит текст, пытающийся управлять agent behavior.
Tool возвращает untrusted content, которое модель затем воспринимает как command.
Вредная/ложная instruction-like запись попадает в long-term memory и влияет на будущие tasks.
Message другого agent воспринимается как higher-authority instruction без проверки sender/capabilities/context.
Agent с законным доступом выполняет действие в интересах untrusted input, который сам такого права не имел.
Untrusted content пытается заставить agent раскрыть private context, memory, credentials или protected artifacts.
Task A получает data/capability tenant B через retrieval, cache, tool params или shared state.
Workflow начинает как read-only анализ, но model chain постепенно предлагает external write/delete/publish.
Tokens, API keys, session material, signing secrets.
Tenant data, user files, personal/internal content.
Send, publish, delete, buy, modify, execute, delegate.
Tasks, approvals, workflow status, memory and routing decisions.
Попытка превратить untrusted text в privileged instruction.
Попытка получить другой workspace/customer scope.
Вредная публикация, письмо, artifact или API effect.
Попытка скрыть источник действия, изменить provenance или загрязнить logs.
web/email/PDF/tool/connector
source_id + tenant + timestamp + trust class.
content + provenance + taint labels + allowed use.
Например: evidence_only / no_tool_authority.
Перед tool/action/memory/write система проверяет, может ли untrusted content влиять на этот sink.
{
"source_ref": "web://example/42",
"tenant_id": "tenant_A",
"trust": "UNTRUSTED_EXTERNAL",
"content_role": "EVIDENCE",
"taint": [
"EXTERNAL_TEXT",
"NO_INSTRUCTION_AUTHORITY"
],
"allowed_sinks": [
"SUMMARY",
"CLAIM_EVIDENCE"
],
"forbidden_sinks": [
"SECRET_ACCESS",
"PERMISSION_CHANGE"
]
}Метка не защищает сама по себе. Она позволяет Context Manager, policy layer, memory, tool wrappers и release pipeline принимать deterministic решения.
Например, external instruction-like text можно показать модели для анализа, но нельзя использовать как основание расширить permissions или сменить tenant.
get_invoice(id) безопаснее arbitrary SQL/read-all tool.
Не смешивать read и destructive/write capability в одном generic endpoint.
Security-critical scope добавляется host-side, а не берётся свободно из model text.
General code/shell execution требует отдельного sandbox/containment path — тема №53.
Strict enums, ranges, paths, destinations, object counts.
Read may be allowed, publish may require approval.
Security retries/webhooks не должны повторять side effect.
После action проверить authoritative system state.
| Sink | Проверить | Типичный control |
|---|---|---|
| External model/provider | Data class, tenant, residency/provider policy. | Redaction/minimization/provider allowlist. |
| Email/message | Recipient, attachment refs, sensitive content. | Recipient scope + approval + redaction. |
| Public publish | Destination, exact payload, claims/secrets. | Policy gate + HITL + release validation. |
| Tool/connector | Parameters, resource, data leaving boundary. | Schema + authz + egress policy. |
| Subagent | What context/capabilities are delegated. | Need-to-know context + capability subset. |
| Artifact export | PII/secrets/embedded metadata. | Scan/redact/encrypt/destination policy. |
Любая фраза из web/email/tool result может стать durable instruction-like memory.
Memory Consolidation извлекает candidate fact/preference/procedure, сохраняет provenance и проверяет scope.
Memory никогда не должна сама выдавать permissions, credentials или менять policy hierarchy.
{
"sender_agent": "research_agent",
"task_id": "SUB-42",
"contract": "research.result.v2",
"capability_scope": "read_only",
"tenant_id": "tenant_A",
"result_ref": "artifact://...",
"trust": "SCOPED_AGENT_OUTPUT"
}Agent-to-agent message can propose, never mint privilege.
Instruction-like content in documents, suspicious intent, ambiguous egress, policy interpretation.
Classifier itself can miss, overblock or be affected by adversarial context.
Classifier may trigger stronger checks, but permissions, tenant binding, secret isolation and PEP remain deterministic.
| Event | Response |
|---|---|
| Untrusted instruction-like content | Keep as data; continue with scoped extraction/summary if task allows. |
| Attempted cross-tenant access | Hard DENY, security event, no retry with altered phrasing. |
| Secret requested by model | Do not inject raw secret; expose only capability/tool result. |
| High-risk action from untrusted source | Re-authorize intent, policy gate, HITL if required. |
| Suspicious memory candidate | Do not consolidate automatically; quarantine/review. |
| Security control unavailable | Risk-based fail closed for sensitive paths. |
source_ref, tenant, trust class, connector/tool.
blocked sink, attempted scope expansion, suspicious injection class.
policy/permission/security result + reason code.
blocked, sanitized, approved, executed, no side effect.
External content не может переопределить host rules.
Protected data не уходит в forbidden sink.
Untrusted content не получает write/admin capability.
Cross-tenant resource substitution blocked.
Hostile external text не становится durable control rule.
Subagent cannot expand parent scope or bypass approval.
Sensitive operation fails safely.
System still processes harmless instruction-like content as data when appropriate.
| Failure | Expected behavior |
|---|---|
| Permission service unavailable | Sensitive writes fail closed / safe manual path. |
| Secret store unavailable | No bypass by asking model for credential; bounded retry/fail. |
| Provenance missing | Treat content as lower trust, not automatically trusted. |
| Classifier unavailable | Hard scopes still enforce; semantic-only features degrade safely. |
| Duplicate security webhook | Idempotent handling; no duplicate side effect. |
| State/version mismatch | Re-evaluate policy/approval instead of using stale decision. |
Page may contain instruction-like text. Browser agent should extract task-relevant data; navigation/click/write actions remain policy-bound.
Email sender/content cannot grant tool permission. Replies/sends require authenticated task context and policy.
Extracted text inherits source trust/provenance. Embedded commands are content unless explicit trusted workflow says otherwise.
Unauthorized sensitive action/data release caused by untrusted input. Target: zero.
Unauthorized tenant/resource crossover. Target: zero.
Protected data reaching forbidden sinks.
% external content carrying provenance/trust metadata.
% sensitive tools/actions behind real enforcement.
Benign content incorrectly blocked as hostile.
Unverified external control-like content promoted to durable memory.
Known security incident classes covered by permanent tests.
security/ ├── trust.py ├── provenance.py ├── taint.py ├── sink_policy.py ├── redaction.py ├── incident.py └── tests/ context/ ├── builder.py └── source_labels.py tools/ ├── narrow_wrappers/ └── schemas/ invariants: external_text != privileged_instruction model_output != authorization approval != permission tenant_scope = host_bound secret != model_context sensitive_action -> PEP dangerous_code -> sandbox
Semantic security classifier можно добавить поверх этого, но не вместо hard boundaries.
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да, до того как агент получит реальные tools, connectors и private data. |
| Separate Component? | YES как отдельная security responsibility, но enforcement распределён по Context Manager, Production Fabric, tool wrappers, permissions и policy. |
| Минимум 80% ценности? | Trust/provenance, least privilege, host-bound scope, secret isolation, pre-action PEP, egress controls, memory gate, security evals. |
| Когда overkill? | Если строить многоступенчатый AI security swarm для read-only локального прототипа без private data/tools, вместо простых deterministic boundaries. |
| Trigger? | External/untrusted content, private context, tools, connectors, A2A, memory writes, external side effects. |
| Как измерить uplift? | Critical escapes, exfiltration/cross-tenant rate, false positives, tool gate coverage, memory poisoning, incident regressions. |
| Можно ли rule/tool/code вместо LLM-agent? | Да, большая часть должна быть code/policy/permission/sandbox. LLM security judge — только semantic assist. |
External content не может самостоятельно стать privileged instruction.
LLM output не создаёт permissions, credentials, tenant scope или approval.
Уменьшать blast radius до попыток semantic detection.
Principal/tenant/resource не доверять свободному model output.
Модель видит capability metadata, не credential material.
Security применяется перед action/egress/memory write, а не только на input.
A2A не расширяет scope и не наследует authority автоматически.
Недоступность critical security control не должна превращаться в bypass.
Security evals измеряют, произошёл ли unauthorized effect, а не только распознан ли suspicious text.
EXTERNAL WORLD
web / email / docs / tool results / connectors / agents
↓
SOURCE ID + PROVENANCE + TRUST LABEL
↓
CONTEXT MANAGER
trusted control separated from untrusted data
↓
MODEL / AGENT
may analyze and propose
↓
STRUCTURED OUTPUT / ACTION INTENT
↓
HOST SECURITY BOUNDARY
├─ schema
├─ principal
├─ tenant/resource binding
├─ permission
├─ policy
├─ data egress
├─ approval
└─ secret isolation
↓
TOOL / CONNECTOR / MEMORY / ARTIFACT / A2A
↓
[ SANDBOX if untrusted code execution ]
↓
ACTUAL EFFECT
↓
READBACK / VERIFICATION
↓
SECURITY TRACE + INCIDENT REGRESSION
NEVER ASSUME:
RETRIEVED TEXT = TRUSTED INSTRUCTION
MODEL OUTPUT = AUTHORIZATION
HUMAN APPROVAL = PERMISSION
VALID JSON = SAFE ACTION
ANOTHER AGENT = ROOT OF TRUST
CORE PRINCIPLE:
THE MODEL MAY READ UNTRUSTED CONTENT.
IT MAY EVEN REASON ABOUT INSTRUCTIONS INSIDE THAT CONTENT.
BUT ONLY THE HOST SYSTEM
MAY DECIDE WHAT HAS AUTHORITY,
WHAT HAS PERMISSION,
AND WHAT MAY ACTUALLY HAPPEN.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №52 Prompt Injection & Agent Security.
B–E. Existing boundary and placement. The existing conceptual boundary, class PRODUCTION, default ON and owner Production Fabric + Context Manager remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.