Retry повторяет операцию, когда есть основания ожидать, что временная причина исчезнет. Circuit Breaker временно прекращает вызовы явно нездоровой зависимости. Fallback переводит систему на заранее допустимый запасной путь или degraded mode.
№57 Queues & Workers владеет lifecycle конкретной job и её attempt counter; №63 определяет общую retry/fallback/breaker policy для зависимостей и операций. №34 Model Escalation выбирает более сильную модель по cognitive need; fallback меняет путь из-за недоступности/ошибки, а не потому что задача сложная. №45 Verification решает, достаточен ли result; fallback может потребовать повторной verification. №48 Guardrails ограничивает допустимые actions; fallback не может обходить policy. №62 Cache может дать stale fallback только если это разрешено freshness contract. №64 Rate Limits/Quotas/Budgets ограничивает нагрузку и расходы; retry обязан уважать эти бюджеты.
Prerequisites: №34 Model Escalation, №45 Verification, №46 Observability, №48 Guardrails, №50 Contracts, №57 Queues, №62 Caching. Forward references: №64 Rate Limits/Quotas/Budgets, №65 Model Gateway, №68 Durable Workflow, №69 Distributed Reliability, №70 Model Serving.
REQUEST-TIME: YES — timeout/retry/breaker/fallback decisions occur around live calls. CONTROL PLANE: retry classes, limits, thresholds, breaker windows, fallback ladders. DATA PLANE: attempts, errors, breaker state, fallback result metadata. OFFLINE: failure injection, chaos/load tests, threshold tuning, incident regression.
Success: operation succeeds within end-to-end deadline or returns explicit bounded degraded/terminal result. Retryable: transient timeout, temporary unavailable, some rate limits. Permanent: invalid input, permission deny, policy deny, unsupported action. Idempotency: retries of side-effecting operations require effect key/transactional safeguard. Persist/trace: attempt number, error class, backoff, breaker state, fallback chosen, final disposition. Security: fallback cannot widen permissions or silently skip approvals.
№63 не владеет queue lifecycle, model quality escalation, quotas, authorization, workflow replay or distributed consistency as a whole. Она владеет BOUNDED RECOVERY FROM TRANSIENT DEPENDENCY FAILURE AND EXPLICIT GRACEFUL DEGRADATION.
Network timeout, 502/503, connection reset, temporary model overload.
429 / quota window. Respect Retry-After and reduce pressure.
Version/CAS conflict may require re-read and re-evaluation, not blind same retry.
Schema violation, unsupported parameter, malformed request.
Retry cannot ethically or technically bypass authorization.
Prefer bounded conservative behavior; do not retry forever because type is unknown.
retryable, reason_code, retry_after, safe_to_retry, side_effect_status. Natural-language error string is a poor retry policy API.User/job has 12 s total remaining.
Call timeout 3 s; after failure recompute remaining budget.
No retry if backoff + next attempt cannot finish within remaining deadline.
{
"operation": "model.generate",
"deadline_ms": 12000,
"attempt_timeout_ms": 3500,
"max_attempts": 3,
"retryable": [
"TIMEOUT",
"UNAVAILABLE",
"RATE_LIMITED"
],
"backoff": {
"kind": "EXPONENTIAL_JITTER",
"base_ms": 250,
"max_ms": 3000
},
"idempotency": "REQUIRED_FOR_WRITE",
"fallback_policy": "model-primary-v2",
"breaker_policy": "provider-A"
}base = 250 ms attempt 1 failed wait random(0, 250) attempt 2 failed wait random(0, 500) attempt 3 failed wait random(0, 1000) cap at max_backoff respect Retry-After when provider supplies it stop if deadline exhausted
Workflow 3× → worker 3× → connector 3× → HTTP SDK 3× = до 81 сетевой попытки.
Верхний слой определяет operation deadline/attempts, нижние clients либо retry disabled, либо имеют строго ограниченный transport retry.
Remaining deadline передаётся вниз, чтобы каждый layer не создавал собственный бесконечный budget.
| Operation | Retry risk | Required protection |
|---|---|---|
| GET / READ | Usually duplicate computation only. | Timeout/deadline; freshness considered. |
| UPSERT BY STABLE KEY | Usually manageable. | Unique key / expected version. |
| CREATE EXTERNAL OBJECT | May create duplicates. | Provider idempotency key / client operation ID. |
| SEND / PUBLISH / PAYMENT-LIKE EFFECT | Duplicate real-world effect. | Effect ledger, exact idempotency token, read-back/reconciliation. |
| DELETE / IRREVERSIBLE ACTION | Uncertain completion after timeout. | Read current state before retry; policy/approval may need recheck. |
Open provider-A while provider-B remains usable.
Embeddings endpoint may fail while chat works.
One tenant's revoked credential must not open global breaker.
Regional dependency failure can be isolated if routing supports it.
| Signal | Useful? | Notes |
|---|---|---|
| Failure ratio over rolling window | YES | e.g. 50% failures after minimum sample count. |
| Consecutive failures | YES, simple MVP | Easy but sensitive to burstiness. |
| Latency threshold | YES | Slow dependency can be as harmful as failing dependency. |
| Rate-limit response | CONDITIONAL | Often use quota-aware cooldown rather than generic breaker. |
| Invalid input / 403 | NO for shared breaker | Caller/request problem, not dependency health. |
После OPEN ждать bounded interval или provider signal.
Allow 1–N test calls, not full traffic flood.
Enough successful probes → CLOSED; failure → OPEN with next cooldown.
Provider A unavailable → provider B with compatible contract.
Primary model down → tested fallback model; output reverified where needed.
Return slightly stale read only if data class permits.
Return text without optional enrichment/image/secondary data.
Cloud unavailable → local model/tool if capability and policy permit.
For high-risk work, fallback may be queue for later or human review rather than weaker automation.
{
"fallback_id": "model-primary-v2",
"trigger": [
"UNAVAILABLE",
"BREAKER_OPEN"
],
"steps": [
{
"target": "provider_B/model_X",
"max_attempts": 1,
"requires_verification": true
},
{
"target": "DEGRADED_TEMPLATE",
"allowed_for_risk": ["LOW", "MEDIUM"]
}
],
"max_total_latency_ms": 12000,
"max_total_cost": "...",
"never_bypass": [
"POLICY",
"PERMISSION",
"APPROVAL"
]
}Cheap model produced uncertainty/verifier failure; route to stronger model because task is cognitively harder.
Chosen model/provider cannot serve request due to outage, overload or technical failure; use compatible alternate path.
Fallback returns expected structured output or explicit degraded schema.
Fallback model/tool has measured pass rate for allowed task classes.
Fallback path may require stronger verification than primary.
Internal result metadata records fallback/degraded mode; user disclosure if material to interpretation.
| Data | Stale fallback? | Reason |
|---|---|---|
| Static reference documentation | YES | Low volatility; bounded stale age. |
| Weather/news/current market-like fact | CONDITIONAL | Age must be visible/acceptable. |
| User permissions | USUALLY NO | Security state can change. |
| Financial/account balance | NO for authoritative action | Stale state can create wrong external effect. |
| Generated FAQ answer | YES if source/policy versions compatible | Semantic stability can be evaluated. |
Provider/model/tool classes get bounded worker/connection pools.
Slow OCR jobs do not starve interactive model calls.
One tenant's retries do not consume all global capacity.
Breaker + pool per dependency prevents cascading saturation.
If first safe read is unusually slow, start second request after percentile threshold; first successful response wins.
Do not hedge irreversible writes unless exact idempotency semantics are guaranteed.
Use only for measured tail-latency problem and cap hedge rate.
Например, 2–3 total attempts inside end-to-end deadline.
Additional retry traffic should stay below bounded share of healthy traffic.
Prevent one failure domain from consuming system-wide retry capacity.
№57 repeats one failed job/activity under idempotency and attempt limits.
Durable workflow may resume from checkpoint/history instead of repeating already completed external effects.
Policy/permission deny is terminal for current request unless authority legitimately changes.
Alternate model/provider/tool cannot bypass mandatory checks.
Long retry/fallback sequence may outlive approval token; bind and validate before effect.
Fallback provider must satisfy tenant/data-location/security policy.
Attempts per operation by dependency/error class.
% failed first attempts recovered by later retry.
Total dependency calls / logical operations.
Open rate/duration/half-open probe success.
% operations using alternate/degraded path.
Eval/verification difference primary vs fallback.
Extra time caused by retries/backoff/fallback.
Timed-out side effects needing reconciliation.
initial_error, each attempt, breaker decision, fallback target and final disposition в одном trace/run.First call times out, second succeeds; deadline respected.
Retry-After respected; no hot loop.
Failure threshold opens circuit; calls fail fast/fallback.
Limited probes close circuit without traffic flood.
System reconciles before retry; no duplicate side effect.
Fallback preserves schema/policy and passes required verification.
Ensure lower layers don't multiply attempts unexpectedly.
Recovery stops when total latency/cost budget exhausted.
Logical operations saved by retry after first-attempt failure.
Dependency attempts / logical operations. Watch during incidents.
Time/calls spent OPEN by dependency scope.
Share of operations that leave primary path.
Verification/eval difference vs primary.
Extra p95 latency caused by retries and fallback ladder.
Side-effect timeouts requiring reconciliation.
Overload events caused or amplified by retry traffic.
resilience/
├── errors.py
├── deadline.py
├── retry.py
├── breaker.py
├── fallback.py
├── policies.py
└── tests/
call_with_resilience(op, ctx):
deadline = ctx.deadline
if breaker.is_open(op.dependency):
return fallback_or_fail("BREAKER_OPEN")
for attempt in policy.attempts:
if deadline.remaining() <= 0:
return TIMEOUT
try:
result = call(
timeout=min(
policy.attempt_timeout,
deadline.remaining()
)
)
breaker.record_success()
return result
except Error as e:
classify(e)
breaker.record_if_health_failure(e)
if not e.retryable:
return fallback_or_fail(e)
if not safe_to_retry(op, e):
return reconcile_or_fail(e)
sleep(backoff_with_jitter())
return fallback_or_fail("ATTEMPTS_EXHAUSTED")Shared/distributed breaker state is not required until multi-instance behavior proves it useful.
| Signal | Potential upgrade |
|---|---|
| Many service instances hit same dependency | Shared health signals / coordinated rate control; breaker state may remain local if telemetry is global. |
| Provider quotas cause synchronized retry pressure | Central rate/quota budget — №64. |
| Many model providers/routes | Model Gateway with health-aware routing — №65. |
| Long-running multi-step recovery | Durable workflow/replay/compensation — №68. |
| Cross-service duplicate effects / delivery seams | Distributed reliability patterns — №69. |
| Local model serving overload | Serving-level admission, batching, health/capacity — №70. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да, минимальную resilience policy почти всегда. Сеть, модели и внешние APIs неизбежно дают временные сбои. |
| Separate Component? | YES логически. Обычно common library/middleware + policy registry, не отдельный сервис на старте. |
| Минимум 80% ценности? | Typed failures, deadline/timeouts, bounded retry, backoff+jitter, idempotency, one retry owner, simple breaker, tested fallback, metrics. |
| Когда overkill? | Complex distributed breaker/hedging/chaos platform до появления достаточной нагрузки и реальных incident patterns. |
| Trigger? | Любая fallible network/model/tool dependency; advanced mechanisms only after measured failure patterns. |
| Как измерить uplift? | Recovered transient failures, lower incident duration, retry amplification, breaker protection, fallback success/quality, duplicate-effect rate, p95 latency tax. |
| Можно ли rule/tool/code вместо LLM-agent? | Да, это правильный default. Error classification may include semantic exceptions, but retry/breaker/deadline/idempotency mechanics must be deterministic. |
Transient, rate-limited, conflict, invalid, denied and unknown are different.
Total user/job budget controls how much recovery is allowed.
Never hot-loop an unhealthy dependency.
Avoid multiplicative nested retries.
Timeout after side effect creates unknown outcome, not automatic retry.
Per provider/endpoint/credential/region as appropriate.
Alternate path preserves policy, schema, data constraints and quality floor.
Degraded mode is optional; some operations should stop instead.
Failure injection and fallback evals are part of production readiness.
CALL / TOOL / MODEL / CONNECTOR / DB
↓
CHECK END-TO-END DEADLINE
↓
CHECK CIRCUIT
├─ OPEN
│ ↓
│ FAIL FAST
│ ↓
│ FALLBACK / DEFER / FAIL
│
└─ CLOSED / HALF-OPEN
↓
ATTEMPT
↓
SUCCESS
└──────────────→ RETURN
FAILURE
↓
CLASSIFY ERROR
├─ INVALID / DENIED
│ ↓
│ NO RETRY
│ ↓
│ SAFE FALLBACK OR FAIL
│
├─ UNKNOWN WRITE OUTCOME
│ ↓
│ RECONCILE / IDEMPOTENCY
│
└─ TRANSIENT / RATE LIMITED
↓
SAFE TO RETRY?
↓
DEADLINE LEFT?
↓
RETRY BUDGET LEFT?
↓
BACKOFF + JITTER
↓
NEXT ATTEMPT
SUSTAINED HEALTH FAILURE:
↓
CIRCUIT OPEN
↓
COOLDOWN
↓
HALF-OPEN LIMITED PROBES
├─ success → CLOSED
└─ failure → OPEN
FALLBACK LADDER:
primary
↓
compatible alternate
↓
bounded degraded mode
↓
stale-safe cache if allowed
↓
human/defer
↓
explicit failure
NEVER FALL BACK AROUND:
permissions
policy
approval
data residency
required safety checks
CORE PRINCIPLE:
RETRY ONLY WHEN
TIME CAN PLAUSIBLY FIX THE FAILURE.
OPEN THE CIRCUIT WHEN
MORE CALLS WILL ONLY MAKE THINGS WORSE.
FALL BACK ONLY TO
A PATH THAT IS ALREADY KNOWN,
ALLOWED AND TESTED.
AND FOR SIDE EFFECTS:
A TIMEOUT DOES NOT MEAN
"NOTHING HAPPENED".
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №63 Fallbacks, Retry & Circuit Breakers.
B–E. Existing boundary and placement. The existing conceptual boundary, class PRODUCTION, default ON and owner Production Fabric remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.