Распределённая система ломается не потому, что «сервер упал», а потому что часть операции могла успеть выполниться, часть — нет, ответ потерялся, сообщение пришло дважды, события поменялись местами или два исполнителя одновременно решили, что владеют одной работой.
№57 Queues & Workers владеет backlog/claim/ack/worker execution; №69 определяет cross-boundary correctness: duplicate delivery, inbox dedupe, outbox, leases, fencing, ordering and reconciliation. №59 Broker транспортирует messages/events; transport does not magically make business effects exactly-once. №63 Retry решает, когда повторять transient operation; №69 делает repeated attempt безопасным. №68 Durable Workflow хранит process history and resumes workflow; №69 описывает reliability patterns между workflow engine, queue, DB и external systems. №42 Events задаёт event semantics and trigger rules; №69 deals with duplicate/out-of-order/delivery behavior.
Prerequisites: №10 State Management, №42 Events & Triggers, №46 Observability, №50 Contracts, №57 Queues & Workers, №59 Broker, №60 Artifact Store, №63 Retry, №68 Workflow Engine. Forward references: №70 Model Serving, №76 Data Governance, №77 Production Architecture.
REQUEST-TIME: idempotency, CAS, leases, dedupe, transactional writes. CONTROL PLANE: consistency classes, retry/dedupe windows, lease durations, partition/ordering keys, compensation policy. DATA PLANE: operation ledger, outbox/inbox rows, versions, leases, fencing tokens, delivery attempts. OFFLINE: reconciliation, repair, stuck/duplicate scans, DLQ analysis, invariant tests.
Success: one logical operation produces the intended business state despite duplicate delivery/process crash/retry. Retryable: transport/service/lock-store transient failures. Permanent: invariant violation, stale fencing token, invalid state transition, policy failure. Idempotency: stable operation identity required across attempts. Persist: operation status, dedupe record, state version, outbox/inbox, lease owner/expiry/token, compensation/reconciliation status. Security: idempotency keys and correlation IDs must be scoped to tenant/principal so one caller cannot collide with another.
№69 не владеет queue implementation, broker transport, workflow business logic, scheduler semantics, dependency retry policy or database replication algorithms generally. Она владеет CROSS-BOUNDARY CORRECTNESS UNDER DUPLICATION, REORDERING, PARTIAL FAILURE AND CONCURRENT OWNERSHIP.
Remote service committed, network dropped reply. Caller sees timeout and may retry.
Consumer crashes after side effect but before ACK; broker redelivers.
Parallel partitions, retries or different channels change observed order.
Lease expires, old worker still alive, new worker takes over.
Local state changed but downstream notification was never published.
Message published, transaction rolled back or crashed before commit.
Services/regions temporarily cannot agree on shared state.
Replica/cache returns previous version and caller makes obsolete decision.
| Semantic | What may happen | Typical use |
|---|---|---|
| AT-MOST-ONCE | Message may be lost, but won't be retried by transport. | Low-value telemetry where duplicates worse than loss or data can be recomputed. |
| AT-LEAST-ONCE | Message should eventually arrive, duplicates possible. | Most reliable queues/workers/events; pair with idempotent consumers. |
| EXACTLY-ONCE PROCESSING | Usually means exactly-once within a bounded transactional substrate. | Possible inside some stream/DB systems, but external effects still require idempotency. |
{
"operation_id": "OP-01J...",
"tenant_id": "tenant_A",
"operation_type": "publish.post",
"resource_ref": "post://42",
"input_hash": "sha256:...",
"status": "COMPLETED",
"attempts": 3,
"effect_ref": "provider://post/9981",
"created_at": "...",
"completed_at": "..."
}SET status='ACTIVE' can often be repeated safely under version/state checks.
INSERT ... UNIQUE(operation_id) or provider idempotency key collapses duplicates.
Before/after external action, persist business operation state and reconcile unknown outcome.
| Bad key | Problem | Better |
|---|---|---|
| timestamp | Retry generates a new value, so duplicate is invisible. | Stable operation UUID created before first attempt. |
| user_id only | Blocks unrelated operations by same user. | tenant + operation type + client operation ID. |
| payload hash only | Two legitimate identical requests may collide. | Explicit intent identity + hash used only for validation. |
| global unscoped key | Cross-tenant collision/security issue. | Tenant/principal scope included in unique constraint. |
message_id / event_id / operation_id.
INSERT ... ON CONFLICT DO NOTHING inside local transaction.
Existing inbox row → duplicate; skip/reuse previous result.
outbox( event_id text primary key, aggregate_type text, aggregate_id text, aggregate_version bigint, event_type text, payload_json jsonb, payload_ref text, tenant_id text, created_at timestamptz, published_at timestamptz null, attempts int, last_error jsonb )
SERVICE A
local transaction:
update A state
insert outbox event E1
commit
↓
OUTBOX RELAY
↓
BROKER
↓
message E1
↓
SERVICE B
local transaction:
insert inbox(E1) UNIQUE
if duplicate → no-op
update B state
optionally insert B outbox event E2
commit
↓
ACK E1
RESULT:
transport may duplicate
relay may retry
consumer may crash
but each service applies
each event once logically
to its own local state
resource: id = 42 state = READY version = 7 worker A reads version 7 worker B reads version 7 worker A: UPDATE resource SET state='RUNNING', version=8 WHERE id=42 AND version=7 → 1 row updated worker B: same WHERE version=7 → 0 rows updated B knows its decision was stale and must reload/re-evaluate.
Compare-and-swap / optimistic locking works especially well when:
| Incoming event | Current state | Correct behavior |
|---|---|---|
| PROCESSING v4 | READY v3 | Apply if transition/version valid. |
| READY v3 | PROCESSING v4 | Reject/ignore stale event. |
| COMPLETED v5 | PROCESSING v4 | Apply. |
| PROCESSING v4 duplicate | COMPLETED v5 | No rollback; dedupe/stale handling. |
Messages for one business entity go through same ordering partition where practical.
Per-aggregate monotonically increasing version reveals missing/out-of-order events.
Order A events independently from unrelated order B events.
Short reorder buffer can wait for missing sequence only if latency and completeness justify it.
Current claimant.
If owner dies, lease becomes reclaimable.
Long work periodically extends lease while healthy.
Use authoritative datastore constraints for short critical transitions.
If task needs exclusive processing, use bounded lease + expiry + fencing where effect matters.
Can deadlock, suffer stale ownership, coordinator outage and hidden liveness problems.
BAD:
if not exists(operation_id):
insert(effect)
RACE:
worker A sees absent
worker B sees absent
both insert
GOOD:
INSERT INTO effect_ledger(
tenant_id,
operation_id,
...
)
VALUES (...)
ON CONFLICT (tenant_id, operation_id)
DO NOTHING
DATABASE decides atomically
which logical operation wins.
Workflow knows steps/compensations and commands services. Easier to inspect, reason about and recover for complex processes.
Loose coupling, but process semantics become distributed across consumers; harder to see global state and failure compensation.
Committed intents not yet acknowledged by broker.
Operations stuck IN_PROGRESS beyond timeout.
Local says ACTIVE, provider says absent/closed.
Observed versions 17 and 19 but 18 never arrived.
effect_ledger(
tenant_id text,
operation_id text,
effect_type text,
resource_ref text,
input_hash text,
state text,
provider_ref text,
started_at timestamptz,
completed_at timestamptz,
last_checked_at timestamptz,
error_json jsonb,
primary key(
tenant_id,
operation_id
)
)
state:
RESERVED
EXECUTING
UNKNOWN
COMPLETED
FAILED
COMPENSATEDOwn ledger allows:
After bounded retry, message/work enters DLQ/quarantine with full error context.
One malformed event should not crash every consumer restart forever.
Operators can inspect, patch policy/code, then replay with same message/operation identity.
| Failure | Retry? | Reliability action |
|---|---|---|
| Network timeout | YES bounded | Retry same operation_id; idempotent consumer/effect. |
| Schema invalid | NO | Quarantine + producer/contract fix. |
| Stale version | NO blind retry | Reload state; ignore/recompute transition. |
| Duplicate event | NO new work | Inbox dedupe and ACK. |
| Unknown external effect | RECONCILE FIRST | Query provider by operation/resource ID. |
Permissions, spend reservations, unique business ownership, workflow transition.
User should see own recent update immediately where UX requires.
Analytics dashboards, secondary indexes, search replicas, derived summaries.
Cache/read model may lag by ≤N seconds if domain accepts it.
For a given aggregate/fact, define which DB/service owns writes.
Search/vector/cache/analytics views may lag and are rebuilt from authoritative facts.
Derived views compare version/checksum/watermark against authoritative owner.
Old leader loses coordination but continues writing; new leader is elected and also writes.
Every leadership term gets higher epoch; protected resources reject stale epochs.
Consensus/DB service can serialize leadership, but application effects still need token validation.
Two timestamps don't reliably prove causal order.
Aggregate version/sequence is stronger for business transitions.
Use monotonic clocks for local timeout measurement where available.
Useful for observability/expiry, but not a universal concurrency ordering primitive.
Consumers know how to parse/migrate old event shape.
Detect stale/gap/out-of-order updates.
Useful for tracing semantic change and replay incidents.
Agent retries «send email» after ambiguous response.
Several workers update same record or publish same entity simultaneously.
Regenerating action arguments can create a new business operation instead of retrying the old one.
Once an external action is approved/scheduled, persist exact arguments hash + operation_id and retry that contract, not a fresh model generation.
operation_id + exact resource + args_hash.
Human approves that exact immutable intent.
Same operation_id and same approved args. If args change → new approval.
Attacker reuses another principal's idempotency key to retrieve/suppress operation.
Unique constraints include trusted tenant/principal scope.
Only authorized producer/principal may mutate workflow/resource state.
Replaying old effectful messages requires operator permissions and policy revalidation where needed.
| Item | Replay default | Reason |
|---|---|---|
| Derived search index update | YES | Rebuildable from authoritative source. |
| Analytics event | YES with dedupe | Usually side-effect-light and recomputable. |
| Email / publish action | ONLY same idempotent operation | Duplicate external effect is visible to user/world. |
| Permission-sensitive action | REVALIDATE | Authority may have changed since original attempt. |
| Irreversible destructive operation | MANUAL/STRICT | Replay risk may exceed automation benefit. |
Messages/events observed more than once.
Duplicates safely suppressed/reused by inbox/operation ledger.
Commit → successful broker publish delay.
Missing aggregate versions/events needing repair.
Operations stuck awaiting reconciliation.
Writes blocked from expired/stale owners.
Oldest unresolved dead-letter item and count by reason.
Cross-system mismatches detected per scan.
Outbox eventually emits message after restart.
Republish duplicate; inbox suppresses logical reprocessing.
Same operation_id reconciles/dedupes external effect.
Fencing rejects stale owner mutation.
Consumer detects version gap/stale order and follows policy.
State advances once logically.
Business transaction stays valid; relay catches up after recovery.
Repair job detects and resolves/alerts deterministic inconsistency.
reliability/ ├── idempotency.py ├── outbox.py ├── inbox.py ├── versions.py ├── leases.py ├── fencing.py ├── effects.py ├── reconcile.py └── tests/ processed_messages( tenant_id text, message_id text, processed_at timestamptz, primary key(tenant_id, message_id) ) outbox( event_id text primary key, tenant_id text, aggregate_id text, aggregate_ver bigint, event_type text, payload_json jsonb, published_at timestamptz null, attempts int ) operation_ledger( tenant_id text, operation_id text, operation_type text, input_hash text, status text, result_ref text, primary key(tenant_id, operation_id) ) leases( resource_key text primary key, owner_id text, expires_at timestamptz, fence_token bigint )
Do not start with multi-region consensus, custom distributed lock service or global event ordering unless the actual deployment requires them.
| Signal | Potential upgrade |
|---|---|
| Very high message volume | Partitioned outbox relays, CDC-based outbox publication, scalable inbox retention. |
| Multiple regions write same aggregate | Explicit ownership, consensus/leader epochs, conflict-free or merge semantics. |
| Long cross-service business transactions | Durable saga orchestration integrated with №68. |
| External effects lack idempotency APIs | Stronger effect ledger + read-back reconciliation + human repair path. |
| High lock contention | Redesign ownership/sharding/CAS before adding more lock complexity. |
| Derived read models are critical | Watermarks, sequence-gap recovery, automated rebuild/reconciliation. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да на distributed boundaries. Если всё живёт в одной transaction, использовать ACID и не усложнять. |
| Separate Component? | YES логически. Но это чаще shared libraries/tables/protocols, а не один giant «reliability service». |
| Минимум 80% ценности? | Operation identity, unique constraints, outbox/inbox, CAS, leases/fencing, DLQ, reconciliation, failure-injection tests. |
| Когда overkill? | Consensus, distributed locks, global order и complex sagas для single-process/single-DB application. |
| Trigger? | Operation crosses transaction boundaries, messages can duplicate/reorder, replicas compete, or external effects can have unknown outcomes. |
| Как измерить uplift? | Duplicate-effect rate, reconciliation drift, outbox lag, DLQ age, stale-write rejects, manual repair incidents, data-consistency incidents. |
| Можно ли rule/tool/code вместо LLM-agent? | Не просто можно — нужно. Distributed correctness must be deterministic. LLM can help analyze incidents, but cannot be the concurrency-control protocol. |
Any queue/event/effectful retry path must tolerate repeated delivery.
Retry the same logical intent, not a newly generated action.
Use DB atomicity/uniqueness/CAS before distributed coordination.
Persist state and publication intent atomically.
Dedupe before applying repeated message to local state.
Stale/out-of-order work cannot overwrite newer truth.
Use fencing when stale owners can still mutate protected resources.
Online protocols reduce failure probability; reconciliation catches the remainder.
Correctness is proven when process dies between commit, publish, effect and ACK.
LOGICAL OPERATION
operation_id
tenant scope
exact input hash
↓
LOCAL SERVICE TRANSACTION
↓
STATE UPDATE
version = N+1
+
OUTBOX INSERT
event_id
aggregate_version
↓
COMMIT
↓
OUTBOX RELAY
↓
BROKER / QUEUE
at-least-once
duplicate possible
reorder possible
↓
CONSUMER
↓
INBOX DEDUPE
UNIQUE(message_id)
↓
CHECK VERSION / STATE
↓
APPLY LOCAL EFFECT
↓
OPTIONAL NEXT OUTBOX
↓
COMMIT
↓
ACK
EXTERNAL SIDE EFFECT:
prepare exact operation
↓
operation ledger
↓
execute with idempotency key
↓
response?
YES:
record COMPLETED
TIMEOUT / UNKNOWN:
do not invent a new operation
↓
reconcile by operation/resource ID
↓
already happened?
yes → record success
no → safe retry same operation_id
CONCURRENT OWNERS:
lease owner A / token 41
↓
lease expires
↓
owner B / token 42
↓
shared resource accepts 42
rejects stale 41
OUT-OF-ORDER:
aggregate v9 arrives before v8
↓
version/gap policy
↓
buffer / reload / ignore stale / reconcile
↓
NEVER silently revert v9 to v8
PARTIAL MULTI-SERVICE PROCESS:
forward steps
↓
failure
↓
durable saga
↓
idempotent compensations
↓
explicit final state
BACKGROUND SAFETY NET:
reconciliation
outbox backlog
unknown effects
stale leases
sequence gaps
source/read-model drift
DLQ age
↓
repair / alert / manual review
BOUNDARIES:
№57 QUEUE
transports runnable work
№59 BROKER
transports messages/events
№63 RETRY
decides when to repeat
№68 DURABLE WORKFLOW
remembers process lifecycle
№69 DISTRIBUTED RELIABILITY
ensures retries, duplicates,
ordering and partial failures
do not corrupt business state
CORE PRINCIPLE:
DO NOT TRY TO MAKE
THE NETWORK "EXACTLY ONCE".
MAKE THE BUSINESS OPERATION
IDENTIFIABLE,
IDEMPOTENT,
VERSIONED,
RECONCILABLE
AND SAFE TO REDELIVER.
THE NETWORK MAY DUPLICATE.
THE WORKER MAY DIE.
THE ACK MAY DISAPPEAR.
THE OLD OWNER MAY WAKE UP.
THE SYSTEM SHOULD STILL
CONVERGE TO ONE VALID
LOGICAL BUSINESS STATE.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №69 Distributed Reliability Patterns.
B–E. Existing boundary and placement. The existing conceptual boundary, class PRODUCTION, default CONDITIONAL and owner Production Fabric remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.