Model Gateway — единая production-граница между приложением и моделями/провайдерами. Он нормализует API, применяет policy, quotas, credentials, observability, retries/fallbacks и выдаёт стабильный внутренний contract. Model Routing выбирает конкретный model/provider/serving endpoint для данного запроса по capability, policy, risk, cost, latency, availability и quality requirements.
№34 Model Escalation владеет cognitive policy: когда дешёвой модели недостаточно и нужен более сильный tier; №65 исполняет route к допустимой model/provider capacity. №04 Routing шире и выбирает исполнителя/agent/tool path; №65 routing ограничен model targets. №63 Fallbacks отвечает на availability failures; gateway является enforcement point для model fallback. №64 Limits/Budgets владеет counters/ceilings; gateway проверяет их перед model execution. №51 Permissions/Secrets владеет credentials/scopes; gateway использует host-side provider credentials. №70 Model Serving владеет inference servers, batching and runtime capacity; gateway выбирает serving endpoint, но не управляет kernel-level inference internals.
Prerequisites: №04 Routing, №34 Model Escalation, №46 Observability, №48 Guardrails, №50 Contracts, №51 Permissions & Secrets, №63 Resilience, №64 Limits/Budgets. Forward references: №70 Model Serving, №71 Local LLM, №72 Quantization, №73 Multimodal AI, №76 Governance.
REQUEST-TIME: YES — every model call passes through gateway/router. CONTROL PLANE: model registry, capabilities, route policies, credentials refs, price/latency classes, region/data restrictions. DATA PLANE: normalized requests/responses, route decisions, usage/health signals. OFFLINE: routing evals, model comparisons, canary/shadow, cost-quality calibration.
Success: request routed to an eligible target and returns normalized output within deadline/budget/policy. Retryable: transient provider/endpoint failure. Permanent: no eligible model, unsupported capability, policy/data-residency conflict, invalid request. Idempotency: model generation itself is usually not side-effecting; tool calls emitted by model are separate operations. Persist/trace: route decision, candidate targets, selected model/provider/version, latency/tokens/cost, fallback/escalation reason. Security: model cannot choose provider credential or bypass route policy.
№65 не владеет task decomposition, agent routing generally, model training, serving internals, cognitive escalation criteria or user-facing quality verification. Она владеет STANDARDIZED MODEL ACCESS AND ENFORCED SELECTION OF ELIGIBLE MODEL/PROVIDER/SERVING TARGETS.
R01 calls SDK A. R06 calls SDK B. Tool wrapper embeds provider C. Each has own retries, auth and response format.
One internal contract, one enforcement boundary, one registry and routing policy.
Cloud A, Cloud B, local vLLM-like endpoint, specialized multimodal model.
{
"request_id": "MR-...",
"task_class": "RESEARCH_SYNTHESIS",
"messages": [...],
"input_artifacts": [...],
"required": {
"modalities": ["text"],
"structured_output": "answer.v3",
"tool_calling": true,
"min_context_tokens": 64000
},
"constraints": {
"deadline_ms": 12000,
"max_cost": "...",
"data_region": "EU",
"risk": "MEDIUM",
"preferred_tier": "STANDARD"
},
"routing": {
"policy": "route-v8",
"allow_fallback": true
}
}Messages, tools, JSON schema, multimodal inputs, max tokens, reasoning controls.
Provider secret resolved host-side from secure secret store.
429, timeout, invalid schema, unsupported capability mapped to common reason codes.
Tokens, cached tokens, audio/image units, latency and provider-specific billing units.
Provider tool-call shape normalized into internal tool request contract.
Text deltas, tool-call deltas, usage/final events converted to stable internal stream.
{
"model_ref": "model://providerA/model-x",
"provider": "providerA",
"model_id": "model-x",
"status": "ACTIVE",
"capabilities": {
"text": true,
"vision": true,
"audio": false,
"tool_calling": true,
"structured_output": true,
"context_tokens": 128000
},
"classes": {
"quality_tier": "HIGH",
"latency_tier": "MEDIUM",
"cost_tier": "HIGH"
},
"deployment": {
"regions": ["EU", "US"],
"data_policy": ["CONFIDENTIAL_OK"]
},
"health_ref": "health://...",
"pricing_version": "2026-08-31"
}Classification, extraction, code, research synthesis, vision, multimodal, long-context.
Task/risk segment evals define which models are sufficient.
Interactive 2 s vs offline 5 min can route differently.
Estimated prompt/output/tool behavior must fit resource budget.
Region, confidentiality, tenant allowlist, regulated workload.
Breaker, recent error/latency, capacity headroom.
Required context window and supported media.
Warm local model, cached prefix, tenant deployment, session stickiness where useful.
{
"route_id": "ROUTE-...",
"request_id": "MR-...",
"policy_version": "route-v8",
"task_class": "RESEARCH_SYNTHESIS",
"required": {
"min_quality_tier": "STANDARD",
"tool_calling": true,
"context_tokens": 64000
},
"eligible": [
"model://A/x",
"model://B/y"
],
"excluded": [
{
"model": "model://C/z",
"reason": "DATA_REGION_MISMATCH"
}
],
"selected": "model://B/y",
"selection_reason": "BEST_SCORE",
"fallback_order": ["model://A/x"],
"score": {
"quality": 0.92,
"latency": 0.78,
"cost": 0.71,
"health": 1.0
}
}task_class -> preferred target CLASSIFY_SMALL -> local-small EXTRACT_STRUCTURED -> cloud-fast-json GENERAL_ASSIST -> standard-model RESEARCH_SYNTHESIS -> strong-model VISION_PARSE -> multimodal-model CODE_REVIEW -> code-capable-model fallback: preferred unavailable -> compatible alternate
requires_vision = true → vision models only.
confidential → approved local/private providers.
risk=HIGH → models with verified score ≥ threshold + stronger verification.
deadline < 2s → fast tier with sufficient quality.
estimated cost > remaining → cheaper eligible model or degrade.
breaker=open / headroom low → remove or strongly penalize target.
score(model, request) =
0.45 * quality_fit
+ 0.20 * latency_fit
+ 0.15 * cost_fit
+ 0.10 * health
+ 0.10 * affinity
subject to HARD constraints:
capability == satisfied
policy == allowed
context >= required
budget == fits
region == allowed
breaker != OPENWeights должны калиброваться через offline evals and production metrics.
Для большинства систем полезнее иметь 3–5 явных route classes, чем один «магический» universal score.
Task starts on cheap sufficient tier.
Verifier failure, uncertainty, capability mismatch, high risk.
Select eligible stronger target under policy, health and budget.
Recent timeout/5xx/429 by endpoint/model/region.
Route away when deadline cannot be met.
№63 breaker state is a hard/strong routing signal.
№64 remaining RPM/TPM/concurrency reduces overload routing.
{
"target": "model://providerA/x/eu1",
"status": "DEGRADED",
"window": "5m",
"success_rate": 0.94,
"p50_latency_ms": 820,
"p95_latency_ms": 4100,
"rate_limit_headroom": 0.18,
"concurrency_headroom": 0.31,
"breaker": "CLOSED",
"last_error": "RATE_LIMITED",
"updated_at": "..."
}Routing can filter/penalize by class while retaining raw metrics for observability.
EU-only, on-prem-only, tenant-specific deployment.
Public, internal, confidential, regulated classes map to approved targets.
Route only to endpoints whose data handling matches policy.
Local/self-hosted target may be required for restricted data even if cloud model is stronger.
Classification, extraction, simple rewriting, offline/private workloads.
Hard reasoning, complex multimodal, long-context when policy allows.
Same request contract; local serving differences hidden behind adapter/gateway.
| Constraint | Route consequence |
|---|---|
| Input + tools + output need 90k tokens | Models below required context are ineligible. |
| Need native vision | Text-only targets are ineligible unless upstream preprocessing changes task. |
| Need structured JSON schema | Prefer/require targets with tested structured-output compliance. |
| Need tool calling | Target must support compatible tool-call contract or use alternate orchestration pattern. |
| Need realtime streaming | Route to endpoints with streaming/realtime capability and latency profile. |
May beat larger model on strict schema adherence and speed.
Different model may lead on executable correctness.
Stronger synthesis/evidence use may justify higher cost.
Specialized multimodal target may be required.
Does policy choose models that meet quality floor by segment?
Cost per successful task vs always-strong baseline.
p50/p95 and SLA compliance by route.
Capability/policy/region/budget-invalid routes. Target 0.
How often cheap-first works vs requires expensive escalation.
Availability recovery without dropping below floor.
Difference vs best eligible target known from offline eval.
Avoid oscillating targets due to noisy short-window metrics.
Selected production request also evaluated on candidate target offline/async; compare quality/cost/latency.
1–5% eligible traffic goes to new model/policy with rollback threshold.
Increase share only after eval/production metrics remain acceptable.
Keep same compatible model within session when behavior continuity matters.
Provider/model with warm prefix/cache may reduce latency/cost if policy supports it.
Prefer nearby/private endpoint to reduce latency and data movement.
Prefer explicit version/revision for reproducibility when provider supports it.
Continuous evals detect quality/schema/tool changes after provider updates.
Store actual provider/model/version metadata returned at execution.
Disable/deprioritize target if regression exceeds threshold.
Gateway accepts canonical JSON Schema / contract ref from №50.
Maps to provider-native structured output / constrained decoding / tool trick as supported.
Never trust provider guarantee alone; validate normalized result against canonical schema.
| Internal event | Meaning |
|---|---|
| TEXT_DELTA | Incremental assistant text. |
| TOOL_CALL_DELTA | Incremental structured tool proposal fields. |
| USAGE_UPDATE | Optional provider usage/cost signal. |
| ROUTE_METADATA | Selected target/fallback/escalation metadata. |
| FINAL | Normalized final result + finish reason. |
| ERROR | Typed error; retry/fallback may occur depending on stage. |
Provider API keys never placed in prompt/model-visible config.
Caller cannot spoof route to another tenant's deployment/credential.
Sensitive payload only to approved regions/providers/endpoints.
Trace metadata without dumping sensitive prompts/responses by default.
Gateway transport does not elevate retrieved/user text into trusted instructions.
Model IDs/endpoints are selected from registry, not arbitrary user/model-provided URLs.
Every route decision references policy/version and identity context.
Availability fallback cannot widen region/data/permission constraints.
Calls by model/provider/task/tenant/route policy.
p50/p95/p99 by target and task class.
Input/output/cached tokens or equivalent units.
Estimated/actual spend by route/task/tenant.
429/timeout/5xx/schema/tool-call failures.
Cheap-first → strong model transitions.
Availability route changes and quality outcome.
Verification/eval pass rates per task segment.
Verified/eval success by routed model and task segment.
Total model cost including escalation/fallback divided by successful tasks.
End-to-end model-stage latency by route policy.
Share of cheap-first calls requiring stronger model.
Availability-driven alternate target usage.
Capability/policy/region-invalid selection. Target 0.
Quality/cost gap vs best eligible target under offline oracle.
Unnecessary target oscillation driven by noisy metrics.
Vision/tool/context requirements exclude incompatible targets.
Restricted data never routes to disallowed provider/region.
Expensive target rejected/degraded when reservation doesn't fit.
Open breaker removes target; recovery returns it gradually.
Primary outage selects only compatible tested alternates.
Stronger tier selection follows escalation signal and hard filters.
Registry/model version changes create expected route/canary behavior.
Same inputs/control-plane state produce same route decision.
model_gateway/
├── contracts.py
├── registry.py
├── router.py
├── policies.py
├── health.py
├── usage.py
├── adapters/
│ ├── provider_a.py
│ ├── provider_b.py
│ └── local.py
└── tests/
route(request, ctx):
candidates = registry.active()
candidates = filter_capability(
candidates, request.required
)
candidates = filter_policy(
candidates, ctx.tenant, ctx.data_class
)
candidates = filter_health(candidates)
candidates = filter_budget(
candidates, ctx.remaining_budget
)
selected = static_or_score(candidates)
return RouteDecision(
selected=selected,
fallback_order=...
)
generate(request):
decision = route(...)
reserve_usage(...)
call adapter
normalize response
commit usage
record route + provenance
return responseNo LLM router, no reinforcement-learning router, no complex bandit required initially.
| Signal | Potential upgrade |
|---|---|
| Many models/providers with overlapping capabilities | Score-based routing by eval/cost/latency/health. |
| Rapid provider price/latency variation | Dynamic cost/health metadata and short-lived route weights. |
| High traffic / meaningful route regret | Contextual bandit/online optimization after strong guardrails and offline eval. |
| Local model cluster | Serving-aware capacity routing integrated with №70. |
| Data residency per tenant/region | Region-aware deployment registry and hard route constraints. |
| Many model versions | Canary/shadow automation and continuous routing eval pipeline. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да. Даже one-provider system выигрывает от стабильного gateway seam; routing complexity может оставаться минимальной. |
| Separate Component? | YES логически. На старте может быть module/library внутри R09 Model Router, не отдельный network service. |
| Минимум 80% ценности? | Internal contract, registry, static routing, capability/policy filter, budget/health checks, adapters, fallback, metrics, provenance. |
| Когда overkill? | LLM-router/bandit/real-time dynamic scoring для трёх моделей без стабильных eval baselines. |
| Trigger? | Minimum gateway ON for any production model call; advanced routing once there are multiple eligible targets and meaningful trade-offs. |
| Как измерить uplift? | Cost per successful task, p95 latency, quality pass rate, route violations, provider outage impact, escalation efficiency, model-switch migration effort. |
| Можно ли rule/tool/code вместо LLM-agent? | Да, и это default. Hard constraints and most routing are deterministic. LLM-based routing is optional only for semantic task classification after strict outer filters. |
Core modules call gateway, not provider SDKs directly.
Capability/policy/region/budget are hard eligibility constraints.
Explicit task-class mapping is the right first router.
Quality-driven strengthening and outage-driven alternate routing have different owners/reasons.
No universal «best model» without capability/eval context.
Provider/model/version/endpoint belongs in route decision and provenance.
Don't route purely from static model rankings.
Normalize model tool proposals, execute only through R07.
Measure policy quality/cost/latency, not only individual model benchmarks.
R01 / R02 / R03 / R06 / R08
↓
MODEL REQUEST CONTRACT
task_class
messages/context
modality
tools
structured output
deadline
risk
data class / region
budget
↓
R09 MODEL GATEWAY
↓
MODEL REGISTRY
provider
model
endpoint
capabilities
context
modalities
quality tier
cost tier
latency tier
region / policy
health
capacity
↓
ELIGIBILITY FILTER
capability
policy
data residency
context size
structured output
tool support
breaker state
rate / quota / budget
↓
ROUTING
static task map
OR
bounded score among eligible targets
↓
ROUTE DECISION
selected
excluded + reasons
fallback order
policy version
↓
PROVIDER / LOCAL ADAPTER
↓
MODEL ENDPOINT
↓
NORMALIZE
output
tool proposal
usage
finish reason
provider flags
↓
VALIDATE CONTRACT
↓
RETURN TO CALLER
CROSS-CUTTING:
№34
decides when stronger model is worth/needed
№63
retry / breaker / availability fallback
№64
rate / quota / budget / reservation
№51
provider credentials and secret handling
№46
traces / latency / tokens / errors
№61
exact model/version lineage
№70
serving capacity and inference runtime
KEY DISTINCTIONS:
GATEWAY
= HOW THE SYSTEM TALKS TO MODELS
MODEL ROUTING
= WHICH ELIGIBLE TARGET GETS THIS CALL
MODEL ESCALATION
= WHETHER THE TASK NEEDS A STRONGER TIER
FALLBACK
= WHAT TO DO WHEN THE CHOSEN TARGET IS UNAVAILABLE
CORE PRINCIPLE:
THE APPLICATION SHOULD NOT SAY:
"CALL PROVIDER X MODEL Y
WITH SDK Z."
IT SHOULD SAY:
"I NEED THIS MODEL CAPABILITY,
UNDER THESE QUALITY,
LATENCY, COST, POLICY
AND DATA CONSTRAINTS."
THE GATEWAY THEN CHOOSES
AN ELIGIBLE TARGET,
ENFORCES THE RULES,
RECORDS THE DECISION,
AND HIDES PROVIDER-SPECIFIC
MECHANICS FROM THE REST
OF THE SYSTEM.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №65 Model Gateway & Model Routing.
B–E. Existing boundary and placement. The existing conceptual boundary, class PRODUCTION, default ON and owner Model Router remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.