Model Serving — production-слой, который загружает model weights, принимает inference requests, планирует их на accelerator-ах, управляет batching, KV cache, concurrency, streaming, memory и replicas и превращает вычисление модели в надёжный service endpoint.
№65 Model Gateway & Routing normalizes requests and selects eligible model/provider/endpoint; №70 owns runtime execution inside selected inference endpoint: batching, accelerator scheduling, memory/KV, streaming, throughput and health. №71 Local LLM / Self-Hosted owns deployment choice/topology/privacy/operations strategy; №70 is serving mechanics regardless whether endpoint is on one workstation, private cloud or cluster. №72 Quantization owns precision/compression choices; №70 consumes an already selected model artifact/precision and measures serving effects. №64 Limits/Budgets owns higher-level quotas; №70 has local admission/capacity limits. №63 Resilience owns retry/breaker semantics; serving exposes health/typed overload failures.
Prerequisites: №46 Observability, №50 Contracts, №51 Permissions & Secrets, №63 Resilience, №64 Limits/Budgets, №65 Model Gateway. Forward references: №71 Local LLM / Self-Hosted Models, №72 Quantization, №73 Multimodal AI, №75 Voice/Realtime, №76 Data Governance, №77 Production Architecture.
REQUEST-TIME: admission, scheduling, prefill/decode, batching, streaming, cancellation. CONTROL PLANE: model revision, precision, max context, batch policy, replica topology, scheduler policy, capacity/SLO targets. DATA PLANE: tokens/tensors, KV-cache blocks, request queues, streaming deltas, usage metrics. OFFLINE: model loading/build artifacts, warmup, benchmark, profiling, capacity planning, regression/canary tests.
Success: accepted request completes or streams output within declared limits and exact model revision. Retryable: transient serving unavailable, replica crash, some overload/timeouts if upstream budget allows. Permanent: context too large, unsupported modality/config, model revision absent, hard memory requirement impossible. Idempotency: inference is usually computation-only; retry may produce different text because generation can be nondeterministic. Persist/trace: request/model revision, queue delay, batch, prefill/decode timing, token counts, cancellation/failure reason, replica/accelerator identity.
№70 не владеет provider abstraction, cognitive model selection, model training/fine-tuning, self-hosting business decision, quantization algorithms, prompt design or application-level retries. Она владеет THE EXECUTION RUNTIME THAT TURNS A MODEL ARTIFACT INTO A CAPACITY-MANAGED INFERENCE SERVICE.
Модель обрабатывает весь входной контекст, вычисляет hidden states and creates KV cache. Long prompts increase prefill compute and memory. This stage often benefits from high parallel compute throughput.
Каждый следующий token depends on previous tokens. Decode is iterative, latency-sensitive and heavily influenced by memory bandwidth, KV-cache footprint and batching.
AVAILABLE DEVICE MEMORY ┌──────────────────────────────────────┐ │ MODEL WEIGHTS │ ├──────────────────────────────────────┤ │ KV CACHE FOR ACTIVE REQUESTS │ │ request A: long context │ │ request B: medium context │ │ request C: decoding │ ├──────────────────────────────────────┤ │ TEMP / ATTENTION / KERNEL WORKSPACE │ ├──────────────────────────────────────┤ │ RUNTIME / FRAGMENTATION / HEADROOM │ └──────────────────────────────────────┘ DO NOT plan at 100% theoretical memory. Admission must leave safety headroom for runtime allocations and fragmentation.
Parallel work amortizes kernel launch/compute overhead and increases throughput.
Interactive request may sit idle while server waits to form batch.
Static batches waste compute when requests finish at different times or have uneven prompt/output lengths.
Good baseline but long requests can create head-of-line blocking.
Separate urgent user traffic from background batch jobs.
No single user/tenant monopolizes decode slots/KV memory.
Schedule requests that can still meet deadline; reject those already impossible.
Reject/defer when waiting requests exceed useful threshold.
Estimate input + max output impact before admission.
Estimated queue + service time should fit remaining deadline.
Reserve some capacity for interactive/high-priority traffic.
| Limit | Serving impact | Control |
|---|---|---|
| MAX INPUT TOKENS | Prefill compute and KV footprint. | Reject/truncate upstream according to context policy. |
| MAX OUTPUT TOKENS | Decode time + future KV growth. | Hard generation ceiling. |
| MAX TOTAL TOKENS | Protect model context limit. | Input + reserved output ≤ model supported window. |
| MAX CONCURRENT TOKENS | Aggregate KV/cache pressure. | Scheduler/admission-level global capacity bound. |
User sees response start before full generation ends.
Text/tool-call chunks are sent as decode progresses.
Retry cannot silently concatenate another model's fresh generation as if nothing happened.
Repeated exact token prefix can reuse precomputed KV-like state/runtime cache depending on serving implementation.
Especially useful for long stable system prompts or shared document prefixes.
Cache key must bind model revision, tokenizer, relevant runtime settings and exact prefix tokens.
Can take seconds/minutes for large models.
Allocate serving structures and distributed groups.
Trigger lazy compilation/kernel initialization and measure readiness.
Only after model revision is loaded and health/capacity checks pass.
/health while model weights are still loading.New replica not immediately available; large model artifact transfer dominates.
Keep baseline ready capacity for interactive traffic.
Scale ahead of known traffic windows instead of after queue already explodes.
Overall generated token throughput or per-request decode rate.
Queue + prefill + scheduling until first streamed token.
Average decode interval after first token.
Spacing/jitter between streamed tokens.
How long accepted request waits before execution.
Should approach zero under correct admission.
Memory used/reserved for active sequence state.
Helpful alongside useful throughput and SLOs, not alone.
TTFT =
request validation
+ admission decision
+ queue wait
+ batch scheduling delay
+ prefill compute
+ first decode step
+ transport/stream flush
Example:
validation 5 ms
queue 1200 ms ← bottleneck
prefill 420 ms
first decode 35 ms
transport 15 ms
-----------------------
TTFT 1675 ms
"model is slow" would be the wrong diagnosis.
Large active batches, more queue tolerance, maximize aggregate tokens/sec and cost efficiency.
Small queue, reserved headroom, bounded batch delay, priority scheduling, lower TTFT/TPOT.
One huge context can monopolize compute/memory and increase TTFT for short requests.
Long-context/batch requests can use different queues/pools/priorities.
Some serving schedulers can interleave/chunk prefill or preempt lower-priority sequences.
Interactive/realtime work enters saturated server.
Pause sequence, evict/recompute KV, or reserve capacity beforehand.
Preemption can destroy efficiency and should be measured, not assumed free.
Good baseline; one failure domain and limited concurrency.
Each replica loads model and handles independent request batches.
Route by queue/KV/headroom rather than naive round-robin where possible.
Gateway/router stops new traffic to unhealthy replica and retries/fallbacks upstream.
One layer computation is partitioned across multiple accelerators; requires fast interconnect and synchronization.
Different layer groups live on different devices; introduces pipeline scheduling/bubbles.
Expert/model/sequence parallel strategies exist; choose only when model/scale demands it.
Full model copy; handles batch A.
Select healthy replica with capacity/headroom.
Full model copy; handles batch B.
Persistent rising queue time means demand exceeds ready capacity.
Active context load may saturate before compute utilization does.
Compare delivered work to capacity profile.
Scale based on user-facing service target.
| Workload dimension | Why it matters |
|---|---|
| Input token distribution | Determines prefill compute and initial KV footprint. |
| Output token distribution | Determines decode duration and active sequence lifetime. |
| Concurrency arrival pattern | Steady 10 RPS differs from bursts of 100 every minute. |
| Streaming vs non-streaming | Changes SLO and connection/runtime behavior. |
| Sampling / structured output / tools | Can alter decode behavior and termination length. |
| Priority classes | Interactive/batch mix changes scheduler target. |
CAPACITY PROFILE: model-X / revision-R / precision-P interactive profile: p50 input = 2k tokens p95 input = 8k p50 output = 500 p95 output = 2k safe envelope: concurrency <= ... aggregate active tokens <= ... KV occupancy <= ... queue delay p95 <= ... TTFT p95 <= ... TPOT p95 <= ... OOM = 0 overload rejection <= target IF workload shifts to: 32k input 8k output the old "RPS capacity" is no longer valid.
{
"deployment_ref": "serve://model-x/prod-eu/g17",
"model_ref": "model://model-x",
"model_revision": "sha256:...",
"tokenizer_revision": "sha256:...",
"precision": "BF16",
"runtime_version": "serve-runtime-v...",
"max_context": 131072,
"parallelism": {
"tensor": 1,
"pipeline": 1
},
"scheduler_policy": "interactive-v3",
"status": "READY",
"capacity_profile_ref": "capacity://..."
}Runtime process/event loop is alive and not irrecoverably wedged.
Exact model loaded, scheduler healthy, enough resources, not draining.
Optional low-frequency synthetic inference checks model/runtime path end-to-end.
Return typed OVERLOADED / retry-after estimate before consuming model resources.
Queue offline job if waiting remains useful and queue is bounded.
Expose saturation/headroom so №65 can choose another healthy compatible endpoint.
Account for workspace, fragmentation and transient allocations.
Admission considers input + reserved max output.
Per model/device limit protects memory.
If runtime state becomes unsafe, drain/restart replica rather than continuing corrupted capacity accounting.
Readiness false / router removes endpoint from new selections.
Let accepted generations complete or cancel based on shutdown deadline.
Only after active requests are zero or termination policy reached.
Accept several verified draft tokens per expensive target-model step when acceptance rate is good.
Draft model and verification can hurt if workload/model combination has low acceptance or high overhead.
Compare TPOT, throughput, memory and quality-equivalence under real workload.
FP/BF/INT formats, calibration, compression and quality/performance implications.
Load time, memory footprint, supported kernels, throughput, TTFT/TPOT, batching and replica capacity of that artifact.
Privacy, economics, hardware, operations burden, offline/local requirements, deployment topology.
Batching, KV, scheduler, replicas, memory, metrics, SLO, health and capacity.
№71 chooses topology; №70 supplies measured capacity characteristics of that topology.
Media preprocessing may occur before model runtime.
Additional model stage can dominate prefill-like latency.
Images/video frames can expand into many internal tokens/features.
Mixed modalities complicate batching and memory planning.
Realtime UX suffers from bursty token/audio delivery even if average throughput looks good.
Realtime sessions may need dedicated pool or priority class.
Barge-in requires immediate stop/release of current generation.
Serving endpoint usually should not be directly exposed to untrusted clients.
№65/host authenticates and normalizes request before serving.
Tenant/workload class comes from trusted context, not model/user spoofing.
Serving metrics should not dump prompts/responses by default.
p50/p95/p99 by model, workload class and prompt length bucket.
Decode performance after first token.
Total serving throughput and per-request rate.
Admission-to-schedule waiting time.
Allocated/free blocks, fragmentation, active-token footprint.
Compute/memory utilization correlated with useful throughput.
Target near zero; indicates bad admission/capacity profile.
Typed capacity rejects vs accepted traffic/SLO.
Measure raw prefill/decode latency without queueing.
Find throughput plateau and where TTFT/queue starts exploding.
Short/long prompts and outputs together reveal scheduler fairness.
Detect memory leaks, fragmentation, KV cleanup bugs and performance drift.
Verify admission rejects before memory exhaustion.
Resources are released promptly under mass cancellation.
Router stops traffic; pool recovers without thundering herd.
Compare new runtime/model artifact on same workload.
model_serving/ ├── server.py ├── contracts.py ├── scheduler.py ├── admission.py ├── streaming.py ├── health.py ├── metrics.py ├── deployment.py └── benchmarks/ InferenceRequest: request_id model_revision input max_output_tokens sampling stream deadline workload_class InferenceRuntime: validate() estimate_memory() admit() schedule() prefill() decode() stream() cancel() release() ServingDeployment: exact model/tokenizer revision precision runtime version max context queue cap active token cap concurrency cap health state capacity profile
Start on one device/node if the model fits. Add replicas before complex cross-device sharding when throughput is the main problem.
| Observed bottleneck | Potential upgrade |
|---|---|
| Queue/SLO saturation but model fits one device | Add horizontal replicas and capacity-aware load balancing. |
| Model does not fit one device | Tensor/pipeline/model sharding across accelerators. |
| Decode latency dominates | Profile kernels/bandwidth, serving scheduler, speculative decoding; later evaluate quantization №72. |
| Prefill dominates | Prefix reuse, prompt-length classes, chunked/interleaved prefill where runtime supports it. |
| KV memory limits concurrency | Better paged/block KV management, lower precision where validated, tighter admission, more memory/replicas. |
| Cold starts hurt SLO | Warm replica floor, artifact locality, pre-scaling. |
| Multiple local models | Dedicated model pools, load/unload policy, №65 capacity-aware routing. |
| Realtime workload appears | Separate low-latency serving class/pool integrated with №75. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Да, если вы реально владеете inference runtime. Если только вызываете hosted API — provider уже реализует большую часть №70. |
| Separate Component? | YES. Inference server/serving pool is distinct production component behind №65 Gateway. |
| Минимум 80% ценности? | One serving endpoint, bounded admission, batching, KV accounting, streaming/cancel, health, exact revision, realistic SLO metrics, safe rollout. |
| Когда overkill? | Distributed tensor/pipeline parallel cluster for a model that easily fits one accelerator and has low traffic. |
| Trigger? | Need self/private model endpoint, predictable latency/cost, local model capacity, or direct control over batching/memory/throughput. |
| Как измерить uplift? | TTFT/TPOT, tokens/sec, queue delay, SLO success, accelerator utilization, OOM rate, cost per generated token/successful request, capacity headroom. |
| Можно ли rule/tool/code вместо LLM-agent? | Да, полностью. Serving scheduler/admission/health are deterministic systems. LLM itself is the workload, not the serving controller. |
They have different bottlenecks and workload sensitivity.
Concurrency is bounded by active sequence memory, not only model weights.
Overload should reject/defer before latency and memory collapse.
Throughput and interactive latency must be balanced explicitly.
Stop generation when output is no longer needed.
Do not route traffic before load/warmup completes.
Weights, tokenizer, precision and runtime version belong in trace/provenance.
Prompt/output length and concurrency mix determine actual capacity.
Replicate, shard or optimize only after profiling shows what limits SLO/cost.
№65 MODEL GATEWAY / ROUTER
↓
SELECT SERVING DEPLOYMENT
model revision
region
capability
health
capacity
↓
INFERENCE REQUEST
input
max output
stream
deadline
workload class
↓
SERVING ENDPOINT
↓
VALIDATE
model loaded?
context valid?
modality supported?
↓
ADMISSION
queue capacity
active token budget
KV memory
concurrency
deadline feasibility
↓
SCHEDULER
FIFO / priority / fair share
↓
PREFILL
process input tokens
build KV state
↓
CONTINUOUS BATCH
active sequences enter/leave
↓
DECODE
token by token
↓
STREAM
TTFT
inter-token latency
↓
FINISH / CANCEL
↓
FREE KV / CAPACITY
↓
REPORT
tokens
queue delay
prefill
decode
TTFT
TPOT
replica
model revision
CAPACITY MODEL:
accelerator memory
=
weights
+ KV cache
+ runtime/workspace
+ safety headroom
OVERLOAD:
do not accept unlimited work
↓
reject / defer / route away
SCALING:
if model fits one device:
add replicas first for throughput
if model does not fit:
model parallelism
if decode is slow:
profile scheduler / bandwidth /
kernels / precision / speculative options
if prefill is slow:
inspect prompt length /
prefix reuse / prefill scheduling
if KV is full:
tighten admission /
better KV management /
more memory or replicas
BOUNDARIES:
№65
WHICH model endpoint should receive the call?
№70
HOW does that endpoint execute the call efficiently?
№71
SHOULD / HOW do we operate models ourselves?
№72
IN WHAT precision/compressed form is model represented?
CORE METRICS:
TTFT
time until first token
TPOT / ITL
speed and smoothness of decode
TPS
useful token throughput
QUEUE DELAY
overload signal
KV OCCUPANCY
active memory pressure
OOM
admission correctness failure
SLO SUCCESS
real product target
CORE PRINCIPLE:
MODEL SERVING
IS NOT "THE GPU IS BUSY."
A GOOD SERVING SYSTEM
KEEPS THE MODEL
BUSY ENOUGH
WITHOUT DESTROYING
LATENCY, FAIRNESS,
MEMORY SAFETY
OR RELIABILITY.
THE GOAL IS NOT
MAXIMUM UTILIZATION.
THE GOAL IS
PREDICTABLE,
MEASURED,
COST-EFFICIENT
INFERENCE
UNDER REAL WORKLOAD.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №70 Model Serving & Inference.
B–E. Existing boundary and placement. The existing conceptual boundary, class PRODUCTION, default CONDITIONAL and owner Model Infrastructure + Model Router + Production Fabric remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.