Local LLM / Self-Hosted Models — стратегия, при которой организация сама владеет inference deployment: model artifacts, runtime, hardware, network boundary, обновлениями, capacity и эксплуатацией. Это может быть ноутбук с одной моделью, workstation с GPU, private server, on-prem cluster или полностью air-gapped среда.
№70 Model Serving & Inference владеет batching/KV/scheduler/throughput/latency mechanics; №71 владеет deployment choice, hardware topology, artifact ownership, private network, operations and economics. №65 Model Gateway & Routing выбирает local/cloud target и скрывает endpoint details from application. №72 Quantization меняет precision/representation of model weights; №71 decides which artifact to deploy but does not own quantization algorithms. №39 Fine-Tuning/LoRA/Distillation creates adapted weights/adapters; №71 hosts approved resulting artifacts. №76 Governance sets data residency/privacy rules; №71 is one way to satisfy them, not governance itself.
Prerequisites: №39 Fine-Tuning/LoRA/Distillation, №46 Observability, №51 Permissions & Secrets, №63 Resilience, №64 Limits/Budgets, №65 Model Gateway, №70 Model Serving. Forward references: №72 Quantization, №73 Multimodal AI, №76 Data Governance, №77 Production AI Architecture.
REQUEST-TIME: local inference target selected by №65; №70 executes. CONTROL PLANE: approved models, licenses, hardware topology, deployment configs, network/data policy, update channel. DATA PLANE: inference traffic, model weights in memory, local KV/cache and runtime telemetry. OFFLINE: artifact acquisition, verification, benchmark, rollout, capacity planning, patching, license/security review and disaster recovery testing.
Success: approved model revision is served within declared SLO/capacity/privacy boundary. Retryable: replica/runtime restart, transient device/runtime fault, private endpoint overload. Permanent: model artifact incompatible with hardware/runtime, insufficient memory, invalid license/policy, corrupted artifact. Persist: model/tokenizer revision, artifact hash, license, deployment config, hardware class, runtime version, benchmark profile, rollout status. Security: model artifacts and serving endpoints are trusted infrastructure assets, not arbitrary files/URLs chosen by model/user.
№71 не владеет inference scheduling internals, model routing policy generally, quantization methods, fine-tuning, prompt architecture or data governance. Она владеет THE DECISION, TOPOLOGY AND OPERATIONS OF RUNNING MODEL DEPLOYMENTS UNDER YOUR OWN INFRASTRUCTURE CONTROL.
Sensitive prompts/documents must remain inside organization-controlled infrastructure.
Inference must occur in a specific country/region/on-prem environment.
Air-gapped, field, industrial, secure or intermittently connected environments.
Predictable sustained volume may amortize hardware better than per-token API spend.
Local inference can remove WAN/provider round-trip when model/hardware are sufficient.
Fine-tuned/private weights unavailable as managed provider endpoint.
Idle self-hosted accelerator still costs capex/opex.
Cloud provider may absorb unpredictable peaks better than your fixed hardware.
Some strongest models may only exist as hosted services.
No one to own driver/runtime patches, failures, monitoring, replacement hardware and upgrades.
Single user, development, offline assistant, experimentation. Operational HA usually not required.
One powerful machine with one/multiple accelerators; good for small team/private workload.
Rack/on-prem/private cloud node behind internal gateway; easier shared access and monitoring.
Multiple serving nodes/replicas, model pools, autoscaling, orchestration, dedicated observability.
Factory/vehicle/branch/device deployment with tight power/memory and connectivity limits.
Artifacts, patches and evaluation data enter through controlled transfer process.
Dedicated infrastructure under organization control; still remote, but policy/ops differ from public API.
№65 chooses among local/private/cloud targets under policy and quality constraints.
Primary accelerator memory capacity for model artifact and active requests.
Autoregressive decode can be strongly memory-bandwidth-sensitive.
Cross-GPU/node communication can dominate distributed inference.
Used for model loading, CPU offload, preprocessing, artifact cache and OS/runtime.
Large weight files affect deployment/warm-start time and disk requirements.
Power delivery, UPS and circuit limits matter for sustained accelerator load.
Cooling determines whether benchmark speed can be maintained over hours.
Tokenization, preprocessing, networking and some inference/offload workloads use CPU resources.
DEVICE MEMORY BUDGET model weights + runtime workspace + KV cache for active requests + temporary tensors + fragmentation/headroom = required VRAM SYSTEM MEMORY BUDGET OS + serving runtime + tokenizer / preprocessing + model loading buffers + CPU-offloaded layers / KV if used + artifact cache + logs / side processes = required RAM IF using CPU offload: model may "fit" but token latency can increase sharply because data moves across CPU/GPU boundary. FIT IS BINARY. USABLE PERFORMANCE IS NOT.
Useful where GPU unavailable, throughput requirement low or edge deployment values simplicity.
Some layers/KV can live in system RAM, allowing larger models than VRAM alone.
PCIe/system-memory movement can make interactive decode much slower.
{
"model_ref": "model://local/model-x",
"publisher": "...",
"family": "model-x",
"revision": "commit-or-release-id",
"weights_sha256": "...",
"tokenizer_sha256": "...",
"format": "...",
"precision": "BF16",
"license": {
"id": "...",
"approved": true,
"review_ref": "legal://..."
},
"source_uri": "approved-registry://...",
"acquired_at": "...",
"runtime_compatibility": ["runtime-profile-v3"],
"status": "APPROVED"
}Check exact license terms for organization/product use.
Important for edge/on-prem customer deployments.
License may impose conditions on modified models/adapters.
Some licenses contain acceptable-use or field-of-use conditions.
Do not load arbitrary model repositories/files directly into production runtime.
Verify approved artifact hash before deployment.
Avoid model packages requiring arbitrary remote/custom code unless reviewed and isolated.
Inspect package format, metadata, dependencies and runtime extensions before approval.
{
"model_ref": "model://local/model-x",
"deployment_ref": "serve://eu-private/model-x/g12",
"capabilities": {
"text": true,
"vision": false,
"tools": true,
"context_tokens": 32768
},
"policy": {
"data_classes": ["PUBLIC","INTERNAL","CONFIDENTIAL"],
"region": "PRIVATE-EU"
},
"quality_tier": "STANDARD",
"latency_tier": "LOW",
"cost_tier": "OWNED_CAPACITY",
"health": "HEALTHY",
"capacity": {
"queue_headroom": 0.72,
"kv_headroom": 0.61
}
}Application should not know:
№65 sees capability/policy/health/cost class and routes accordingly.
Use for routine classification, extraction, rewriting, simple Q&A and confidential workloads.
Task fails quality floor or exceeds capability.
Cloud/private bigger model only if №34 escalation + №65 policy permit it.
| Data class | Local unavailable | Safe default |
|---|---|---|
| PUBLIC | Private model down | Route to compatible hosted model if policy/cost allows. |
| INTERNAL | Private model down | Only approved enterprise/provider endpoints. |
| CONFIDENTIAL | Private model down | Fail/queue/route to another approved private deployment. |
| REGULATED / AIR-GAPPED | Local model down | No network fallback; degrade or wait locally. |
GPU/server/network/storage purchase or reserved infrastructure cost.
24/7 accelerators consume power even when partially utilized.
Deployment, patching, monitoring, incident response, capacity planning and replacement.
Self-hosted economics worsen rapidly when accelerator utilization is low.
Cheaper local model may require more retries, human review or escalation.
Provider redundancy may be expensive, but your own outage has real cost too.
Some workloads justify higher TCO because external processing is unacceptable.
Model sizes/runtimes evolve faster than typical server depreciation plans.
SELF-HOSTED MONTHLY TCO hardware amortization / rental + electricity / cooling + storage / network + operations labor + support / replacement reserve + idle capacity cost = monthly_self_hosted_cost UNIT COST monthly_self_hosted_cost -------------------------------- successful production tasks COMPARE WITH: hosted model API cost + network / provider extras + retries / escalation + quality-related human review IMPORTANT: Do the comparison at the SAME: quality threshold context/output distribution latency target availability target privacy class Otherwise the break-even math is fake.
Serve several compatible internal applications through one gateway/model pool.
Run offline indexing/evaluation/synthetic workloads when interactive demand is low.
Possible if startup latency and hardware/cloud economics make dynamic scale-down useful.
Small model may meet quality floor at much lower compute cost.
Constrained schema + verifier can make smaller local model sufficient.
Style/format transforms may not require frontier reasoning model.
Local model can handle cheap helper steps while hard synthesis escalates.
Hard reasoning/long-context tasks may benefit from stronger local model.
May require multi-GPU/node topology and faster interconnect.
Longer load time, harder rolling deploy, more expensive idle capacity and larger blast radius.
Accelerator driver version must support runtime stack.
Kernel support, model architecture and precision features vary by version.
Not every runtime supports every model architecture/quantization format.
Reproducible images reduce «works on one node» drift.
Power, GPU, driver, motherboard or OS failure stops local AI entirely.
Independent node/replica with same approved model revision.
№65 stops traffic to unhealthy deployment and selects compatible alternate.
Queue work, use smaller local model or policy-approved hosted target.
Approved weights/tokenizer/config stored in controlled artifact repository.
Serving runtime, model registration, policies and network config reproducible.
After restore, compare model checksum and performance/quality smoke tests before traffic.
Local serving can still dump sensitive prompts/responses to disk/log aggregator.
Sensitive tensors/text may end up in system artifacts depending on OS/runtime.
Logs/artifacts copied elsewhere can violate intended local-only boundary.
Collect latency/tokens/errors without raw payloads unless explicitly approved.
USER / APP
↓
AUTH / API LAYER
↓
№65 MODEL GATEWAY
policy
tenant
quota
route
↓
PRIVATE NETWORK
↓
LOCAL MODEL POOL
serve://model-a
serve://model-b
↓
NO DIRECT USER ACCESS
ADMIN PATH:
separate identity
separate network / VPN / bastion
audited deployment commands
MODEL ARTIFACT PATH:
approved registry
↓
controlled artifact repository
↓
serving node cache
NOT:
serving node pulls arbitrary URL
given by user/model at runtime
Weights, runtime images and patches enter via approved media/process.
All runtime dependencies/artifacts available inside secure zone.
Quality and security test suites must also live in the isolated environment.
Security/runtime updates take longer due to controlled transfer/review.
Serving receives workload class/tenant metadata from host-controlled path.
Tenant/workload quotas map to local queue/KV/concurrency budgets.
Ensure one request cannot access another request's cache/state/log content.
Approves model revision, eval thresholds, license and intended task classes.
Runtime, hardware, deployment, monitoring, capacity and incident response.
Artifact source, network/data policy, retention and access review.
Defines what degradation/outage behavior is acceptable.
Ready time by model pool/node/revision.
Memory, temperature, power, utilization, hardware/runtime errors.
Weights + KV + runtime occupancy and OOM protection margin.
Hardware + energy + operations divided by verified useful output.
How much owned accelerator capacity produces needed production work.
Runtime/node/model failures → healthy serving again.
Which exact model/runtime artifacts are loaded where.
Share of local-first tasks that require stronger external target.
Same representative eval set used for hosted candidates.
Real prompt/output distributions and concurrency.
Tokens/tasks per second under SLO, not maximum synthetic peak.
Thermals, fragmentation, leaks, runtime stability and hardware faults.
Router/fallback behavior under local deployment outage.
Useful tasks/tokens per energy/cost when economics matter.
p95 context lengths at expected concurrency.
Artifact load + runtime warmup duration.
Health removes target and returns it only after model reload/readiness.
Traffic routes to replica/degradation policy.
Deployment does not corrupt model cache or block all inference silently.
Typed health/fallback behavior; no infinite hangs.
Node refuses to load invalid weights.
Sustained throughput stays within expected envelope without throttling surprises.
Restricted data class correctly fails/queues locally.
Previous approved deployment can be restored quickly.
Evaluates FP16/BF16/INT8/INT4-like formats, calibration and quality/performance trade-offs.
Chooses approved quantized artifact if it meets quality and hardware economics, then operates it via №70.
Fine-tuning pipeline produces candidate artifact and eval results.
Merge/reference base model + adapter versions explicitly.
License, checksum, security, benchmark, canary, rollback and registry entry.
Choose task-sufficient artifact for constrained device.
Precision reduction can reduce memory/storage and enable local execution.
Artifact rollout must survive intermittent connectivity.
Sustained inference may be power-limited rather than compute-limited.
| Aspect | Developer local | Production self-hosted |
|---|---|---|
| Availability | Best effort. | Declared SLO / monitoring / recovery. |
| Model revision | May change manually. | Pinned, approved, auditable. |
| Security | Single-user trust. | Network/auth/tenant/data policy. |
| Capacity | One user. | Concurrent workload + overload behavior. |
| Updates | Manual. | Canary/rollback/versioned deployment. |
| Observability | Console. | Metrics, alerts, fleet visibility. |
Full local TCO divided by verified useful production tasks.
Local model success by task segment vs hosted alternatives.
Owned capacity producing needed work under SLO.
Ready endpoint uptime by model pool/revision.
Node/runtime/model incident → service restored.
Share of local-first tasks requiring stronger target.
TTFT/TPOT target compliance from №70.
Restricted workloads never escape approved private boundary.
self_hosted_models/
├── model_registry.yaml
├── artifacts/
│ └── approved-model-manifest.json
├── deployment/
│ ├── container-or-service-config
│ └── runtime.env.example
├── benchmarks/
│ ├── workload.json
│ └── results.json
├── policies/
│ └── data-routing.yaml
└── runbook/
├── deploy.md
├── rollback.md
└── incident.md
MVP FLOW:
approve model artifact
↓
verify checksum/license
↓
load to one controlled node
↓
serve through №70 runtime
↓
register endpoint in №65
↓
allow only explicit task/data classes
↓
measure:
quality
TTFT / TPOT
concurrency
VRAM
uptime
TCO
↓
compare with hosted baseline
↓
keep / scale / abandonIf this MVP does not produce quality/privacy/economic value, a cluster will not fix the architecture.
| Observed need | Upgrade |
|---|---|
| Single node meets quality but saturates | Add horizontal serving replicas via №70. |
| Model does not fit one accelerator | Multi-GPU serving topology or evaluate №72 quantization / smaller sufficient model. |
| Multiple task classes need different models | Create explicit local model pools registered in №65. |
| Availability matters | Second independent node, health-aware routing, artifact replication and tested failover. |
| Air-gapped production | Internal artifact registry, offline update/eval pipeline and transfer controls. |
| Costs still high | Increase utilization, route smaller models, batch offline work, evaluate quantization before buying more hardware. |
| Quality gap remains | Use hybrid escalation or №39 adaptation only if evals justify it. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Условно. Только при понятной privacy, offline, economics, latency или custom-model причине. |
| Separate Component? | YES. Private model deployment/pool is a distinct infrastructure target behind Model Gateway. |
| Минимум 80% ценности? | Approved artifact, one node/runtime, private endpoint, route policy, realistic benchmark, capacity/TCO profile, monitoring and rollback. |
| Когда overkill? | GPU cluster for low-volume general chat that hosted API serves cheaply and better. |
| Trigger? | Data cannot leave boundary, workload is large/stable, offline requirement exists, custom weights are required or measured TCO/latency clearly wins. |
| Как измерить uplift? | Quality pass rate, cost per successful task, utilization, latency SLO, availability, privacy-policy coverage and cloud escalation rate. |
| Можно ли rule/tool/code вместо LLM-agent? | Да. Deployment, routing eligibility, updates, checksums and capacity are deterministic infrastructure. No «self-hosting agent» is required. |
Privacy, offline, economics, latency or custom artifacts — not fashion.
Choose by task evals, not maximum model that fits hardware.
Weights, tokenizer, runtime, license and checksum are deployment identity.
Same routing/policy/quotas/audit boundary as hosted models.
Hardware + energy + people + idle + quality gap + downtime.
Logs, swap, backups and observability remain data-governance surfaces.
Approved source, checksum, no arbitrary remote code.
Never auto-pull latest into production.
One-node MVP must show value before cluster complexity.
WHY SELF-HOST?
privacy?
data residency?
offline?
stable high volume?
latency locality?
custom model?
↓
IF NO STRONG DRIVER:
HOSTED API MAY BE BETTER
IF YES:
↓
DEFINE TASKS
↓
QUALITY FLOOR / EVALS
↓
CHOOSE SMALLEST SUFFICIENT MODEL
↓
MODEL ARTIFACT REVIEW
exact revision
weights checksum
tokenizer checksum
license
security review
↓
CHOOSE DEPLOYMENT TOPOLOGY
desktop
workstation
private server
cluster
edge
air-gap
↓
HARDWARE PROFILE
VRAM
RAM
bandwidth
storage
interconnect
power
thermals
↓
№70 SERVING RUNTIME
batching
KV cache
scheduler
admission
health
streaming
↓
PRIVATE MODEL ENDPOINT
↓
REGISTER IN №65
capability
data classes
quality tier
latency
cost class
health
capacity
↓
MODEL ROUTING
PUBLIC / LOW-RISK:
local OR hosted
CONFIDENTIAL:
approved local/private only
HARD TASK:
№34 escalation
↓
stronger eligible target
only if policy permits
OPERATIONS:
artifact repository
↓
node cache
↓
load exact revision
↓
warmup
↓
health
↓
serve
↓
monitor
↓
canary update
↓
rollback if regression
ECONOMICS:
self-hosted cost
=
hardware
+ energy
+ cooling
+ storage/network
+ operations labor
+ idle capacity
+ downtime
+ quality gap
measure:
COST PER SUCCESSFUL TASK
NOT:
raw GPU price
raw token throughput alone
BOUNDARIES:
№65
routes among local/private/cloud targets
№70
executes inference efficiently
№71
owns deployment strategy and operations
№72
changes precision/compression artifact
№76
defines data/privacy/governance rules
CORE PRINCIPLE:
SELF-HOSTING DOES NOT
REMOVE A PROVIDER.
IT MAKES YOU
THE PROVIDER.
YOU NOW OWN:
THE WEIGHTS,
THE HARDWARE,
THE RUNTIME,
THE SECURITY BOUNDARY,
THE CAPACITY,
THE FAILURES,
THE PATCHES,
THE LICENSE,
THE COST
AND THE UPTIME.
THAT IS A GREAT TRADE
WHEN CONTROL MATTERS.
IT IS A BAD TRADE
WHEN THE ONLY REASON IS
"API SEEMS EXPENSIVE."
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №71 Local LLM / Self-Hosted Models.
B–E. Existing boundary and placement. The existing conceptual boundary, class SPECIALIZED, default CONDITIONAL and owner Model Router + Model Infrastructure remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.