Quantization — представление model weights и/или activations в более низкой числовой точности, чтобы уменьшить память, storage, bandwidth и иногда ускорить inference. Вместо хранения/вычисления всего в FP32/BF16/FP16 система может использовать INT8, INT4, FP8-подобные или другие low-bit formats — полностью или частично.
№70 Model Serving owns runtime batching/KV/scheduler/latency and benchmarks the quantized artifact; №72 owns precision/compression transformation and quality/performance validation. №71 Self-Hosted owns deployment/topology decision and chooses which approved artifact to run. №39 Fine-Tuning/LoRA/Distillation changes model behavior/weights through learning; quantization primarily changes numeric representation for inference/storage efficiency. №65 Gateway may route to a quantized deployment but does not choose quantization algorithm. №40 Eval-Driven Optimization / №47 Evals provide measurement discipline; №72 defines quantization-specific comparisons.
Prerequisites: №39 Fine-Tuning/LoRA/Distillation, №40 Eval-Driven Optimization, №46 Observability, №47 Evals, №65 Model Gateway, №70 Model Serving, №71 Local/Self-Hosted. Forward references: №73 Multimodal AI, №77 Production AI Architecture.
REQUEST-TIME: no special policy logic; serving executes quantized kernels. CONTROL PLANE: approved precisions, quantization configs, calibration data version, hardware/runtime compatibility, quality thresholds. DATA PLANE: low-precision weights/scales/zero-points/metadata, possibly activation/KV representations. OFFLINE: quantize, calibrate, evaluate, benchmark, package, sign/checksum, promote/rollback.
Success: candidate artifact loads on target runtime, remains within quality floor, improves intended resource metric and preserves serving stability. Permanent failure: unsupported kernel/format, unacceptable quality regression, incompatible tokenizer/model revision, numerical instability. Persist: base model hash, quantization method, bit-width, group size, scales scheme, calibration dataset hash, tool/runtime version, eval report, benchmark report. Security: quantized artifact is still a model artifact and follows the same supply-chain approval as №71.
№72 не владеет general model compression, distillation, pruning, serving scheduler, GPU allocation, self-hosting strategy or model routing. Она владеет NUMERIC PRECISION REDUCTION OF MODEL RUNTIME ARTIFACTS AND THE VALIDATION OF ITS QUALITY/PERFORMANCE TRADE-OFF.
Low-bit weights reduce disk/object-store footprint and model distribution time.
Smaller weight footprint can make model fit one device or leave more memory for KV cache/concurrency.
Decode often benefits when fewer bytes must move from accelerator memory.
Supported low-precision kernels can improve tokens/sec or reduce hardware requirement.
Weights stored in lower precision; activations may remain FP16/BF16. Often simpler and safer for LLM inference.
Can improve compute efficiency on supported hardware but requires calibration/robust kernels and may hurt quality more.
Lower-precision KV can increase concurrency/context capacity, but affects decode quality/stability and requires runtime support.
| Representation | Typical role | Trade-off |
|---|---|---|
| FP32 | Training/reference/debug in some pipelines. | High memory/compute cost; rarely necessary for inference. |
| BF16 / FP16 | Common high-quality inference baseline. | Good quality; moderate memory footprint. |
| FP8-like | Supported accelerator inference/training paths. | Good efficiency potential; hardware/runtime dependent. |
| INT8-like | Weights and/or activations. | Often strong quality/memory compromise. |
| INT4 / 4-bit-like | Weight-only inference especially for large models. | Large memory reduction; higher sensitivity to method/grouping/model/task. |
| <4-bit | Research/extreme compression/edge cases. | Quality/kernel complexity grows sharply; validate very carefully. |
FLOAT RANGE: -1.20 ... 0 ... +1.20 QUANTIZED INTEGER RANGE: e.g. INT8 → -128 ... 127 ONE SIMPLE SCHEME: q = round(x / scale) + zero_point x_approx = (q - zero_point) * scale ERROR: e = x - x_approx QUANTIZATION DOES NOT STORE THE ORIGINAL CONTINUOUS VALUE. IT STORES A DISCRETE APPROXIMATION. THE ENTIRE GAME IS: choose representation so approximation error hurts model behavior as little as possible while resource savings are useful.
Usually uses scale around zero with no arbitrary offset. Simpler kernels and common for weights whose distributions are roughly centered.
Can better cover non-symmetric ranges, especially some activations, at additional complexity.
Lowest metadata cost, but outliers can waste most integer range.
Better adapts to differing ranges across output/input channels.
Common for low-bit LLM weight quantization; group size trades metadata/kernel complexity for fidelity.
Used in some activation schemes; adds runtime work but adapts to changing input distribution.
Most values cluster near zero, few outliers force a large scale → small values lose precision.
Outlier channels/layers can remain FP16/BF16 while bulk is quantized.
Calibration algorithms may rescale, clip or rearrange activations/weights to reduce error.
Model weights dominate static memory footprint, especially at low concurrency.
Reduces risk and lets existing compute paths handle dynamic values.
Runtime may dequantize into higher precision or use specialized kernels; actual speed varies by hardware.
Include actual languages, domains, prompt lengths and task types.
Calibration can use a manageable sample if it covers relevant activation distributions.
Store exact calibration dataset/version and preprocessing config.
Calibration data can contain sensitive content; handle under same data policy as evaluation/training assets.
Very low bit-width or sensitive model/task may require training adaptation.
Forward path approximates low-precision effects so weights adapt to them.
Requires data, compute, optimization and much more lifecycle discipline than PTQ.
Keep first/last/output or empirically sensitive modules in BF16/FP16.
Quantize the majority where quality impact is small.
Some operations stay higher precision even in nominal low-precision pipeline.
Mixed formats only help if runtime executes them efficiently.
Different prompts can produce very different activation distributions and outliers.
Use representative calibration or runtime scaling depending on method.
On suitable hardware, lower-precision activation matmuls can increase throughput and reduce bandwidth.
KV footprint scales with context and concurrent sequences.
Lower KV precision may increase concurrency or context capacity on same hardware.
Approximation is used repeatedly during decode, so long-context regressions must be tested carefully.
IDEALIZED WEIGHT STORAGE: parameters × bits_per_weight / 8 Example: 70B parameters FP16 ideal: 70e9 × 16 / 8 ≈ 140 GB INT8 ideal: ≈ 70 GB INT4 ideal: ≈ 35 GB BUT REAL ARTIFACT / VRAM ALSO HAS: scales zero-points metadata non-quantized layers padding/alignment runtime buffers SO: "70B INT4 = exactly 35 GB" is only a rough lower-bound intuition.
Broad score moves little.
Exact reasoning/logit margins may degrade disproportionately.
Small changes can raise JSON/tool-call format failures.
Low precision may behave differently at long sequence lengths.
token A logit = 5.01
token B = 4.99
A = 4.97
B = 5.00
Next context changes → future token distribution diverges.
Useful for low-noise artifact comparison when model/runtime permit deterministic execution.
Base and quantized model receive identical task inputs/context.
When production uses sampling, compare pass rates/distributions rather than single outputs.
| Dimension | Baseline | Candidate | Decision rule |
|---|---|---|---|
| Task pass rate | Full precision | Quantized | Regression ≤ allowed threshold by critical segment. |
| Structured output success | JSON/tool-call pass % | Candidate % | No material increase in contract failures. |
| Long-context quality | Reference tests | Candidate tests | No unacceptable degradation. |
| VRAM | Measured | Measured | Must improve target memory bottleneck. |
| TTFT / TPOT | Same hardware/runtime | Candidate | Gain or no unacceptable regression. |
| Throughput | tokens/sec | tokens/sec | Measured under same SLO. |
| Stability | OOM/error rate | Candidate | No new numerical/runtime instability. |
Measure loaded model and KV/workspace under real concurrency.
Lower precision may or may not speed prompt processing.
Often where memory-bandwidth gains can show up.
Compare under same prompt/output distribution and SLO.
Useful for edge/high-volume TCO.
Smaller artifact can improve model distribution/startup.
Weight savings may allow more active sequences even without faster single request.
Long runs reveal kernel/runtime bugs and memory fragmentation.
Hardware/runtime executes quantized matmul efficiently.
Weights are compressed in memory but expanded for compute, limiting speed gain.
Format/kernel/model architecture mismatch may make candidate unusable.
Bit width, group size, scales, symmetry, outlier handling.
Container/layout/metadata for runtime loading.
Actual accelerator operator path determines performance.
{
"artifact_ref": "modelq://model-x/q17",
"base_model_ref": "model://model-x/r42",
"base_model_hash": "sha256:...",
"quantization": {
"method": "weight_only",
"algorithm": "...",
"weight_bits": 4,
"group_size": 128,
"symmetric": true,
"activation_bits": 16,
"kv_bits": 16
},
"calibration": {
"dataset_hash": "sha256:...",
"sample_count": 512
},
"tool_version": "...",
"artifact_hash": "sha256:...",
"compatible_runtime": ["serve-profile-v8"],
"quality_report_ref": "eval://...",
"benchmark_ref": "bench://...",
"status": "APPROVED"
}First produce final adapted model, then quantize the exact deployment candidate and evaluate it.
Runtime may support quantized base with higher-precision adapters; compatibility must be tested.
If low precision is a hard deployment requirement, training may explicitly optimize for it.
| Technique | What changes | Main goal |
|---|---|---|
| QUANTIZATION | Precision/encoding of parameters/activations. | Memory/bandwidth/compute efficiency. |
| DISTILLATION | Student model trained to imitate teacher. | Smaller/faster learned model. |
| PRUNING | Remove/zero parameters/structures. | Reduce compute/model size if runtime exploits sparsity. |
| FINE-TUNING | Update weights/adapters on task/domain data. | Change behavior/quality. |
Does candidate still produce valid schema-conforming outputs?
Does low precision change function/tool choice on borderline cases?
Names, numbers, IDs and enum values may be more sensitive than broad prose similarity.
Can model still recover exact relevant facts from long context?
Does it preserve relation/order at long lengths?
Quantized KV should be tested at p95/p99 expected context, not just short examples.
Measure OOM, latency and output degeneration as context grows.
Russian, English and other production languages separately.
Legal, medical, code, finance, internal terminology can have different sensitivity.
Exact factual extraction should be evaluated as its own task class.
Low-bit values expanded before compute.
Generic kernels may underperform well-optimized BF16 path.
Some optimizations only win at certain batch/concurrency sizes.
Even slower single request can be acceptable if quantization makes deployment possible or increases concurrency enough.
EXAMPLE POLICY:
critical tasks:
structured output pass rate
regression <= 0.2 percentage points
high-risk factual tasks:
verification pass rate
no statistically meaningful regression
general chat:
preference win/loss
quantized not worse than threshold
code:
unit-test pass rate
regression <= 1%
long-context:
retrieval accuracy
regression <= 0.5%
IF candidate violates a critical segment:
reject
OR use it only for eligible task classes
via №65 routing.
Critical/high-risk or tasks where quantized artifact failed quality threshold.
General production workload where supported hardware gives good throughput.
Local/edge/high-volume helper tasks where evals prove sufficiency.
Quantized candidate points to immutable base model hash.
Bit-width/group/calibration/tool/runtime metadata.
Do not assume quantization settings transfer unchanged to a new model revision.
Task/eval regression vs pinned full-precision baseline.
Loaded/static and active workload memory reduction.
Time-to-first-token change under same workload.
Inter-token/decode speed improvement or regression.
Tokens/sec under same SLO and concurrency.
More active sequences/context or reduced device count.
Kernel/numerical/load/OOM issues by artifact.
Full deployment cost normalized by verified successful tasks.
Artifact loads reliably on every supported node/runtime version.
No kernel crashes, memory leaks or numerical degeneration after sustained traffic.
Test declared context/output limits, especially with KV quantization.
Find actual saturation and OOM-safe limit.
Corrupted quantized file is rejected before serving.
Router returns traffic to previous model revision quickly.
High-volume contract tests reveal small reliability regressions.
Critical production segments separately evaluated.
quantization/ ├── configs/ │ └── q4_candidate.yaml ├── calibrate.py ├── quantize.py ├── package.py ├── eval_quality.py ├── benchmark_serving.py ├── reports/ └── registry.json candidate config: base_model_ref: model://x/r42 method: weight_only weight_bits: 4 group_size: 128 symmetric: true activation_bits: 16 kv_bits: 16 calibration_ref: dataset://qcal/v3 target_runtime: serve-profile-v8 target_hardware: gpu-class-A pipeline: base artifact ↓ quantize ↓ artifact checksum ↓ load on target runtime ↓ quality eval ↓ serving benchmark ↓ compare against baseline ↓ APPROVE / REJECT ↓ canary deployment
If the candidate does not materially improve the target bottleneck at acceptable quality, stop. Do not add QAT/more exotic formats just because they exist.
| Observed need | Next move |
|---|---|
| 4-bit quality too low | Try higher precision, different grouping/method, mixed precision or exclude sensitive task classes. |
| Weights fit but KV dominates | Evaluate KV-cache quantization or more memory/replicas. |
| Memory improves but speed does not | Check runtime kernel support/hardware path; benchmark alternative format. |
| INT8 is almost enough but hardware underutilized | Evaluate supported activation quantization / FP8-like path. |
| PTQ cannot meet hard low-bit target | Consider QAT or distillation only if economics justify training pipeline. |
| Different workloads have different sensitivity | Maintain multiple precision deployments and let №65 route by eligibility. |
| Edge device still cannot fit model | Smaller/distilled model may be better than ever more aggressive quantization. |
| Вопрос | Ответ |
|---|---|
| Стоит ли реализовывать? | Условно. После full-precision baseline, если есть измеренный memory/cost/latency bottleneck. |
| Separate Component? | YES логически. Offline artifact transformation/evaluation pipeline feeding Model Infrastructure. |
| Минимум 80% ценности? | Weight-only PTQ, exact artifact/config metadata, quality eval, target-hardware benchmark, canary and rollback. |
| Когда overkill? | QAT/mixed ultra-low-bit pipeline для модели, которая уже помещается и удовлетворяет SLO/TCO. |
| Trigger? | Model doesn't fit, KV headroom too low, owned hardware cost is high, edge constraints, or low-precision hardware kernels can materially improve serving. |
| Как измерить uplift? | VRAM reduction, capacity increase, TTFT/TPOT/TPS delta, power/cost per successful task and task-specific quality delta. |
| Можно ли rule/tool/code вместо LLM-agent? | Да. Quantization is deterministic artifact tooling + eval pipeline. LLM-agent is not needed to choose scales at runtime. |
Know full-precision quality and serving profile before optimizing.
Memory, KV capacity, TPOT, power or TCO — not «lower bits» as a goal.
Method, group size, scales, runtime, hardware and calibration define behavior.
Tool calls, JSON, code, long context and domain language may regress unevenly.
Theoretical compression does not guarantee speedup.
Use the cheapest reversible optimization first.
Sensitive layers/tasks can stay higher precision.
Version, checksum, eval, benchmark, canary and rollback.
Different precision deployments can serve different task classes.
FULL-PRECISION BASELINE
exact model revision
exact tokenizer
task evals
serving benchmark
↓
IDENTIFY BOTTLENECK
model does not fit?
VRAM leaves no KV room?
decode is bandwidth-limited?
edge power/storage too high?
TCO too high?
↓
SELECT QUANTIZATION TARGET
weights only
weights + activations
KV cache
mixed precision
↓
SELECT METHOD / FORMAT
bit-width
group size
scale scheme
symmetry
outlier handling
calibration
target runtime
target hardware
↓
CREATE IMMUTABLE CANDIDATE
base model hash
quantizer version
calibration hash
artifact checksum
↓
LOAD ON TARGET HARDWARE
↓
QUALITY EVAL
critical tasks
structured outputs
tools
code/math
languages/domain
long context
↓
SERVING BENCHMARK
artifact size
loaded VRAM
TTFT
TPOT / ITL
TPS
concurrency
power
stability
↓
COMPARE WITH BASELINE
↓
IF:
quality within budget
AND target bottleneck improves
AND runtime stable
↓
CANARY
↓
PROMOTE
↓
REGISTER DISTINCT DEPLOYMENT IN №65
ELSE:
reject
try higher precision /
different method /
mixed precision /
smaller model /
more hardware
↓
DO NOT FORCE QUANTIZATION
BOUNDARIES:
№39
changes model behavior through learning
№70
serves artifact and measures runtime performance
№71
decides where/how to self-host
№72
changes numeric precision of deployment artifact
№65
routes tasks to approved deployment profiles
CORE PRINCIPLE:
QUANTIZATION IS NOT
"MAKE THE MODEL 4-BIT."
IT IS:
"CAN WE REPRESENT
THIS MODEL WITH
LESS NUMERIC PRECISION
ON THIS HARDWARE
FOR THIS WORKLOAD
WITHOUT LOSING
THE QUALITY WE ACTUALLY NEED?"
THE WIN IS NOT
THE SMALLEST FILE.
THE WIN IS
LOWER REAL SYSTEM COST
AT THE SAME
ACCEPTABLE TASK QUALITY.
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №72 Quantization.
B–E. Existing boundary and placement. The existing conceptual boundary, class SPECIALIZED, default CONDITIONAL and owner Model Infrastructure remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.