AI Metrics Framework — единая система измерения AI-продукта от реального использования до бизнес-эффекта. Она связывает Usage → Quality → Economics → Autonomy → Delivery и не позволяет выдавать рост одной красивой цифры за успех всей системы.
№47 Evals measures controlled task quality; №46 Observability records runtime behavior; №84 turns these and product/business events into an integrated management metric system. №85–93 own individual metrics. №94 owns dashboard composition. №92 Ablation Delta owns causal uplift of architecture mechanisms.
Strong dependencies: №40 Eval-Driven Optimization, №46 Observability, №47 Evals, №50 Contracts, №61 Provenance, №64 Budgets, №77 Production Architecture. Metrics should resolve to stable task/run/use-case identities and quality outcomes.
CONTROL: metric definitions, denominators, eligibility rules, cohorts, thresholds, targets. DATA: user/task/run/outcome/cost/review/deployment events. RUNTIME: instrumentation only—do not put analytical aggregation on critical path. OFFLINE: cohorting, aggregation, causal analysis, executive reporting, eval comparison.
Success: every KPI has definition, numerator, denominator, eligibility, time window, segmentation, source and owner. Failure: changing denominator, mixing attempts with successful operations, counting waiting time as saved human time, averaging incompatible use cases, or optimizing one KPI while guardrail quality collapses.
№84 does not decide whether an answer is correct, does not implement tracing and does not prove causality from correlation. It owns metric architecture: what should be measured together, how metrics relate, and how to prevent misleading conclusions.
Adoption Rate, active users, habitual usage, eligible population, cohort retention.
If nobody uses the capability, downstream value cannot materialize.
Acceptance Rate, task eval success, edit depth, verifier outcomes, user correction.
Usage without quality can be novelty or forced adoption.
Time Savings, Cost per Operation, human review cost, failed-run cost, throughput value.
Quality alone does not guarantee positive ROI.
Autonomy Rate, Escalation Rate, approval dependence, eligible risk classes.
Autonomy must be quality- and risk-adjusted.
AI Velocity, experiment throughput, eval lead time, release cadence, ablation evidence.
A good system also improves economically over time.
Useful completed work, verified value, time/cost saved, business throughput.
Adoption, coverage, activation, weekly usage, eligible task penetration.
Quality, safety, privacy, latency, failure/error rates, human override.
Escalation reasons, retry rate, model route, use case, cohort, source class, reviewer load.
| Layer | Leading | Lagging |
|---|---|---|
| Usage | Activation, WAU, task starts | Habitual adoption / retained usage |
| Quality | Verifier pass, edit prediction | Acceptance as-is, rework, business outcome |
| Economics | Model cost/run, active human minutes | Cost per successful operation, realized time savings |
| Autonomy | Eligible automation decisions | Autonomous successful completions |
| Delivery | Experiments started | Idea → production time, measurable uplift shipped |
Доля eligible users, реально использующих AI в заданный период.
Доля eligible task volume / task types / process steps, где AI реально участвует.
Экономия именно активного человеческого времени, с учётом review/rework.
Как часто AI-результат принимают as-is, с minor edit, major rewrite или reject.
Доля AI-задач, которые требуют передачи человеку; важно разделять policy vs failure escalation.
Models + tools + retrieval + infra + retries + failures + human review per successful completion.
Доля eligible задач, завершённых без обязательного человеческого вмешательства.
Прирост от конкретной надстройки относительно системы без неё.
Время от одобренной AI-идеи/гипотезы до production with measurable outcome.
What metrics mean, how denominators are designed, what dimensions/guardrails are required, how metrics should be read together.
Which cards/charts/alerts/slices are shown to management, product, ops and engineering; diagnostic drill-down and composite views.
BAD: autonomy_rate = autonomous_tasks / all_tasks Problem: many tasks may be legally, technically or intentionally ineligible for autonomy. BETTER: autonomy_rate = successful autonomous completions / autonomy-eligible tasks AND record: total tasks eligible tasks attempted autonomous tasks successful autonomous tasks policy-blocked tasks human-required tasks
Users with access + relevant role/use case, not entire company headcount.
Tasks for which AI is allowed and technically capable, not every job in the organization.
Only tasks permitted to run without mandatory approval.
Comparable output types with a meaningful “accepted as-is” state.
Human process baseline under comparable workload.
AI vs non-AI cohort where allocation is fair enough for comparison.
Same system with one mechanism removed/disabled.
New users, power users, team, region, role.
Writing, research, coding, classification, support, actions.
Autonomy and escalation rates must not be averaged blindly across risk classes.
Pre/post version, model change, onboarding cohort, release week.
| Window | Good for | Risk |
|---|---|---|
| Per-run / per-task | Quality, cost, latency, escalation reason | Too noisy for product adoption/value |
| Daily | Operational anomaly, launch monitoring | Day-of-week / low volume volatility |
| Weekly | WAU adoption, coverage, stable product behavior | Can hide short incidents |
| Monthly | Business/management trend, cohorts | Slow feedback loop |
| Rolling window | Smoothing + ongoing alerts | May hide release boundary unless annotated |
task_type: summarize_document research_question draft_marketing_post classify_ticket create_code_patch approve_refund browser_action ... dimensions: risk_class complexity expected_human_baseline autonomy_eligible review_required business_value_weight
Who has actually adopted the system?
Was the real unit of work completed successfully?
How many attempts/retries/model calls occurred to complete one task?
ONE TASK
may contain:
1 workflow instance
4 model calls
3 tool calls
2 retries
1 human review
If cost denominator = model calls,
you measure inference economics.
If denominator = successful task,
you measure product economics.
These are different metrics.
Representative benchmark/eval pass rate.
Accepted as-is/minor/major/reject distribution.
Critical material error should outweigh cosmetic edit.
High-stakes autonomy cannot use same acceptable-error threshold as low-stakes drafting.
| Observed pattern | Likely interpretation |
|---|---|
| Adoption ↑, Acceptance ↓ | Usage is growing faster than usefulness; forced rollout / novelty / quality regression possible. |
| Coverage ↑, Escalation ↑ | AI is touching more work but may be exceeding capability/risk boundary. |
| Autonomy ↑, Quality ↓ | Unsafe automation expansion. |
| Acceptance ↑, Cost/Operation ↑↑ | Quality improved but unit economics may be unsustainable. |
| Time Savings ↑, Human Review ↑ | Check whether “saved” time excludes review overhead incorrectly. |
| AI Velocity ↑, Ablation Delta ≈ 0 | Team ships faster but changes may not create measurable value. |
| Adoption stable, Coverage ↑ | Existing users are delegating more task volume / use cases. |
Raw coverage: 100 AI-handled tasks / 200 eligible tasks = 50% But: 90 AI-handled tasks are trivial 10 are high-value Weighted coverage: sum(task_weight × AI_handled) -------------------------------- sum(task_weight × eligible) Weights may reflect: baseline human minutes business value frequency strategic importance CAUTION: weights must be governed, versioned and transparent.
cost_per_successful_operation = model inference + embedding / rerank + web / external tools + browser / compute + storage / vector / DB + queue / workflow infra + retries + failed runs + human review + manual recovery + amortized platform cost (if desired) -------------------------------------- successful completed operations
Typing, reading, searching, calculating, editing, coordination.
Prompting, reviewing, correcting, approval, recovery.
AI processing wait can affect cycle time, but is not necessarily human labor.
time_savings = baseline_human_active_minutes − post_AI_human_active_minutes net_value_time = time_savings − additional coordination/rework − manual recovery overhead
AI suggests; human does action.
AI produces output; human must review/commit.
Low-risk auto; higher-risk approval/escalation.
Eligible operation completes without mandatory human.
System acts, observes outcome, recovers within bounded policy.
High-risk task is designed to require human approval.
Missing material information; human/user input required.
System cannot solve reliably with available models/tools.
Tool error, provider failure, invalid output, retry exhausted.
FULL SYSTEM: RAG + reranker + critic + strong-model escalation ABLATION: same tasks same period/model conditions remove critic compare: acceptance eval pass latency cost/operation escalation human review Ablation Delta: full − ablated If quality delta ≈ 0 but cost/latency ↑, critic may not justify itself.
Where feasible and ethical.
Different rollout waves provide concurrent reference.
Needs seasonality/workload/composition controls.
Disable one mechanism under controlled eval/test conditions.
Rate without denominator count is easy to misread.
Show confidence/credible interval when rates drive decisions.
Do not rank teams/use cases from tiny n without warning.
output_status: accepted_as_is minor_edit major_edit rejected not_reviewed unknown DO NOT: acceptance = accepted / (accepted + rejected) if 60% of outputs were never reviewed. Track: review_coverage = reviewed_outputs / eligible_outputs
| Field | Example |
|---|---|
| Metric ID | acceptance_rate_v2 |
| Business definition | Share of reviewed eligible outputs accepted without edits. |
| Numerator | accepted_as_is outputs |
| Denominator | reviewed eligible outputs |
| Eligibility | task_type in supported generation tasks |
| Window | weekly rolling 28d |
| Required dimensions | use_case, cohort, model_route, risk_class |
| Data owner | AI Analytics |
| Business owner | Product Lead |
| Version | v2, effective 2026-09-01 |
{
"event_id":"EVT-...",
"event_type":"task_completed",
"occurred_at":"...",
"tenant_id":"...",
"user_id":"...",
"task_id":"TASK-...",
"run_id":"RUN-...",
"use_case":"research_question",
"risk_class":"medium",
"eligible":{
"ai":true,
"autonomy":true
},
"outcome":{
"success":true,
"acceptance":"minor_edit",
"escalated":false,
"autonomous":true
},
"human":{
"active_seconds":82,
"review_seconds":55
},
"cost":{
"model":0.18,
"tools":0.04,
"infra":0.02,
"human_estimated":1.10
},
"versions":{
"product":"2026.09.12",
"model_route":"router-v8",
"workflow":"research-v4"
}
}
metric_task( task_id tenant_id user_id use_case risk_class started_at completed_at successful ai_eligible autonomy_eligible baseline_class product_version ) metric_run( run_id task_id attempt_no mode model_route cost latency outcome ) metric_review( review_id task_id reviewer_type acceptance_class edit_seconds reason_code ) metric_human_time( task_id phase active_seconds measurement_method ) metric_deployment( version deployed_at cohort experiment_id )
Stable end-to-end unit of work.
Links observability spans, retries and model/tool calls.
Links metric outcome to A/B or ablation treatment.
Product/workflow/model/policy version for cohort analysis.
Run/span/tool/model timing, failures, route, retries.
Task outcome, human time, review status, costs, eligibility, cohort.
Adoption, coverage, acceptance, cost, autonomy, velocity + guardrails.
Detect regressions, compare models/prompts/architecture under controlled task sets.
Acceptance, escalation, rework, business success and cost under actual users/tasks.
Start, completion, review, cost and human time coverage.
Idempotent event ingestion / stable event_id.
Rejected output cannot simultaneously be accepted_as_is.
Operational monitoring and management reporting have different SLA.
Prefer task/use-case attributes over raw prompt/output copies.
Aggregate management metrics need not expose user content.
Individual-level slices only when justified and authorized.
Align analytics retention with №76 governance/privacy policy.
| Audience | Primary view | Examples |
|---|---|---|
| Executive / Management | Value + trend + guardrails | Adoption, time saved, cost/success, autonomy, quality, velocity. |
| Product | Use-case cohorts | Activation, habitual adoption, coverage, acceptance, task taxonomy. |
| AI / Engineering | Mechanism and quality | Evals, ablation, model route, retries, latency, tool failures. |
| Operations | Flow / exceptions | Escalation reason, queue age, approval SLA, failed operations. |
| Governance / Risk | Guardrails | Risk-class autonomy, policy escalations, human override, incidents. |
METRIC CARD Name: Autonomy Rate Purpose: measure successful autonomous completion among tasks permitted to be autonomous. Numerator: successful autonomous completions Denominator: autonomy-eligible tasks Exclusions: mandatory-HITL risk classes test traffic duplicate/replayed tasks Window: weekly, rolling 28d Required slices: use_case risk_class cohort product_version Guardrails: acceptance_rate eval_pass_rate severe_error_rate Data sources: task_completed approval_event policy_decision Owner: Product + AI Analytics Version: metric_v1
Measure enough representative history to know normal variability.
Business/product goal, not merely current +10%.
Quality/safety/cost/latency boundary.
Statistically/operationally meaningful deviation requiring action.
2026-08:
adoption_rate_v1 =
weekly active users / licensed users
2026-09:
adoption_rate_v2 =
weekly active eligible users
/ eligible users
Do NOT splice v1 and v2
into one continuous line
without annotation.
Store:
metric_id
version
effective_from
definition
denominator_rule
exclusions
CHANGE: enable reranker v3 PRIMARY: acceptance_rate +3pp GUARDRAILS: cost_per_operation ≤ +8% p95 latency ≤ +120ms severe_error_rate no increase DIAGNOSTICS: retrieval recall evidence coverage model route task complexity COHORT: research_question English + Russian medium complexity BASELINE: v2 reranker / control cohort DECISION: ship if primary improves and guardrails remain within bounds.
% executive/product KPIs with formal metric cards.
% eligible outputs with valid review/outcome labels.
% successful tasks joinable to runs, costs and reviews.
% tasks/users with explicit eligibility classification.
% claimed-value metrics with valid baseline/control.
Changes in workload mix tracked and visible.
Metric data lag relative to reporting SLA.
% expensive architecture mechanisms with measured uplift evidence.
MVP EVENTS 1. user_activated 2. task_started 3. task_completed 4. output_reviewed 5. human_escalated 6. external_action_committed 7. deployment_changed MVP METRICS 1. Adoption Rate 2. Coverage 3. Acceptance Rate 4. Human Active Time Savings 5. Cost per Successful Operation 6. Autonomy / Escalation pair MVP SLICES 1. use_case 2. risk_class 3. team / cohort 4. product_version 5. model_route MVP RULE: no metric is shown without N, eligibility rule, period and definition version.
| Question | Required |
|---|---|
| What decision does this metric support? | Yes |
| What is the exact numerator? | Yes |
| What is the exact denominator? | Yes |
| What is eligible / excluded? | Yes |
| What is the unit: user, task, operation, run? | Yes |
| What is the period? | Yes |
| What slices are mandatory? | Yes |
| What is the baseline/control? | For value/uplift claims |
| What quality/risk guardrails pair with it? | For adoption/economics/autonomy |
| What is sample size / label coverage? | Yes |
| Who owns the definition? | Yes |
| What version is this? | Yes |
You are designing an AI Metrics Framework.
Do not start from dashboard charts.
1. Define the business/product outcome.
2. Define the unit:
user, task, operation, run, workflow or experiment.
3. Build the value chain:
Usage → Quality → Economics → Autonomy → Delivery.
4. For every proposed metric specify:
- purpose;
- numerator;
- denominator;
- eligibility;
- exclusions;
- time window;
- required cohort dimensions;
- data source;
- owner;
- definition version.
5. Distinguish:
leading,
lagging,
guardrail,
diagnostic metrics.
6. Never mix:
attempts with successful operations,
elapsed time with human active time,
total tasks with autonomy-eligible tasks.
7. Require baseline/control for claims of improvement.
8. Pair autonomy/economics metrics with quality/risk guardrails.
9. Track missing-label coverage explicitly.
10. If architecture mechanism is expensive,
require ablation evidence.
11. Show N/sample size for rates.
12. Flag correlation as correlation unless causal design supports more.
13. Prefer a small trustworthy KPI set
over dozens of ungoverned metrics.
| Question | Answer |
|---|---|
| Separate component? | N/A for framework. Instrumentation pipeline, warehouse and dashboard are implementation components. |
| Minimum 80% value? | Task identity + eligibility + adoption + coverage + acceptance + human time + cost/success + autonomy/escalation + version/cohort slices. |
| When overkill? | Tiny prototype before product use exists; still keep basic eval/cost instrumentation. |
| Trigger? | “Is this AI actually useful?”, “Did this architecture improvement pay?”, “Can we safely increase autonomy?”, “What does adoption really mean?”. |
| How to prove uplift? | Baseline/control/ablation + quality guardrails + stable denominators + representative cohorts. |
| Can rules/code replace LLM? | Yes. Metric computation should be deterministic SQL/code. LLM may explain anomalies, suggest slices or draft metric definitions, never calculate authoritative KPIs from prose. |
Usage alone is not value.
Never publish a rate without eligibility definition.
Runs/model calls are diagnostics and costs, not business completion.
No causal savings claim from current-state observation alone.
Global averages hide important failures.
Autonomy, time savings and cost are meaningless if output quality collapses.
Cost/success includes retries and failures.
A changed denominator is a changed metric.
Every expensive architecture layer must earn its cost with measurable delta.
AI SYSTEM ↓ INSTRUMENT TASKS user_id task_id run_id use_case risk_class eligibility versions ↓ MEASURE USAGE ADOPTION RATE are eligible users active? COVERAGE how much eligible work is AI touching? ↓ MEASURE QUALITY ACCEPTANCE accepted as-is? minor edit? major rewrite? reject? EVALS controlled benchmark quality SEVERE ERROR / GUARDRAILS ↓ MEASURE ECONOMICS TIME SAVINGS human active before minus human active after COST / SUCCESS models tools infra retries failures human review divided by successful operations ↓ MEASURE AUTONOMY AUTONOMY RATE successful autonomous completions / autonomy-eligible tasks ESCALATION RATE human escalations / eligible AI tasks split: policy ambiguity capability failure ↓ MEASURE ARCHITECTURE VALUE ABLATION DELTA full system minus system without mechanism ↓ MEASURE DELIVERY AI VELOCITY approved idea → experiment → eval → production → measurable outcome ═══════════════════════════════════ READ TOGETHER: ADOPTION ↑ but ACCEPTANCE ↓ = usage ≠ value AUTONOMY ↑ but QUALITY ↓ = unsafe automation COST ↓ but REWORK ↑ = local savings, system loss VELOCITY ↑ but ABLATION DELTA ≈ 0 = shipping activity, little value ═══════════════════════════════════ EVERY KPI MUST HAVE: PURPOSE NUMERATOR DENOMINATOR ELIGIBILITY EXCLUSIONS UNIT PERIOD COHORTS BASELINE GUARDRAILS DATA SOURCE OWNER VERSION SAMPLE SIZE ═══════════════════════════════════ THE CENTRAL QUESTION IS NOT: “HOW MUCH AI ARE WE USING?” IT IS: “HOW MUCH ELIGIBLE WORK IS BEING COMPLETED BETTER, FASTER, CHEAPER AND SAFELY ENOUGH BECAUSE OF AI — AND WHICH PART OF THE ARCHITECTURE ACTUALLY CREATED THAT DELTA?”
A. Related ECC ideas. Context-as-cache, scoped memory, lifecycle hooks, selective capabilities, feature flags, deterministic enforcement, provider-neutral adapters and eval-gated learning are applied only where relevant to №84 AI Metrics Framework.
B–E. Existing boundary and placement. The existing conceptual boundary, class METRIC, default N/A and owner Evals / Analytics / Management remain authoritative. Runtime/control/data/offline placement is unchanged; durable state stays outside model context.
F–H. Hooks and contracts. Use bounded PRE_MODEL/POST_MODEL, PRE_TOOL/POST_TOOL, CHECKPOINT and TASK_COMPLETED events as applicable. Illustrative fields and canonical contracts are defined in NEW_CONTRACTS_SPEC.md; no universal schema is implied.
I–J. Security and evaluation. Host-side schema, permission, secret, budget, idempotency and audit checks take precedence over LLM output. Optional mechanisms require a feature flag and WITH/WITHOUT ablation; measure quality, acceptance, correction, latency, cost, escalations and severe errors.
K–L. Task profiles and cross-references. A TaskProfile selects the relevant skill, tool/context slice, memory scope and enforcement profile independently from FAST/STANDARD/DEEP. See cross-reference map, hook spec and ablation plan. Provider adapters remain outside the core.