ACCEPTANCE
RATE.

Strict Acceptance Rate =

accepted_as_is
-----------------------
reviewed eligible outputs
× 100%

Broad Acceptance Rate =

accepted_as_is + minor_edit
---------------------------
reviewed eligible outputs
× 100%

Major Edit Rate =

major_edit
----------
reviewed eligible outputs

Reject Rate =

rejected
--------
reviewed eligible outputs

Review Coverage =

reviewed eligible outputs
-------------------------
all eligible AI outputs
× 100%

Example:

1,000 AI outputs
600 reviewed
300 as-is
180 minor
80 major
40 rejected
400 unreviewed

Strict Acceptance among reviewed:
  300 / 600 = 50%

Broad Acceptance among reviewed:
  (300 + 180) / 600 = 80%

But overall review coverage:
  60%

You cannot claim:
  “80% of all outputs are accepted.”

Correct:
  “80% broad acceptance
   among reviewed outputs,
   with 60% review coverage.”

Example text heuristic:

minor if:
  semantic structure preserved
  AND material facts unchanged
  AND edit_time < threshold
  AND edit_ratio < threshold

major if:
  core argument/structure changed
  OR factual/material correction
  OR substantial rewrite
  OR reviewer says output unusable without rework

Final policy should be
use-case specific and versioned.

Possible text edit ratio:

1 − similarity(ai_output, final_output)

or:

changed_tokens
--------------
max(ai_tokens, final_tokens)

Useful:
  compare cohorts
  detect rewrite burden

Not sufficient:
  one changed number may be critical
  while 30 formatting edits may be trivial.

Example utility score:

as_is       = 1.00
minor_edit  = 0.75
major_edit  = 0.25
rejected    = 0.00

Weighted Acceptance Score =

Σ(class_weight × count)
-----------------------
reviewed eligible outputs

CAUTION:
  weights are management assumptions,
  not universal truth.

Always retain raw class distribution.

Example:

marketing draft:
  broad acceptance 91%

research synthesis:
  78%

code patch:
  66%

contract analysis:
  54%

customer-facing action:
  97% reviewed + mandatory approval

Global:
  76%

The global number hides
very different error tolerance,
workflow and review semantics.

Possible record:

reviewer_1:
  minor_edit

reviewer_2:
  major_edit

Do not silently:
  take majority
  without policy.

Track:
  disagreement_rate
  adjudicated_label
  reviewer_role
  rubric_version

High disagreement may mean:
  rubric unclear
  output type subjective
  reviewer calibration weak

reason_code examples:

style
tone
format
brevity
missing_detail
factual_correction
source_update
logic_correction
policy_correction
user_preference
brand_language
code_bug
test_failure
schema_fix

Use case A:
  Acceptance = 92%
  Coverage = 5%

Use case B:
  Acceptance = 80%
  Coverage = 70%

Possible interpretation:

A:
  excellent on narrow/easy slice

B:
  lower acceptance but much broader
  real workload exposure

Need:
  complexity/risk-adjusted analysis
  before declaring A “better”.

Model A:
  AI cost = $0.20
  broad acceptance = 70%
  review/edit = 12 min

Model B:
  AI cost = $0.80
  broad acceptance = 90%
  review/edit = 5 min

If human minute cost is high,
Model B may have lower
total Cost per Successful Operation.

Acceptance is one bridge
between model quality
and human economics.

Two useful rates:

Broad Acceptance among reviewed AI outputs =
  (as_is + minor)
  / reviewed AI outputs

Accepted Coverage =
  accepted AI outputs
  / all eligible tasks

Why both?

Acceptance:
  output quality conditional on AI reaching review

Accepted Coverage:
  total workflow value across eligible work,
  including attempts/failures/no-AI paths.

Attempt 1:
  verifier FAIL

AI revises

Attempt 2:
  verifier PASS

Human:
  accepted as-is

Acceptance class:
  AS-IS for final reviewed output

But diagnostics retain:
  first_pass_acceptance = false
  self_repair_count = 1
  final_acceptance = as_is
  added cost/latency = tracked

Sample:
  500 user-accepted outputs

Expert audit:
  440 valid
  40 material issues
  20 severe issues

User acceptance:
  100% for sample

Expert-valid rate:
  88%

Severe miss rate:
  4%

Conclusion:
  high acceptance ≠ safe quality.

Use:
  periodic expert audit
  by risk/use case
  especially before autonomy expansion.

Production:
  50,000 autonomous outputs/month

No human review:
  46,000

Sample:
  1,000 random outputs
  stratified by risk/use case

Expert review:
  as-is 720
  minor 180
  major 80
  reject 20

Estimated broad acceptance:
  90%

with confidence interval
and sample design recorded.

This is not identical
to natural user acceptance,
but provides a quality estimate
when human-in-loop labels vanish.

Use case: research_synthesis
Reviewed N = 2,400
Review coverage = 82%

AS-IS:
  46%

MINOR:
  32%

MAJOR:
  15%

REJECT:
  7%

Broad Acceptance:
  78%

Median review time:
  4.8 min

Median edit time:
  2.1 min

Top reject reason:
  missing evidence

Control:
  RAG + model

Treatment:
  RAG + model + reranker

Same eval / production cohort.

Control:
  broad acceptance 76%
  cost $0.35
  review 7.2 min

Treatment:
  broad acceptance 82%
  cost $0.43
  review 5.8 min

Ablation Delta:
  +6pp acceptance
  +$0.08 AI cost
  −1.4 human review min

Decision:
  evaluate net economics
  and confidence.

{
  "review_id":"REV-...",
  "task_id":"TASK-...",
  "run_id":"RUN-...",
  "output_ref":"artifact://.../output-v3",
  "reviewer":{
    "type":"end_user",
    "role":"marketing_manager"
  },
  "review":{
    "status":"reviewed",
    "acceptance_class":"minor_edit",
    "label_source":"explicit",
    "reason_codes":["tone","brevity"],
    "review_seconds":95,
    "edit_seconds":130,
    "edit_ratio":0.08
  },
  "task":{
    "use_case":"draft_marketing_post",
    "risk_class":"low",
    "complexity":"medium"
  },
  "versions":{
    "product":"2026.09.1",
    "prompt":"post-v7",
    "model_route":"router-v8",
    "rubric":"acceptance-v2"
  },
  "occurred_at":"..."
}

acceptance_review(
  review_id
  task_id
  run_id
  output_ref
  occurred_at

  reviewer_type
  reviewer_role

  review_status
  acceptance_class
  label_source

  reason_codes[]
  review_seconds
  edit_seconds
  edit_ratio

  use_case
  complexity
  risk_class

  product_version
  model_route
  rubric_version
)

WITH reviewed AS (
  SELECT *
  FROM acceptance_review
  WHERE review_status = 'reviewed'
    AND occurred_at >= :start
    AND occurred_at < :end
)
SELECT
  COUNT(*) FILTER (
    WHERE acceptance_class = 'as_is'
  )::float / NULLIF(COUNT(*),0)
  AS strict_acceptance,

  COUNT(*) FILTER (
    WHERE acceptance_class IN ('as_is','minor_edit')
  )::float / NULLIF(COUNT(*),0)
  AS broad_acceptance,

  COUNT(*) FILTER (
    WHERE acceptance_class = 'major_edit'
  )::float / NULLIF(COUNT(*),0)
  AS major_edit_rate,

  COUNT(*) FILTER (
    WHERE acceptance_class = 'rejected'
  )::float / NULLIF(COUNT(*),0)
  AS reject_rate
FROM reviewed;

ACCEPTANCE ↓
  ↓
Review coverage changed?
  yes → missingness / reviewer behavior
  no
  ↓
Reviewer mix changed?
  yes → stratify/calibrate
  no
  ↓
Task mix / complexity changed?
  yes → compare matched cohorts
  no
  ↓
Product/model/prompt version changed?
  yes → ablation / rollback / eval
  no
  ↓
Reject reason:
  factual → RAG/model/verification
  missing detail → prompt/context
  stale → ingestion/provenance
  style → prompt/UX/personalization
  schema → structured output contract
  tool failure → Tool/Production Fabric
  policy → guardrails/permissions
  ↓
Check:
  first-pass vs final acceptance
  review/edit time
  cost
  severe error rate

DATA

1. task_id
2. output_ref/version
3. use_case
4. review_status
5. acceptance_class:
   as_is
   minor_edit
   major_edit
   rejected
6. label_source
7. review/edit seconds
8. reason_code
9. model/product version
10. risk / complexity

MVP METRICS

1. Strict Acceptance Rate
2. Broad Acceptance Rate
3. Major Edit Rate
4. Reject Rate
5. Review Coverage
6. median review time
7. median edit time
8. acceptance by use case
9. acceptance by complexity
10. top rejection/edit reasons

GUARDRAILS

1. offline eval quality
2. severe error rate
3. Time Savings
4. Coverage
5. Cost per Successful Operation

You are designing Acceptance Rate analytics for an AI product.

1. Define eligible AI outputs.
2. Define review states:
   reviewed,
   not_reviewed,
   unknown.
3. For reviewed outputs classify:
   accepted_as_is,
   minor_edit,
   major_edit,
   rejected.
4. Define minor vs major edit rules
   per output type/use case.
5. Never treat missing review as accepted.
6. Track Review Coverage separately.
7. Compute:
   Strict Acceptance,
   Broad Acceptance,
   Major Edit Rate,
   Reject Rate.
8. Store:
   exact output version/ref,
   reviewer type/role,
   label source,
   reason codes,
   review/edit time,
   rubric version.
9. Separate:
   first-pass acceptance
   from final acceptance after AI self-repair/retries.
10. Segment by:
    use case,
    risk,
    complexity,
    model route,
    RAG mode,
    product version,
    reviewer cohort.
11. Pair with:
    Evals / Verification,
    Time Savings,
    Coverage,
    Cost per Operation.
12. Use random expert audit samples
    when human review coverage is low or autonomy is high.
13. Official rates should be calculated deterministically
    in SQL/code.
14. LLM may classify edit/rejection reasons
    only if validated and auditable.

AI OUTPUT
  ↓
WAS IT REVIEWED?

NO
  → NOT REVIEWED / UNKNOWN
  → contributes to Review Coverage denominator
  → does NOT count as accepted

YES
  ↓
WHAT DID HUMAN DO?

USED WITHOUT MATERIAL CHANGE
  → ACCEPTED AS-IS

SMALL NON-MATERIAL CHANGES
  → MINOR EDIT

SUBSTANTIAL REWRITE / MATERIAL FIX
  → MAJOR EDIT

DISCARDED / REPLACED / REDONE
  → REJECTED

═══════════════════════════════════

PRIMARY METRICS:

STRICT ACCEPTANCE
  as-is / reviewed

BROAD ACCEPTANCE
  (as-is + minor) / reviewed

MAJOR EDIT RATE

REJECT RATE

REVIEW COVERAGE
  reviewed / eligible outputs

═══════════════════════════════════

ALWAYS CONNECT TO:

EVALS / VERIFICATION
  was output actually correct?

TIME SAVINGS
  how much human work remained?

COVERAGE
  how much real work reaches this quality?

COST / SUCCESS
  is higher acceptance economically worth it?

AUTONOMY
  can humans safely leave the loop?

═══════════════════════════════════

DO NOT CONFUSE:

ACCEPTED
with TRUE

UNREVIEWED
with ACCEPTED

FINAL ACCEPTANCE
with FIRST-PASS QUALITY

HIGH ACCEPTANCE
with HIGH COVERAGE

HIGH ACCEPTANCE
with LOW COST

═══════════════════════════════════

THE CENTRAL QUESTION IS NOT:

“DID THE MODEL PRODUCE AN OUTPUT?”

IT IS:

“WHEN A REAL HUMAN
HAD TO USE THIS OUTPUT
IN REAL WORK,

DID THEY TAKE IT AS-IS,
MAKE A SMALL FIX,
REBUILD A LARGE PART,
OR THROW IT AWAY?”

THAT DISTRIBUTION
IS THE REAL
ACCEPTANCE SIGNAL.