Classify and triage AI answer issues

The short answer

Preserve the complete answer before diagnosing it. Then identify one disputed claim, verify the current evidence, assign the narrowest supported issue type, assess severity from likely consequence rather than tone, and route the case to the owner who can address the source or reporting path.

Issue type and severity are separate. A stale price can be high severity when it changes a purchase decision, while an obviously false but immaterial detail may be low severity. “Hallucination” is not a sufficient case classification because it does not identify what failed or what action follows.

Use one answer-capture log per observed run and one case-intake record per investigated issue.

Preserve evidence before triage

Save the exact prompt, relevant earlier turns, complete answer, visible citations, linked destinations, product and mode, date and time, market, language, and available account or personalization conditions. Preserve screenshots when presentation affects meaning, but keep selectable answer text and URLs as well.

Do not replace the original observation when repeating the prompt. A changed answer does not erase the reported one, and a repeated answer does not prove that every user receives it.

Classification decision tree

Questionable answer
├─ Can the exact output and context be reviewed?
│  ├─ No → Insufficient evidence. Request a complete capture; do not diagnose.
│  └─ Yes
├─ Does the concern identify a checkable material claim or source relationship?
│  ├─ No
│  │  ├─ Opinion, preference, criticism, or ranking → Not a factual correction case.
│  │  └─ Meaning remains unclear → Unverifiable; record what evidence is missing.
│  └─ Yes
├─ Are distinct people, organizations, products, or locations combined?
│  └─ Yes → Wrong entity or ambiguous identity.
├─ Does reliable current evidence contradict the claim?
│  ├─ Yes, no established formerly correct state → False; record history as unknown if needed.
│  ├─ Yes, it was formerly correct → Stale.
│  └─ Not directly
├─ Does an omission or qualification materially change the meaning?
│  ├─ Missing necessary fact or condition → Incomplete.
│  └─ Correct details create a materially wrong impression → Misleading.
├─ Is the primary failure in a displayed source relationship?
│  ├─ Destination fails → Broken citation.
│  ├─ Source is about another subject → Irrelevant citation.
│  ├─ Source does not establish the claim → Unsupported citation.
│  ├─ Answer changes the source's meaning → Misquotation or misrepresentation.
│  └─ Named source cannot be found or verified → Apparently fabricated reference.
├─ Is there evidence of deceptive identity or content?
│  └─ Yes → Suspected impersonation or manipulation; specialist escalation.
└─ Evidence remains insufficient or conflicting → Unverifiable or expert dispute.

Assign one primary class that determines the next action. When the answer substitutes or combines the wrong subject, use wrong entity as primary; record resulting factual or citation failures as secondary where useful. Add secondary classes only when they describe distinct failures, such as a stale claim linked to an unsupported citation.

Apply the labels consistently

Avoid classifying an entire answer from one sentence. Quote the smallest passage that contains the issue, then state the corrected fact and evidence.

Determine severity independently

Assess the most credible likely consequence if a relevant person relies on the answer. Consider audience, reach where evidence exists, decision impact, subject sensitivity, persistence, reversibility, and evidence strength. Do not infer broad reach from one observation.

Escalate uncertainty itself when the possible harm is severe. A credible but not yet proven safety or fraud report may require urgent specialist review without prematurely labeling the factual cause as confirmed.

Same issue type, different severity

Hypothetical examples. A museum answer repeats a former mailing-box number in a historical description. The current visitor address is correct and no present decision depends on that detail: stale, low severity. An answer repeats an obsolete phone number for an urgent service, directing someone away from needed assistance: also stale, potentially critical if credible imminent harm is involved. In both cases, an archived record establishes the earlier fact; consequence and urgency determine the response.

If current records contradict a claimed founding date but its historical origin is unknown, record false with history unknown. Do not spend the investigation trying to prove that it was never correct before assigning the narrower actionable finding.

Triage workflow

1. Open and secure the case

Create the case ID, link the observation, remove unnecessary personal or confidential information, set access controls, and name the triage owner.

2. Write the disputed claim

Quote the exact passage and state the affected entity, audience, and decision. Split a report into separate cases when different claims require different evidence or owners.

3. Establish the current record

Use current authoritative evidence and record effective dates, versions, markets, and limitations. For an organization-controlled fact, link the entity fact record or a claim-and-source record. If qualified experts disagree, record the disagreement rather than choosing the preferred answer.

4. Classify and assess severity

Apply the decision tree, record the primary and any secondary class, state the evidence behind the label, and set severity separately. Mark both as provisional until the required reviewer approves them.

5. Choose the response path

Correct the system that owns the fact before polishing a downstream display. A page edit cannot resolve an outdated feed that republishes the old value.

6. Hand off the correction and recheck

Give the owner the preserved observations, approved fact, classification, severity, and next action. Continue with the correction and recheck playbook for controlled changes, outside reporting, comparable rechecks, and closure. Keep submitted, accepted, source corrected, and later answer changed as separate states.

Rapid triage card

Case ID:
Observation ID and evidence location:
Exact disputed passage:
Affected entity:
Affected audience and decision:

Verified current fact:
Evidence and effective date:
Evidence limitations:

Primary class:
Secondary class, if needed:
Classification rationale:

Severity:
Likely consequence:
Urgency or response target:

Probable source or cause:
First response path:
Owner:
Specialist reviewer, if required:
Next action and due date:
Recheck date and conditions:

Quality check

Before acting, confirm that the team has preserved the complete output, isolated a material claim, used current evidence, distinguished issue type from severity, avoided “hallucination” as the only diagnosis, protected sensitive information, assigned an owner, chosen a response proportional to the evidence, and defined a comparable recheck.