Verification Checklists

Human review without structure is biased toward surface-level assessment. Readers naturally evaluate whether text sounds right, flows well, and addresses the requested topic. These checks catch obvious failures — nonsensical sentences, irrelevant tangents, formatting errors — but miss the category of errors that AI produces most dangerously: plausible-sounding claims that are factually wrong.

This vulnerability is compounded by anchoring bias. When a reviewer reads AI output knowing what they asked for, they unconsciously evaluate it against their expectations rather than against external truth. If the output matches the general shape of a correct answer, the reviewer's brain fills in the rest. Research on human-AI interaction consistently finds that people over-trust AI outputs that are presented fluently, even when trained to be sceptical.

Checklists break this pattern by forcing the reviewer to ask specific, targeted questions that override the default reading mode. Instead of 'does this look right?', the reviewer asks 'are all named sources real (see hallucination detection)?', 'do the numbers add up?', 'are there any claims I cannot independently verify?' Each question directs attention to a specific failure mode that casual reading would likely miss.

The checklist approach draws on decades of evidence from aviation, medicine, and other high-reliability fields where systematic verification outperforms expert judgement. The same principle applies to AI output review: structured processes catch errors that expertise alone misses, not because the expert lacks knowledge, but because attention is a limited resource that must be directed deliberately.

The most effective checklists are built empirically — from actual errors you have caught in past AI outputs, not from abstract risk categories. Start by keeping a log of every correction you make to AI-generated content over two weeks. Group the corrections into categories: factual errors, missing context, structural problems, tone mismatches, code bugs, formatting issues. The top three to five categories become your core checklist items.

For code-related work, checklists commonly include: null/undefined handling, error handling for async operations, input validation and sanitisation, authentication and authorisation checks, SQL injection or equivalent injection risks, and edge cases around empty inputs or boundary values — gaps often highlighted in AI safety analysis. These categories represent the failure modes where AI code generation is most consistently unreliable.

For written content, checklists focus on: verifiability of named claims, accuracy of quoted statistics, existence of cited sources, consistency of tone with brand guidelines, and completeness of coverage against the original brief. Each item should be framed as a yes/no question that can be answered through a specific action — 'Search for this source and confirm it exists' rather than 'Check that sources are good.'

The checklist should be living documentation. As new failure modes are discovered and old ones become less common (perhaps because improved prompts prevent them), the checklist should be updated. A quarterly review of checklist items — removing ones that no longer catch errors and adding ones prompted by recent failures — keeps the verification effort focused on the current risk landscape rather than historical patterns.

Models can identify errors in their own output when explicitly prompted to look for them — a capability they do not exercise during initial generation. This works because generation and evaluation are different cognitive tasks. During generation, the model optimises for coherent continuation; during evaluation, prompted with specific criteria, it activates critical analysis patterns that can detect issues in existing text.

The technique is straightforward: take the generated output and feed it back to the model (or a different model) along with a structured checklist. 'Review this code for: (1) unhandled exceptions, (2) SQL injection vulnerabilities, (3) missing input validation, (4) race conditions.' The evaluator prompt activates different patterns than the generator prompt, and the model genuinely catches errors it just made.

Using a different model for verification — a technique known as cross-model verification — adds another layer of reliability. If the generating model has a systematic blind spot — perhaps it consistently uses a deprecated API method because that pattern is common in its training data — a different model with different training may flag the issue. This cross-model verification approach catches errors that stem from model-specific biases rather than universal AI limitations.

However, AI self-review has a systematic limitation: it tends to miss errors that stem from training data biases rather than logical mistakes. If the model learned an incorrect pattern — an outdated API method, a deprecated library function — the same bias that caused the error will cause the self-review to pass it. This is why AI self-review supplements human verification but does not replace it, particularly for stakes-based review tier-three outputs where the cost of error is high.

The primary adoption challenge is that checklists feel like overhead until they catch a significant error. Teams that introduce checklists alongside a log of errors they have caught build buy-in faster than teams that introduce checklists as a compliance requirement. Tracking 'checklist saves' — errors caught by the checklist that would have reached production without it — provides concrete evidence of value that motivates consistent use.

Checklists should be integrated into the team's existing review workflow rather than added as a separate step. If the team uses pull request reviews for code, the checklist becomes part of the PR template. For code-heavy teams, this integrates naturally with AI code review. If the team uses editorial review for content, the checklist becomes part of the editorial handoff. Embedding verification into processes the team already follows reduces friction and increases compliance compared to introducing a standalone review ceremony.

Different team members may need different checklists depending on their output types. A developer generating code, a marketer generating content, and an analyst generating reports face different AI failure modes. Maintaining role-specific checklists that target each role's actual risk profile is more effective than a one-size-fits-all checklist that includes irrelevant items for most users.

Periodic checklist retrospectives — reviewing which items caught errors and which were consistently passed — keep the checklist efficient. An item that has not caught an error in three months may indicate that improved prompts have eliminated that failure mode, or it may indicate that the team has stopped checking it carefully. Distinguishing between these two scenarios determines whether the item should be removed or reinforced.