INTRODUCING THE AR PRACTICE   ·   A practical field guide to artificial resources, work and people
THE AR FIELD GUIDE / Assurance

Confidence needs evidence.

High output speed and persuasive language are not proof of quality. Assurance connects AI work to testable outcomes and accountable decisions.

OUTCOME ASSURANCE

Know when to trust the work.

Define what “good” means before evaluating a system. The evidence might include factual accuracy, safety, consistency, completeness, fairness, resilience and whether the result actually helps the intended user.

Different applications need different assurance levels. Drafting a low-consequence internal summary is not the same as authorising a payment or influencing a safety-critical decision.

A QUESTION OF RESPONSIBILITY

“Who is accountable when the output looks plausible, but is wrong?”

Allocate ownership for the outcome, rather than merely appointing someone responsible for buying or maintaining the tool.

A PROPORTIONATE APPROACH

Three levels of scrutiny

LOWER CONSEQUENCE

Routine assistance

Spot checks, basic acceptance criteria, clear data handling and a simple correction route.

Example: drafting an internal meeting summary.

MATERIAL CONSEQUENCE

Operational decisions

Representative test sets, measurable failure limits, human approval and documented exception handling.

Example: triaging customer claims or operational work orders.

HIGH CONSEQUENCE

Critical decisions

Independent challenge, domain-specific controls, stronger documentation, monitoring and clear non-automation boundaries.

Example: decisions affecting safety, legal rights or significant financial exposure.

These are illustrative tiers, not a legal classification or substitute for applicable sector requirements.

BUILD AN ASSURANCE CASE

Six questions that
require evidence.

  1. 01
    What is the intended outcome?

    Define the task boundary, stakeholders, unacceptable outcomes and performance baseline.

  2. 02
    How will success be judged?

    Set criteria for accuracy, completeness, timeliness, usability and other important qualities.

  3. 03
    What could go wrong?

    Consider normal error, edge cases, malicious inputs, data quality, bias, outages and unintended actions.

  4. 04
    What has actually been tested?

    Use representative and challenging cases; record failures, limitations and independent review where justified.

  5. 05
    Who can detect, stop and correct failure?

    Define monitoring, permission boundaries, escalation, human review and recovery.

  6. 06
    When must assurance be repeated?

    Reassess when models, tools, prompts, users, data sources or operating conditions materially change.

↑ The opportunity

Where value can be created

  • Discover defects before they become customer or operational incidents.
  • Earn confidence in a capability through transparent, repeatable evidence.
  • Improve quality by turning observed failure patterns into system design changes.
↓ The risk

What needs attention

  • Plausible answers produce false confidence and weak checking.
  • Human approvals become superficial when throughput pressures rise.
  • Changes in models, data or tools invalidate an earlier evaluation.
Decision checkpoint: Record the acceptance criteria, evidence owner, test outcomes, residual limitations and explicit sign-off for the use case.
STANDARDS AS USEFUL REFERENCES

Connect practice to recognised thinking.

Standards and frameworks can structure discussions, but a reference to a standard is not itself proof that an individual AI system is safe or effective.

Relevant starting points include NIST's AI Risk Management Framework and ISO/IEC 42001. Legal, contractual and industry-specific obligations should be assessed separately.

Evidence to retain

  1. Defined purpose and approved scope.
  2. Model / workflow versions and dependencies.
  3. Evaluation design, datasets and results.
  4. Known failure modes and residual risks.
  5. Named owner and approval record.
  6. Monitoring, incident and review history.
MAKE THE NEXT DECISION BETTER

Explore performance in the round.

Try a configurable example that considers effectiveness, quality, assurance and human wellbeing together.