← Back to projects
AI EVALUATION · TRUST & SAFETY · HUMAN-IN-THE-LOOP

LLM Safety & Quality Gate

A hands-on workflow showing how AI evaluation signals become risk-based operational decisions.

Grounding, quality and confidence, then policy risk; routing sends the output to automation, human review or policy escalation.
The v1 evaluation and routing workflow as built in n8n.
The v1 workflow as actually built in n8n.

Why I built it & impact

Built as a hands-on work sample for AI Prompt Engineering / Trust Review Operations roles, to demonstrate real evaluation judgment rather than just describe it. The real test run exposed a genuine routing failure that would have let unsafe financial advice through — evidence that this isn't a theoretical exercise, but a working diagnostic for where automation should and shouldn't be trusted.

What I explored

Designed an LLM quality gate to evaluate model outputs across grounding, quality, confidence and policy risk, then determine the appropriate operational route.

Evaluate → Analyse → Route → Action

The workflow combines structured LLM evaluation with threshold-based decision logic to determine automation, human review or policy escalation.

Real test · Financial advice risk

A real test exposed a weakness in V1. The model produced an unsafe financial advice response with 0.05 quality and 0.97 evaluator confidence.

The key insight: “Routing failure, not model failure.”

V1 routing logic

  • Policy flag = true → Policy escalation
  • Quality ≥ 0.75 + Confidence ≥ 0.70 → Auto-approve
  • Confidence < 0.70 → Human review
  • Otherwise → Unscored fallback (no silent drops)

Root cause

High evaluator confidence could prevent a low-quality, high-risk output from reaching human review. Confidence alone was not a safe human-review gate.

V2 direction

Separate quality and risk from evaluator confidence. Policy risk becomes the primary escalation driver, while quality and confidence help determine when automation is safe.

Validation

Validation pending: re-run and edge-case testing.

Success should be assessed through policy-risk precision and recall, human-in-the-loop efficiency, and calibration between evaluator confidence and independently assessed output quality.

Impact metrics — how success is measured

  • Precision & Recall — the gate's ability to distinguish policy-risk cases from content-safe fallbacks
  • Human-in-the-loop efficiency — whether safe automation reduces unnecessary human review without increasing missed risk
  • Calibration — the relationship between evaluator confidence and independently assessed quality

Built with

n8n · Claude · LLM Evaluation · Trust & Safety · Risk-based Routing · RCA

What it demonstrates

LLM evaluation, model behaviour analysis, Trust & Safety reasoning, human-in-the-loop decision design, RCA and evidence-based iteration.

Quality matters.
Interested in AI quality?

Let’s connect.

Email me  →