A hands-on workflow showing how AI evaluation signals become risk-based operational decisions.

Built as a hands-on work sample for AI Prompt Engineering / Trust Review Operations roles, to demonstrate real evaluation judgment rather than just describe it. The real test run exposed a genuine routing failure that would have let unsafe financial advice through — evidence that this isn't a theoretical exercise, but a working diagnostic for where automation should and shouldn't be trusted.
Designed an LLM quality gate to evaluate model outputs across grounding, quality, confidence and policy risk, then determine the appropriate operational route.
The workflow combines structured LLM evaluation with threshold-based decision logic to determine automation, human review or policy escalation.
A real test exposed a weakness in V1. The model produced an unsafe financial advice response with 0.05 quality and 0.97 evaluator confidence.
The key insight: “Routing failure, not model failure.”
High evaluator confidence could prevent a low-quality, high-risk output from reaching human review. Confidence alone was not a safe human-review gate.
Separate quality and risk from evaluator confidence. Policy risk becomes the primary escalation driver, while quality and confidence help determine when automation is safe.
Validation pending: re-run and edge-case testing.
Success should be assessed through policy-risk precision and recall, human-in-the-loop efficiency, and calibration between evaluator confidence and independently assessed output quality.
n8n · Claude · LLM Evaluation · Trust & Safety · Risk-based Routing · RCA
LLM evaluation, model behaviour analysis, Trust & Safety reasoning, human-in-the-loop decision design, RCA and evidence-based iteration.