All projects · Case Studies
Agent Router QA Evaluation — RAG Routing Test Harness
A QA test-and-report package that evaluates an AI agent routing workflow (from Puzzle #75's internal_agent_router_workflow.json) which classifies…
A QA test-and-report package that evaluates an AI agent routing workflow (from Puzzle #75's internal_agent_router_workflow.json) which classifies incoming queries as KB (needs company facts via Pinecone retrieval) vs NO_KB (general knowledge / chit-chat). Contains the test criteria, a 24-question run log, machine-readable results, and a synthesized QA report against a fictional 'Ryobi Coffee Co.' knowledge base.
- Two-axis evaluation kept independent: routing correctness (binary KB/NO_KB) scored before response quality (0-3), so a right answer via a wrong route is flagged as a latent LUCKY defect
- Explicit acceptance thresholds with hard floors; False-NO_KB (company question answered from general knowledge) treated as the gating high-severity metric
- 24-question test set deliberately weighted toward boundary and adversarial traps (false-premise, out-of-KB, mixed-intent) where routers actually break
- Root-cause analysis plus prioritized P0-P4 remediation (classifier prompt few-shots, re-route safety net, confidence gate) and a defined re-test sign-off gate
Built with: Markdown, CSV, n8n (referenced workflow), Pinecone vector DB, RAG routing, Gemini 3 Flash, Antigravity agent harness, Webhook + Switch agent architecture