RAG Eval-as-a-Service
Hosted faithfulness, relevance, and context-recall scoring for any retrieval-augmented generation pipeline, extending the grounding work in the RAG Onboarding case study.
AI-generated scoring / try it with your own text
Scores and justifications on this page are generated live by an AI judge agent (Microsoft Foundry, prompt-based). The four sample interactions are synthetic, illustrative examples designed to show specific failure modes. You can also paste your own question, context, and answer to score a real RAG turn — that text is sent only to the judge agent for scoring and is not stored or used for training.
How it works
RAG Eval Judge is a Microsoft Foundry prompt agent that scores one retrieval-augmented generation turn — a question, the context retrieved for it, and the answer a model generated — on three dimensions: faithfulness (is every claim grounded in the context?), answer relevance (does it address the question?), and context recall (did retrieval find everything needed?). This mirrors the evaluation approach in the RAG evaluation and grounding field note ↗.
Try one of four sample interactions built to demonstrate a specific failure mode, or switch to "Paste your own" and score a real RAG turn from your own pipeline.
Try it
Loading samples…
Pick a sample above to begin.
Why this exists
This product extends the RAG Onboarding project record: once a team can ground a model in business data, the next question is always whether that grounding is actually working. A hosted, consistent scoring pass — the same three dimensions on every turn — is what turns "it looks fine in a demo" into something you can track over time and catch regressions before customers do.
Production checklist
- Judge agent treats all submitted text as data to score, never as instructions to follow.
- Both sample and bring-your-own-text paths are bounded in length and independently rate-limited.
- Dedicated Foundry prompt agent, versioned and isolated from the portfolio chatbot.
- Scheduled daily trace evaluation (task completion, coherence) on live usage.
- Automated backend tests cover both input modes, validation, and failure handling.