OutEvalAI Documentation
Evaluate AI agents with personas, scenarios, and automated reports using OutEvalAI.
OutEvalAI is a workspace for evaluating AI agents before they go live. Product and engineering teams configure usecases under test, define personas and scenarios, and review automated pass/fail reports with transcripts and improvement suggestions.
Use these guides to move from a first evaluation run to a repeatable quality workflow for agent changes.
Choose where to begin
Start with the evaluation model, configure an agent under test, then integrate through the API or SDKs.
Understand evaluation
Learn how personas, scenarios, and AI-to-AI tests fit together.
Configure agents under test
Set up usecases and agents that evaluation runs against.
Run evaluations
Create personas and scenarios, then review automated pass/fail reports.
Explore the API
Browse generated endpoint schemas, examples, and the API playground.
SDKs & MCP
Integrate from TypeScript, Python, or connect AI coding agents through the Model Context Protocol.
Product guides
Focused documentation for the evaluation product surface.
How it works
Understand the OutEvalAI evaluation workflow end to end.
Use Cases
Configure agents under test and the workflows they represent.
Agents
Tune voice, instructions, and behavior for evaluation targets.
Evaluations
Test agent changes against repeatable personas and scenarios.
Usage & Billing
Understand plan limits and workspace consumption.
Recommended workflow
A practical loop for shipping safer agent changes.
- 1
Define the agent under test
Create or select a usecase that represents the conversation you want to evaluate.
- 2
Create personas and scenarios
Describe who the simulated caller is and what success looks like for each test case.
- 3
Run AI-to-AI evaluation
Execute live tests against your usecase and capture transcripts plus structured outcomes.
- 4
Review pass/fail reports
Inspect automated scoring, failure reasons, and suggestions before shipping changes.
- 5
Iterate and re-test
Refine prompts and agent configuration, then re-run scenarios to confirm improvements.
How is this guide?