How OutEvalAI works

OutEvalAI turns agent configuration and evaluation rules into automated AI-to-AI testing workflows. Product and engineering teams configure agents and test cases in the dashboard; OutEvalAI executes evaluation runs, scores conversations, and generates actionable improvement reports.

It is designed around a clear separation of concerns: use cases and agents define the conversation under test, personas and scenarios define test conditions, and evaluation runs compute pass/fail reports. Keeping those responsibilities separate makes test cases reusable and keeps performance improvements easy to verify.

Core workflow

1. Select or create a use case representing the conversation workflow you want to test.
2. Configure or import an agent with its voice, prompt, instructions, and tools.
3. Create test personas to represent realistic caller profiles and behaviors.
4. Define evaluation scenarios with test contexts, opening lines, and expected outcomes.
5. Start an evaluation run to execute AI-to-AI test calls automatically.
6. Review pass/fail reports, transcripts, scores, and failure notes.
7. Refine prompts using AI suggestions and re-run scenarios to confirm fixes.

How the pieces relate

• A workspace owns agents, use cases, test personas, scenarios, and evaluation runs.
• A use case groups an agent workflow under test.
• An agent defines the conversational behavior and prompt instructions.
• A persona defines the caller's identity, background, language, tone, and traits.
• A scenario defines the test situation, context, opening line, and criteria.
• An evaluation run pairs personas and scenarios to produce structured pass/fail results.

Choose the right surface

Start with a controlled rollout

Create representative personas and scenarios that reflect your hardest real-world customer conversations. Run an initial evaluation batch, inspect failure notes and transcripts, then update your agent prompt until all scenarios pass consistently.

Continue with Use Cases or Evaluations.