Agent Eval

Test Platform

Dashboard
Providers
Mock Users
Eval Test Cases
Workspaces
Agents
Tool Catalog
Greetings
LLM Judges
Test Runner
Test Suites

Checking...

Eval Test Cases

0 Test Cases

Loading eval test cases...

📋

No Eval Test Cases

Create your first eval test case to define evaluation criteria for your agents.

Create New Test Case

Basic Info

Name *

Unique identifier for this test case

Description

Tags

Comma-separated tags for filtering

Mock User Goal

Goal *

The simulated user's objective for this test

Test Constraints

Max Turns

1–50 turns allowed

Timeout (ms)

1,000–600,000 ms

Halt Condition

Enable Halt Condition

Optional: halt the conversation early when a substring match is detected, then run graders.

Strategy

Agent Substrings

Halt if any substring matches an agent (ASSISTANT) message

Mock User Substrings

Halt if any substring matches a mock user (USER) message

Case Sensitive

Required

When required, the halt condition must be triggered for the test to pass. If max turns is reached without the halt condition firing, the test fails regardless of grader results.

Graders

Seed Transcript

Source

Upload a JSON file or paste a transcript below. Each message needs a "role" (user/assistant) and "content".

Upload an external trace file (e.g. from OpenAI developer console). The trace will be parsed into seed transcript messages.

Format

Trace File

Upload JSON

Transcript JSON

Optional Context

Mock User ID

Agent Provider ID

Agent System Prompt

Delete Eval Test Case

Are you sure you want to delete ? This action cannot be undone.

Schema Explorer

Eval Test Case Schema

Loading...

Paste JSON to Validate

Test Case Details

Full JSON Config

Agent Eval - Test Platform