Relevance AI introduces Evals for testing AI agents

On February 9, 2026, Relevance AI launched Evals for testing AI agents. Users group tests into suites, generate varied user interactions automatically and apply LLM judges against specific criteria. The same day, the company added the ability to edit bulk run automation settings without restarting. Relevance AI describes Evals as enterprise-grade testing for mission-critical AI deployments. Users create golden test examples and, according to the entry, evaluate agent performance without triggering actual tool executions or external actions.

The details

• Features: organized test suites, generated test scenarios, AI judges and performance tracking over runs.
• Tool simulations are listed as coming soon.
• Available exclusively for select Enterprise customers.
• Accessed from the agent dashboard's Monitor tab.

Context

Evals was followed by Agent Performance Observability for production monitoring on March 13, 2026. On June 29, 2026 Relevance AI added cost breakdowns and Workforce coverage to evals, and its December 4, 2025 Agent Alerts feature covers real-time notifications. Evals launched alongside enterprise analytics dashboards for agent usage and error trends, added on February 17, 2026, and Claude Opus 4.6 availability on February 11, 2026.

Not disclosed

• Pricing or credit usage for eval runs
• A date for tool simulations