Relevance AI adds production monitoring for agents with LLM judges

On March 13, 2026, Relevance AI added production monitoring to its agent platform, describing a lifecycle of build, test and monitor similar to software development. The feature applies automated LLM-based evaluations to live conversations and reports scores and trends in dashboards. The same day, the company added real-time pause, cancel and resume for agent triggers and a people search engine in Chat. Relevance AI says the feature helps teams catch performance degradations shortly after they start rather than weeks later.

The details

• Evaluation criteria are custom, with examples including summary quality, hallucination detection and workflow execution.
• Sampling options: 10%, 50% or 100% of conversations.
• Available exclusively for select Enterprise customers; not available on non-Enterprise plans.
• Accessed from the agent dashboard's Monitor tab under Test Suites.

Context

Relevance AI introduced Evals for pre-deployment testing on February 9, 2026 and Agent Alerts on December 4, 2025. It later added cost breakdowns and Workforce coverage to evals on June 29, 2026. The same week, on March 9, 2026, Relevance AI listed its agents in the Microsoft Teams App Store and added a Teams integration for triggering agents, following Microsoft Teams calendar event triggers on February 24, 2026.

Not disclosed

• Credit or action cost of running LLM judges on production traffic
• Criteria for which Enterprise customers receive access