Conversation evals

An eval is a question you want answered about every conversation your agent has - “Did it confirm the delivery address?”, “Was the tone warm?” - graded by an AI reviewer after the conversation ends.

Evals are switched on per workspace. If you don’t see them, ask us to enable Conversation evals for your workspace.

Set up criteria

Open the agent, then Settings → After the call (or After the chat) → Evals. Add up to 20 criteria. Each has:

• Name (its key is derived from it and stays fixed once saved).
• Type: Pass / fail, or Score 0-10 with a Pass at threshold (default 7). We decide pass or fail from the threshold, not the reviewer.
• Rubric: what a good conversation does. Be concrete.
• Reviewer instructions (optional), for example “Mark N/A when payment never came up”. N/A answers never count against the pass rate.

Click Save evals. Criteria apply to conversations that end from then on; editing or removing a criterion never rewrites past results.

Cost controls: turn grading off per agent, grade a sample (%) of conversations, or leave phone calls out. Each workspace also has a daily limit. All criteria of a conversation are graded in one reviewer call.

When it runs

Automatically once a chat, WhatsApp conversation or call ends, only for real conversations where the customer wrote or spoke; test chats and test calls are skipped. Open a conversation in Inbox → History (or a call in Calls) and choose the Evals tab to see each verdict, score and the reviewer’s reasoning. If a conversation was not graded, the tab says why (for example, not in the sample, or not graded within 48 hours of ending). Run again grades it with your current criteria; earlier runs stay visible.

Analytics and export

WhatsApp → Analytics, Analytics → Chat and Analytics → Outcomes (calls) show pass rates per criterion and agent (the latest run of each conversation; N/A left out). Download CSV exports one row per criterion per conversation.

Webhook

Subscribe an endpoint to chat.evaluated (it is never sent to “all events” endpoints). See the webhooks reference.

Privacy

The reviewer is told not to quote personal details, and treats the conversation as data, never as instructions. When a person’s data is erased, the reasoning on their conversations is removed and any grading still in progress is cancelled; verdicts and scores remain.