Skip to main content
Unit Testing runs AI-generated conversations against your agent before you rely on a change in live calls. Use it to exercise a defined scenario repeatedly, check expected behavior, and compare results across runs. Unit tests complement—not replace—Web Chat, Web Call, and phone testing. They are most useful for repeatable scenarios and regression checks.

​
Create a test case

Open Testing > Unit Testing, select Test Cases, then add a test case. Write the description as a concrete caller scenario. Include the caller’s goal and any important constraints, but do not prescribe the agent’s response—the agent should follow its own configuration.
Each test run can incur usage charges. The current limits and price are shown in the test form before you run it.

​
Choose data sources and evaluators

Use Data Sources to define the conversations or criteria associated with a test case:
  • Call Recordings are recordings you upload for testing.
  • Phonely’s Calls are calls already handled by the agent.
  • AI Evaluator defines an AI-generated caller and the criteria used to judge the interaction.
Select only sources that represent the behavior you want the test case to cover. Keep evaluators focused on one scenario so a failed result is easier to understand.

​
Run tests

Run an individual test case from its card, or select Run All Tests to run every configured case for the agent. Phonely moves the new run to Test Results while it is processed. Changing the agent does not retroactively change an earlier result. Run the relevant cases again after modifying prompts, flows, knowledge, voice behavior, or other settings that affect the scenario.

​
Review results

Open Test Results and select a run to review its overall result and individual test instances. The result identifies the test case snapshot, AI evaluator, and status for each generated conversation. Review failed and inconsistent instances individually. A useful follow-up is to compare their conversation paths and determine whether the issue comes from the scenario, the evaluator, or the agent configuration.
Make the test case description more specific, confirm that the selected data sources and evaluator represent the intended scenario, then run it again. Also verify that the agent version you intended to test contains the relevant changes.
Lower Variance, narrow the scenario, and separate unrelated goals into different test cases. Some variation is expected because the conversations are generated by AI.
Refresh Test Results and retry the case. If it fails again, reduce the number of instances and verify that the agent and selected data sources are still available.
Last modified on August 10, 2026