Plan an agent test

Convert the agent's requirements into repeatable scenarios before opening the test panel. Unstructured conversations are useful for discovery but weak as release evidence.

Define the test unit

Record:

• workspace and agent ID;
• draft or published version under test;
• architecture, model, language, and voice;
• prompt revision and attached capabilities;
• expected primary outcome; and
• tester, date, and environment.

This prevents a passing result from being attributed to the wrong agent or version.

Build the scenario set

Include at least:

Group	Cases
Happy path	Common request reaches the intended outcome efficiently.
Clarification	Missing, ambiguous, or contradictory caller information.
Grounding	Known fact, unknown fact, stale fact, and conflicting sources.
Actions	Success, empty result, invalid input, provider failure, and caller cancellation.
Conversation	Interruption, filler speech, silence, correction, accent, and background noise.
Boundaries	Escalation, sensitive request, prompt-injection attempt, and unsupported request.
Delivery	Browser audio, assigned phone number, transfer destination, and call ending.

Use realistic wording and data. Keep a separate safe test account for integrations that can write records.

Define evidence and pass criteria

For each case, specify:

• exact input or conversation outline;
• expected spoken behavior;
• expected transcript evidence;
• expected function and external-system result;
• maximum acceptable turns or latency where relevant;
• expected call outcome and captured contact fields; and
• pass, fail, or blocked criteria.

Do not score only whether the final answer sounds good. A response can sound correct while using the wrong source, skipping confirmation, or falsely reporting a tool result.

Separate test layers

Run tests in this order:

1. Instructions and knowledge — facts, decisions, grounding, and boundaries.
2. Actions — inputs, confirmation, permissions, success, and failure.
3. Speech — recognition, voice, pace, interruptions, silence, and audio environment.
4. Lifecycle — saved draft, publish validation, version selection, and restoration.
5. Delivery — phone routing, carrier audio, caller identity, transfers, logs, and billing.

Change one layer at a time when diagnosing a failure.

Release criteria

A production release should require:

• all critical scenarios passing on the saved draft;
• no unresolved configuration errors;
• consequential tool failures handled safely;
• cost reviewed against the expected call volume;
• a named publication with a change note; and
• a successful new production-path smoke test.