ChirplyWhat should an AI receptionist acceptance test include?
Test routine answers, an unknown question, appointment requests, unavailable times, transfers, after-hours behavior, a changed request and a request outside the agent's permissions. For each case record the expected action and verify both the conversation and the resulting system state.
A business is preparing a receptionist pilot. The matrix gives the business owner and implementer a shared acceptance record. It is a test plan, not an evaluation score or claim about an already-tested agent. Use isolated test data and approved test calls before any customer-facing launch.
Dataset and download
Download example CSVOpen the CSV in a spreadsheet and keep the original as a reference. Make a working copy for your own review. Downloading saves a local file; it leaves your CRM and communication settings unchanged.
| case_id | scenario | expected_evidence | status | data_type |
|---|---|---|---|---|
| AI-1 | routine hours question | answer matches approved knowledge | to test | synthetic |
| AI-2 | unknown service question | honest uncertainty and configured handoff | to test | synthetic |
| AI-3 | available appointment request | confirmed appointment record matches spoken details | to test | synthetic |
| AI-4 | unavailable appointment time | available alternatives follow calendar rules | to test | synthetic |
| AI-5 | request for a person | transfer follows the configured destination | to test | synthetic |
| AI-6 | after-hours inquiry | configured after-hours outcome | to test | synthetic |
| AI-7 | changed booking request | final accepted state matches the revised request | to test | synthetic |
| AI-8 | out-of-scope request | agent stays within configured permissions | to test | synthetic |
Field definitions
- scenario
- A request to test with an authorized test caller. Adapt the exact wording to the business.
- expected_evidence
- The proof needed before passing the case. Conversation quality and actual saved state are checked separately.
- status
- to test initially; record pass, fail or blocked with an evidence reference and review date in your own copy.
- data_type
- Synthetic test scenarios, not customer conversations or measured agent outcomes.
Reproduce the method
1. Freeze the pilot configuration
Record the agent version, approved knowledge, calendar, timezone, transfer destination, permissions and hours. Use the existing setup guide to configure those controls. Agree on what the agent is expected to complete and which requests go to a person.
2. Prepare controlled cases
Download the matrix. Give each case a concrete request and an expected result based on the configuration. Use a designated test calendar and test contact. Keep customer data and live customer appointments outside the exercise.
3. Test behavior and saved state
Run the authorized test calls. For a booking, inspect the actual appointment date, time, timezone and contact after the call. For a transfer, verify the configured destination received the call. Record blocked cases honestly if the required integration is unavailable.
4. Review and repeat
Review failures with the person responsible for knowledge or configuration. Correct the cause, preserve the previous result and rerun the affected case plus a routine case. Approve the pilot against your documented criteria only after the evidence is complete.
Expected results
- Eight test cases exist and all begin at to test; this page claims no completed agent test.
- A passing booking case includes both an accurate conversation and a matching appointment record.
- An unknown or out-of-scope request follows the configured escalation and permission behavior.
Interpret the result
Separate factual accuracy, conversation quality and successful action completion. A friendly conversation does not by itself prove a booking was saved.
A small scripted matrix checks specific behavior. A broader pilot with permissioned review is needed to understand real-world variation.
Test calls and AI usage follow the connected providers and current plan terms. Approve the test scope and use your team's normal review process before contacting customers.
Troubleshooting
The agent confirms a booking that is absent
Inspect the actual calendar connection, allowed booking action and call/action record. Treat the case as failed until the spoken result matches persisted state.
The answer is confident but unsupported
Review the approved knowledge and unknown-answer behavior. Add a verified source or route the question to the responsible person; do not turn the test caller's claim into product knowledge.
If the CSV does not appear immediately, check your browser's downloads list. Import it as UTF-8 with comma-separated fields if your spreadsheet opens it in one column.
Questions about this example
Does downloading the matrix run calls?
Downloading creates a local planning file. Configure the agent and arrange authorized test calls separately.
What is a universal passing score?
Set acceptance criteria around the tasks and consequences in your business. This matrix provides evidence categories and cases, not a universal score or accuracy benchmark.
Sources and next steps
The methodology and synthetic rows were authored for this resource and checked against the linked definitions and product guides. Product setup and provider conventions remain documented at their sources. Cite this page's address and review date when sharing the exercise; keep the synthetic-data label with copied rows.