Sample: AI Chatbot / RAG Health Check
What the report looks like: each wrong answer recorded, traced to its cause, and turned into a fix list in priority order.
This is a sample, not a client. The business in this report is made up, and so are its numbers, to show the format. It is not a real company and not a result we achieved.
Summary: Example Software Co., help-desk assistant
- Real questions tested
- 20
- Correct
- 12
- Partly correct
- 3
- Wrong or unsupported
- 5
- Typical response time
- 3.4 s
- Cost per answer
- $0.011
Main conclusion: most failures start in search and in out-of-date documents, not in the model. Changing the model would not fix them.
Failures and fixes, in priority order
High priority
Old pricing quoted as current (3 of 20)
Both the 2024 and 2026 price lists are indexed. Search returned the 2024 page first for each pricing question.
What to do: Remove the old list, and add a document date so newer sources rank first.
High priority
Answers given with no supporting source (2 of 20)
Questions on refunds found nothing relevant, but the assistant answered anyway from general knowledge.
What to do: Tell it to say it doesn't know when no passage qualifies, and hand off to a person. Add the missing refund policy.
Medium priority
Product codes not found (3 of 20, partly correct)
Vector search alone misses exact terms like “SKU-204”.
What to do: Add keyword search alongside vector search.
Low priority
Answers slower than they need to be
Each request sends 12 passages, and most of them are not used.
What to do: Add a reranker and send the best 5. This should also lower cost per answer; measure again after the change.
Checked, and no issue found
- No exposed API keys in the pages we tested
- No personal data in the sampled answers
Want this for your own system?
$149 · £119. Usually 5–7 working days. You keep the report whether or not we work together afterwards.