How accurate is Chatty's AI? Our method and results
We compared 300 real answers from Chatty’s AI with what each store shows on its own website. When the AI talked about a product, it got the price, stock and details right 92% of the time.
Checked on 23 October 2026
92%
of product answers right on every detail (91 of 99)
87.5%
of all answers we could compare (119 of 136)
300
real answers picked at random, 15 to 22 October 2026
What we compared
For each answer, we opened the store’s website and compared what the AI said with what the store shows there.
An answer counts as right only when every detail we could compare was right. One wrong price marks the whole answer as not matching.
The results
We could compare 136 of the 300 answers with the store’s website. 119 of them were right on every detail. Here is the result for each type of question:
Product questions
89.6% 43 of 48
Product recommendations
94.1% 48 of 51
Shipping, payment and policy questions
76.7% 23 of 30
After-sale questions
71.4% 5 of 7
What happened to all 300 answers
- 119 Right on every detail
- 17 Not right on every detail
- 95 Actions, not facts
- 14 Handed to a person
- 55 Details the website doesn’t show
Actions are answers such as opening a return form or asking for an order number. Details the website doesn’t show include order status, and a few websites blocked our check.
Where the AI got it wrong
Seventeen of the 136 answers were not right on every detail. They fall into three groups:
9
No longer matched the website
Most were prices or stock the store changed after the chat, for example when a sale ended. Two were shipping rules, such as free shipping from 39 € when the page now says 35 €.
6
Partly right
The delivery time was a day or two off the store’s page, or the answer said two different things about a shipping fee.
2
Wrong
One offered a ring size the product does not come in. One answer came out garbled and stopped mid-sentence.
Shipping and return answers go wrong more often for a simple reason. The AI uses the policy text the store gives Chatty. When that text is different from the store’s own policy page, the AI repeats the old version. Merchants can spot this in Test AI with Review sources.
How we picked the answers
- Answers the AI gave on Chatty stores between 15 and 22 October 2026.
- 75 picked at random for each of four types of question, no more than one per store for each type.
- Test stores, demo stores and our own test chats left out.
- Every answer checked against the store’s website on 23 October 2026.
What this check can’t tell you
- We checked the websites a few days after the chats, so some answers look wrong only because the store changed a price later.
- We couldn’t check answers about things only the store can see, such as a shopper’s order.
- Fewer shipping, return and after-sale answers had details to compare, so those numbers are less certain.
How Chatty keeps answers accurate
Answers from the store’s data
Products, variants, prices, stock, policies and the FAQs the merchant adds.
Shows its sources
In Test AI, Review sources lists the questions, products and pages behind each reply.
Hands over when unsure
Two low-confidence replies, a repeated question or a frustrated shopper sends the chat to the team.
Brings the misses back
Unresolved questions are grouped by reason, and Chatty Lab proposes fixes the merchant approves.
Want the evidence?
We can share the full scoring sheet for all 300 answers, with store names removed.
Questions about this check
Try Chatty on your own store
Install free and check every answer in Test AI before shoppers see it.