How to know your AI is getting better
Without evaluations, every change to an AI system is a guess. A practical way to build them, starting with twenty real examples.
Someone changes a prompt to fix one bad answer, and three good answers quietly get worse. Without a way to check, nobody finds out until a customer does. Evaluations are how you find out first.
Start small and real
Collect twenty real examples of what the system will be asked, with what a good answer looks like. Real beats invented: real questions are messier, and that mess is exactly what you need to test.
Grow it from failures
Every time the system gets something wrong in the real world, add that case to the set. Over a few months it becomes a precise map of your problem's difficult corners, and a guard against the same mistake twice.
Score simply
- Correct: did it get the facts right, checked against the known answer?
- Grounded: did it use the sources it was given, and cite them?
- Safe: did it stay inside the lines you set, such as never promising refunds?
“An average score tells you something changed. Reading the failures tells you what.”
Run the set on every change and keep the history. A small chart of scores over time is the most persuasive document an AI project can show its sponsors.
Written by the team at Bytelights, a product and AI studio in India.