Measuring answer quality
Regression suites, blind comparisons with another assistant, and a nightly review of real anonymised questions, so improvements are measured rather than assumed.
Goals
• Fixed regression questions with must and must-not patterns written against reviewed figures, run every week; new failures are treated as defects.
• Blind A/B reviews of eChat's answers against another assistant's, graded the same way against fresh library passages and an expert's answer.
• Keep real questions anonymously (masked, no identity, deleted after 90 days), let users rate answers, and review the weakest each night.
• Turn the review's findings into proposed test cases that people approve; nothing changes answers automatically.
Skills we are looking for
- LLM evaluation
- Statistics
- Vietnamese NLP