Edge case testing
Deliberately testing unusual, rare, or boundary inputs that your main eval might miss.
Think of it like
Testing a bridge not just with normal traffic but with an overloaded truck on it.
Example
Your chatbot eval includes normal questions, but you add edge cases like "How do you spell your name?", empty queries, and profanity.
How it actually works
Edge cases break systems in predictable ways. Models hallucinate on out-of-distribution inputs. Retrievers fail on ambiguous queries. Building a comprehensive edge case set is tedious but worth it — a single broken edge case in production looks bad. Organize by failure mode: ambiguity, adversarial inputs, off-topic, truncation.
For product teams
One bad edge case in production kills trust. Invest time here.
For engineers
Maintain a curated set of edge cases. Add to it every time you find a production bug.
Related
- Robustness requires it.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome