SMILE: A Smarter, Fairer Metric for Grading Q&A Systems
How do we fairly score an AI’s answer? Old-school metrics (like ROUGE or Exact Match) reward word overlap, not understanding. LLM judges can “feel” meaning, but they’re costly, inconsistent, and can hallucinate.
Meet SMILE (Semantic Metric Integrating Lexical Exactness): a lightweight way to evaluate answers that blends three signals:
- Sentence-level meaning (does the whole answer make sense?)
- Keyword-level meaning (are the key ideas there?)
- Exact keyword matches (are crucial terms correct?)
This balance captures both what is said and how precisely it’s said—something pure semantics or pure overlap can miss. Across text, image, and video question answering, SMILE aligns strongly with human judgments while staying fast and affordable to run.
Why it matters: better metrics mean more reliable benchmarks, fairer model comparisons, and faster progress—without relying on black-box LLM judges.
Paper: https://arxiv.org/abs/2511.17432v1
Paper: https://arxiv.org/abs/2511.17432v1
Register: https://www.AiFeta.com
#AI #NLP #QuestionAnswering #Evaluation #Metrics #ComputerVision #VQA #LLM #Research
