Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models¶
Arxiv Link - 2024-05-01 15:37:11
Abstract¶
As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model's freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like GPT4. While this method has grown in popularity, it is costly, has been shown to introduce intramodel bias, and in this work, we find that very large models are often unnecessary. We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.
Socials¶
| X | |
|---|---|
| 🚀 Exciting advancements in Large Language Models (LLMs)! A recent study delves into evaluating LLM quality with a fresh approach - using a Panel of LLM evaluators (PoLL). The research suggests that employing multiple smaller models in the evaluation process yields superior results compared to a single large model like GPT4, while also reducing costs significantly. Learn more about this innovative methodology here: http://arxiv.org/abs/2404.18796v2 #AI #NLP #LLMs #TechInnovation 👩💻📈 | 🚀 New study on evaluating the quality of Large Language Models (LLMs) reveals a more efficient method - Panel of LLM evaluators (PoLL). Using multiple smaller models outperforms a single large judge, reduces bias, and is over seven times less expensive. Check out the research at: http://arxiv.org/abs/2404.18796v2 #AI #NLP #LLM #Research #TechInnovation |