Verbosity Bias in Preference Labeling by Large Language Models

Arxiv Link - 2023-10-16 05:19:02

Abstract

In recent years, Large Language Models (LLMs) have witnessed a remarkable surge in prevalence, altering the landscape of natural language processing and machine learning. One key factor in improving the performance of LLMs is alignment with humans achieved with Reinforcement Learning from Human Feedback (RLHF), as for many LLMs such as GPT-4, Bard, etc. In addition, recent studies are investigating the replacement of human feedback with feedback from other LLMs named Reinforcement Learning from AI Feedback (RLAIF). We examine the biases that come along with evaluating LLMs with other LLMs and take a closer look into verbosity bias -- a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities. We see that in our problem setting, GPT-4 prefers longer answers more than humans. We also propose a metric to measure this bias.

Socials

LinkedIn

X

🚀 Exciting advancements in Large Language Models (LLMs)! 🤖🧠 The rise of LLMs has reshaped the field of natural language processing and machine learning, with Reinforcement Learning from Human Feedback (RLHF) playing a crucial role in enhancing their performance. Recent studies are delving into the realm of Reinforcement Learning from AI Feedback (RLAIF), exploring how LLMs can learn from one another. 🔍 However, it's essential to consider biases that may arise when evaluating LLMs with other LLMs. A recent study highlighted the verbosity bias, where LLMs tend to favor longer answers, even if shorter responses are equally effective. For example, GPT-4 has shown a preference for lengthier outputs compared to human preferences. Curious to learn more about this bias and its implications? Check out the full study here: http://arxiv.org/abs/2310.10076v1 #AI #NLP #LLMs #MachineLearning #RLHF #RLAIF #BiasInAI #TechResearch #GPT4 #Bard

"🤖🔍 Exciting developments in the world of Large Language Models (LLMs)! Learn how Reinforcement Learning from AI Feedback is shaping the future of LLMs like GPT-4 and Bard. Discover insights on biases in evaluating LLMs and delve into the verbosity bias phenomenon. Check out the study here: http://arxiv.org/abs/2310.10076v1 #AI #NLP #LLMs"

PDF