Can LLM be a Personalized Judge?

Arxiv Link - 2024-06-17 15:41:30

Abstract

Ensuring that large language models (LLMs) reflect diverse user values and preferences is crucial as their user bases expand globally. It is therefore encouraging to see the growing interest in LLM personalization within the research community. However, current works often rely on the LLM-as-a-Judge approach for evaluation without thoroughly examining its validity. In this paper, we investigate the reliability of LLM-as-a-Personalized-Judge, asking LLMs to judge user preferences based on personas. Our findings suggest that directly applying LLM-as-a-Personalized-Judge is less reliable than previously assumed, showing low and inconsistent agreement with human ground truth. The personas typically used are often overly simplistic, resulting in low predictive power. To address these issues, we introduce verbal uncertainty estimation into the LLM-as-a-Personalized-Judge pipeline, allowing the model to express low confidence on uncertain judgments. This adjustment leads to much higher agreement (above 80%) on high-certainty samples for binary tasks. Through human evaluation, we find that the LLM-as-a-Personalized-Judge achieves comparable performance to third-party humans evaluation and even surpasses human performance on high-certainty samples. Our work indicates that certainty-enhanced LLM-as-a-Personalized-Judge offers a promising direction for developing more reliable and scalable methods for evaluating LLM personalization.

Socials

LinkedIn

X

🌟 Exciting Research Alert! 🌟 Ensuring that large language models truly reflect diverse user values and preferences is crucial as their global user bases expand. The latest research delves into the realm of LLM personalization, aiming to enhance user experience. Check out the intriguing findings on the reliability of LLM-as-a-Personalized-Judge approach in evaluating user preferences based on personas. Discover more about this insightful study and its implications for the future of LLM personalization here: http://arxiv.org/abs/2406.11657v1 #AI #LLM #NLP #Personalization #Research #Tech #Innovation #UserExperience

"Just in: Research investigates the reliability of LLM-as-a-Personalized-Judge for evaluating user preferences based on personas. Findings show low reliability without verbal uncertainty estimation. Learn more at: http://arxiv.org/abs/2406.11657v1 #LLM #personalization #AI"

PDF