PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain¶
Arxiv Link - 2023-10-22 02:20:38
Abstract¶
Biomedical language understanding benchmarks are the driving forces for artificial intelligence applications with large language model (LLM) back-ends. However, most current benchmarks: (a) are limited to English which makes it challenging to replicate many of the successes in English for other languages, or (b) focus on knowledge probing of LLMs and neglect to evaluate how LLMs apply these knowledge to perform on a wide range of bio-medical tasks, or (c) have become a publicly available corpus and are leaked to LLMs during pre-training. To facilitate the research in medical LLMs, we re-build the Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark into a large scale prompt-tuning benchmark, PromptCBLUE. Our benchmark is a suitable test-bed and an online platform for evaluating Chinese LLMs' multi-task capabilities on a wide range bio-medical tasks including medical entity recognition, medical text classification, medical natural language inference, medical dialogue understanding and medical content/dialogue generation. To establish evaluation on these tasks, we have experimented and report the results with the current 9 Chinese LLMs fine-tuned with differtent fine-tuning techniques.
Socials¶
| X | |
|---|---|
| 🚀 Exciting News in the World of AI and NLP! 🌟 Biomedical language understanding benchmarks play a crucial role in advancing artificial intelligence applications with large language model (LLM) back-ends. However, existing benchmarks face limitations such as being restricted to English, focusing solely on knowledge probing of LLMs, or being leaked to LLMs during pre-training. To address these challenges and drive innovation in medical LLMs, we introduce PromptCBLUE - a revamped version of the Chinese Biomedical Language Understanding Evaluation benchmark. PromptCBLUE serves as a comprehensive platform for evaluating Chinese LLMs' multi-task capabilities across various bio-medical tasks. Curious about the results of our experiments with 9 Chinese LLMs fine-tuned using different techniques? Dive into our research paper to learn more: http://arxiv.org/abs/2310.14151v1 Let's continue pushing the boundaries of AI and NLP together! 🌐💡 #AI #NLP #LLM #MedicalAI #Research #Innovation |
🚀 Exciting news in the world of AI and NLP! Explore how the new PromptCBLUE benchmark is revolutionizing Chinese biomedical language understanding evaluation for LLMs. Dive into the research and results here: http://arxiv.org/abs/2310.14151v1 #AI #NLP #LLMs #MedTech #Research #TechInnovation 🧬🔍🤖 |