L3S Best Paper of the Quarter (Q2/2024)
Category: NLP, Generative AI
MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization
Authors: Gunjan Balde, Soumyadeep Roy, Mainack Mondal, Niloy Ganguly
Presented at IJCAI 2024, the 33rd International Joint Conference on Artificial Intelligence (A* Conference)
https://arxiv.org/abs/2405.04163
The paper in a nutshell:
Our work presents a dynamic vocabulary adaptation strategy, MEDVOC, for fine-tuning pre-trained language models (PLMs) for improved medical summarization. We propose a novel efficient, dynamic, and target task-aware vocabulary adaptation strategy that treats vocabulary as an optimizable parameter and optimizes the PLM vocabulary based on fragment score conditioned only on the target task’s reference summaries. This approach enhances model performance in challenging scenarios of high OOV (Out-Of-Vocabulary) concentration and long-form generation for medical text summarization.
Which problem do you solve with your research?
In this work, we solve the problem of ill-tokenization that occurs in the case of medical text summarization w.r.t PLM Vocabulary. This issue affects both the encoder and decoder of PLM: At the encoder stage, excessive splitting leads to poor representation and loss of meaning, and at the decoding stage model has to generate more tokens to generate the corresponding medical word.
What is the potential impact of your findings?
The potential impact is beneficial in the field of medical summarization specifically in the scenarios where factual consistency is a key aspect. Our approach is shown to be more factually consistent compared to baselines as pointed out by medical experts during human evaluation setup.
What is new about your research?
Our work is the first work to propose vocabulary adaptation for fine-tuning open-domain pre-trained language models for medical text summarization. Our proposed methods significantly outperform baselines in challenging scenarios when the reference summary contains high Out-of-Vocabulary words and when the summaries are typically longer. Our approach also improves over baselines in zero-shot and few-shot (10,100) settings.
Preprint: Balde, G., Roy, S., Mondal, M., & Ganguly, N. (2024). MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization. arXiv preprint arXiv:2405.04163.
