L3S Best Publication of the Quarter (Q4/2024 – Q1/2025)
Category: Multimodal Deep Learning for Healthcare
Robust Fusion of Time Series and Image Data for Improved Multimodal Clinical Prediction
Authors: Ali Rasekh, Reza Heidari, Amir Hosein Haji Mohammad Rezaie, Parsa Sharifi Sedeh, Zahra Ahmadi, Prasenjit Mitra, Wolfgang Nejdl
Published in IEEE Access
The paper in a nutshell?
In modern healthcare, medical data comes in different forms, such as time series from patient monitoring and medical images like X-rays. However, integrating these diverse data types to make accurate clinical predictions is challenging. Our research introduces a novel deep learning approach that effectively combines time series and image data using specialized encoders and attention mechanisms. This enables the model to learn meaningful patterns from both data types, making clinical predictions more accurate and robust, even in noisy environments. Our method also incorporates an uncertainty-aware loss function to handle imbalanced datasets, improving reliability in real-world medical applications. We validated our approach using the MIMIC dataset and demonstrated significant improvements in predicting patient outcomes and disease phenotyping.
Which problem do you solve with your research?
Our research tackles the challenge of integrating different types of clinical data—such as time series from patient records and medical images—to improve predictive accuracy in healthcare. Traditional AI models often struggle to combine these modalities effectively, leading to suboptimal predictions in tasks like disease phenotyping and mortality prediction.
What is new about your research?
We introduce a novel multimodal fusion technique that uses specialized encoders for each data type and an attention mechanism to selectively focus on the most relevant features. Additionally, we incorporate an uncertainty-aware loss function to improve robustness in noisy medical environments.
What is the potential impact of your findings?
Our method enhances the accuracy and reliability of AI-driven clinical decision-making, helping healthcare professionals make better-informed diagnoses and predictions. This could lead to earlier disease detection, improved treatment planning, and ultimately, better patient outcomes.
Paper link: https://ieeexplore.ieee.org/abstract/document/10752509
