Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging

by   Jielin Qiu, et al.

Self-supervised learning is crucial for clinical imaging applications, given the lack of explicit labels in healthcare. However, conventional approaches that rely on precise vision-language alignment are not always feasible in complex clinical imaging modalities, such as cardiac magnetic resonance (CMR). CMR provides a comprehensive visualization of cardiac anatomy, physiology, and microstructure, making it challenging to interpret. Additionally, CMR reports require synthesizing information from sequences of images and different views, resulting in potentially weak alignment between the study and diagnosis report pair. To overcome these challenges, we propose CMRformer, a multimodal learning framework to jointly learn sequences of CMR images and associated cardiologist's reports. Moreover, one of the major obstacles to improving CMR study is the lack of large, publicly available datasets. To bridge this gap, we collected a large CMR dataset, which consists of 13,787 studies from clinical cases. By utilizing our proposed CMRformer and our collected dataset, we achieved remarkable performance in real-world clinical tasks, such as CMR image retrieval and diagnosis report retrieval. Furthermore, the learned representations are evaluated to be practically helpful for downstream applications, such as disease classification. Our work could potentially expedite progress in the CMR study and lead to more accurate and effective diagnosis and treatment.


page 2

page 11


End-to-End Diagnosis and Segmentation Learning from Cardiac Magnetic Resonance Imaging

Cardiac magnetic resonance (CMR) is used extensively in the diagnosis an...

An Effect of Spatial Filtering in Visualization of Coronary Arteries Imaging

At present, coronary angiography is the well known standard for the diag...

CHMMOTv1 – Cardiac and Hepatic Multi-Echo (T2*) MRI Images and Clinical Dataset for Iron Overload on Thalassemia Patients

Owing to the invasiveness and low accuracy of other tests, including bio...

Labelling imaging datasets on the basis of neuroradiology reports: a validation study

Natural language processing (NLP) shows promise as a means to automate t...

Self-supervised multimodal neuroimaging yields predictive representations for a spectrum of Alzheimer's phenotypes

Recent neuroimaging studies that focus on predicting brain disorders via...

Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?

Recent advancements in Large Language Models (LLMs) have drawn increasin...

Please sign up or login with your details

Forgot password? Click here to reset