ProbMED: A Probabilistic Framework for Medical Multimodal Binding

Medical decision-making requires integrating diverse medical information, from imaging to clinical narratives. These medical modalities are often acquired in a many-to-many manner. However, current medical vision-language pretraining models (Med-VLPMs) fail to directly account for this many-to-many mapping in their model training and embeddings. To address this, we present Probabilistic Modality-Enhanced Diagnosis (ProbMED), a multi-modal Med-VLPM that employs probabilistic contrastive learning to model distributions over embeddings rather than fixed-point, deterministic estimates. ProbMED aligns four distinct modalities—chest X-rays, electrocardiograms, echocardiograms, and clinical text—into a unified probabilistic embedding space. Our framework uses InfoNCE objective with a probabilistic distance metric (Hellinger distance) to integrate inter-modality distributions. To improve intra-modality binding, we introduce a synthetic sampling loss powered by probabilistic embeddings to capture modality-specific mean and variance. Extensive experiments across 13 medical datasets demonstrate that our model outperforms state-of-the-art Med-VLPMs in cross-modality retrieval, zero-shot and few-shot classification. We also show the robust integration of multiple modalities for prognostication, demonstrating the improved intra and inter-modality binding of multimodal medical data embeddings. The anonymized code can be found in https://anonymous.4open.science/r/probMED-8564.