Remote physiological measurement using visible light cameras has emerged as a powerful tool for non-contact health monitoring, yet its reliability degrades under challenging conditions such as low-light environments or diverse skin tones. These limitations have motivated the exploration of alternative sensing modalities, such as near-infrared sensors and radar systems, which offer complementary physiological information due to their distinct sensing principles. While alternative modalities capture complementary physiological cues through distinct sensing principles, existing methods fail to holistically integrate these heterogeneous data. Our key insight is that while visible light, near-infrared, and radar operate on distinct physical principles, they all capture temporally dynamic physiological signatures that can be represented as time-varying signals reflecting underlying physiological processes. Based on this insight, we propose FusionPhys, a novel framework that implements an adaptive integration mechanism to refine physiological information across complementary modalities. We further introduce a sub-modality embedding technique that extends fusion principles to single-modality videos. Extensive experiments across five benchmark datasets demonstrate that FusionPhys achieves competitive performance in diverse sensing configurations, representing a significant advancement toward more reliable and versatile remote physiological measurement systems.