Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data

This paper develops a generalized federated prompt-tuning framework under practical scenarios where local datasets are multi-modal and have different distributional patterns of missing features at the input level. The proposed framework helps bridge the gap between federated learning and multi-modal prompt-tuning which previously focus on either uni-modal or centralized data. A key challenge in bridging this gap is due to the inherent lack of a semantic alignment between prompt instructions that encodes the same distributional patterns of missing data across different clients. To address this challenge, our proposed framework introduces specific client-tuning and server-aggregation designs that learns to simultaneously optimize, align, and aggregate prompt-tuning instructions across clients and data modalities, enabling them to complement one another and be combined effectively. A thorough evaluation of our framework on a variety of multimodal benchmark datasets demonstrates consistent and significant performance improvement over existing state-of-the-art (SOTA) baselines.