Detecting vehicles in aerial imagery is a critical task with applications in traffic monitoring, urban planning, and defense intelligence. Deep learning methods have provided state-of-the-art (SOTA) results for this application. However, a significant challenge arises when models trained on data from one geographic region fail to generalize effectively to other areas. Variability in factors such as environmental conditions, urban layouts, road networks, vehicle types, and image acquisition parameters (e.g., resolution, lighting, and angle) leads to domain shifts that degrade model performance. This paper presents a novel approach to address this challenging problem by leveraging generative AI for the high-quality synthesis of aerial images and corresponding labels to enhance detector training through data augmentation. Our key contribution is the development of a multi-stage, multi-modal knowledge transfer framework utilizing fine-tuned latent diffusion models (LDMs) to mitigate the distribution gap between the source and target environments. Through extensive experiments across diverse aerial imagery domains, we demonstrate significant performance gains (more than 40% in some cases) over existing domain adaptation and weakly supervised learning methods. Our method also outperforms the baseline detectors trained on a source dataset by 4-12%. Furthermore, we introduce two newly annotated aerial datasets from New Zealand and Utah, which along with the code will be publicly released upon paper acceptance to support further research in this field.