Reflective flares are common artifacts in photography that degrade image quality, introducing in-focus flares, which appear as bright, regular spot patterns, and out-of-focus flares, which are diffuse and semi-transparent, obscuring the underlying scene. While previous methods have achieved some success in removing in-focus flares, they struggle with the diffuse nature of out-of-focus flares. The lack of an out-of-focus flare dataset has further hindered the development of effective flare removal models. In this work, we construct a large-scale out-of-focus flare dataset generated based on physical principles. We propose a novel color alignment approach using diffusion models to address the challenges of out-of-focus reflective flare removal. Rather than reconstructing flare-affected regions, our method adjusts the color distribution to reduce artifact visibility while preserving image content. Specifically, we introduce a differentiable histogram loss, derived from the Earth Mover's Distance (EMD), to effectively align color distributions. The proposed approach outperforms existing methods on both synthetic and real-world data, demonstrating improved performance in flare removal.