Concept Bottleneck Models (CBMs) and other interpretable models show great promise for making AI applications more transparent, which is essential in fields like medicine. Despite their success, we demonstrate that CBMs struggle to reliably identify the correct concept values under distribution shifts. To assess the robustness of CBMs to concept variations, we introduce SUB --- a fine-grained image and concept dataset containing 38,400 synthetic images based on the CUB bird dataset. To create SUB, we select a subset of 33 bird classes and 32 concepts from CUB to generate counterfactual bird images where a specific concept, such as wing color or belly pattern, is substituted.To achieve precise control for generated images, we introduce a novel Tied Diffusion Guidance (TDG) method, where noise sharing for two parallel denoising processes ensures that both the correct bird class and the correct bird concept are generated. This novel dataset enables rigorous evaluation of CBMs and similar interpretable models, contributing to the development of more robust methods.Furthermore, we show that the common practice of training CBMs using class-level concept annotations does not lead to generalized recognition of the concepts. Our code and data will be released upon acceptance.