COSTARR: Consolidated Open Set Technique with Attenuation for Robust Recognition

Handling novelty is a common challenge in visual recognition systems. Existing open-set methods rely on the familiarity hypothesis, detecting novelty by the absence of familiar features. We introduce a novel attenuation hypothesis, arguing that small weights learned during training, which attenuate features, play a dual role: they differentiate known classes but also discard information valuable for distinguishing known and unknown classes. How to effectively leverage this attenuation information to enhance open-set recognition remains unclear, so we present COSTARR, a novel approach that combines the requirement of familiar features and the lack of unfamiliar ones. We provide a probabilistic interpretation of the COSTARR score, linking it to the likelihood of correct classification and belonging in a known class. To determine the individual contributions of the pre- and post-attenuated features to COSTARR's performance, we conduct ablation studies that demonstrate both pre-attenuated deep features and the underutilized post-attenuated Hadamard product features are essential for improving open-set recognition. Also, to validate generalizability and efficacy across diverse architectures and datasets, we evaluate COSTARR on a large-scale setting, using ImageNet2012-1K as known data and NINCO, iNaturalist, OpenImage-O and other datasets as unknowns, across multiple modern pre-trained architectures (ViTs, ConvNeXts, and ResNet). The experiments demonstrate that COSTARR generalizes effectively across various architectures and significantly outperforms prior state-of-the-art methods by incorporating previously discarded attenuation information, thus advancing open-set recognition capabilities. Code available upon publication.