Learning Interpretable Queries for Explainable Image Classification with Information Pursuit

Building image classification models that are both highly accurate and interpretable remains a challenge in computer vision. Information Pursuit (IP) is an information-theoretic framework for interpretable-by-design sequential prediction. Given a set of task-relevant and semantic data queries, IP selects a sequence of queries in order of information gain and updates the posterior at each step based on the gathered query-answer pairs. To carry out IP, previous methods construct hand-crafted dictionaries of potential data queries, curated either by a domain expert or by prompting large language models. However, in practice, such hand-crafted dictionaries are limited by the expertise of the curator and the heuristics of prompt engineering, resulting in a gap between the predictive performance of IP versus non-interpretable black-box predictors. In this work, we propose to parameterize the IP queries as a learnable dictionary defined in the latent space of vision-language models such as CLIP. Drawing inspiration from sparse dictionary learning, we propose an alternating optimization algorithm that iterates between solving IP's optimization problem for a fixed query dictionary and optimizing the dictionary to maximize classification accuracy. Empirically, our experiments show that our method learns a query dictionary that reduces the accuracy gap between explainable image classification with IP and black-box methods, while preserving interpretability.