Vision-Language Neural Graph Featurization for Extracting Retinal Lesions

Retinopathy comprises a group of retinal disorders that can lead to severe visual impairment or blindness. The heterogeneous morphology of lesions poses a significant challenge in developing robust diagnostic systems. Supervised approaches rely on large labeled datasets and often struggle with generalization. To address these limitations, we propose an unsupervised vision-language neural graph featurization method. This method first segments fundus images into a set of super-pixels via Simple Linear Iterative Clustering (SLIC). The super-pixel regions are then decomposed into an undirected graph where each super-pixel serve as a node, and spatially adjacent nodes are connected by edges. A Hamiltonian path systematically traverses the graph and iteratively update and propagate node and edge latent space embeddings throughout the graph until convergence is achieved. Then, a normalized cut separates the converged embeddings into two clusters within a latent space that represent the lesion and healthy super-pixel regions of the input scans. The lesion super-pixels are further classified into lesion categories using prompt-based zero-shot vision-language model. The proposed method is rigorously tested on three public datasets, dubbed ODIR, BIOMISA, and IDRiD, achieving F1-scores of 0.89, 0.93, and 0.92, respectively, with significant performance gains over state-of-the-art methods.