Reference-based Super-Resolution via Image-based Retrieval-Augmented Generation Diffusion

Most existing diffusion models have primarily utilized reference images for image-to-image translation rather than for super-resolution (SR). In SR-specific tasks, diffusion methods are only dependent on low-resolution (LR) inputs, limiting their ability to leverage reference information. Prior reference-based diffusion SR methods have demonstrated that incorporating appropriate reference images can significantly enhance reconstruction quality; however, identifying suitable references in real-world scenarios remains a critical challenge. Recently, Retrieval-Augmented Generation (RAG) has emerged as an effective framework that integrates retrieval-based and generation-based information from databases to enhance the accuracy and relevance of response to a given query. Inspired by RAG, we propose an image-based RAG framework (iRAG) for realistic super-resolution. iRAG employs a trainable hashing function to effectively retrieve either real-world or generated reference images given a query LR input. The retrieved patches are then passed to a restoration module, where they are leveraged to augment the retrieved information and generate high-fidelity super-resolved features. Furthermore, to improve the quality of generated references from pre-trained diffusion models, we adopt a hallucination filtering mechanism, leading to overall performance enhancements. Experimental results demonstrate that our approach not only resolves the practical difficulties of reference selection but also delivers superior performance compared to existing diffusion-based and non-diffusion RefSR methods.