Learning to See Inside Opaque Liquid Containers using Speckle Vibrometry

Computer vision seeks to infer a wide range of information about scene objects and events. However, vision systems based on conventional imaging are limited to extracting information only from the visible surfaces of scene objects. For instance, a vision system can detect and identify a Coke can in the scene but cannot determine whether it is full or empty. In this paper, we seek to extend the scope of computer vision to include the novel task of inferring the hidden liquid levels of opaque containers by sensing the tiny vibrations on their surfaces. First, we propose a novel speckle-based vibration sensing system for capturing scene vibrations on a 2D grid of points, at once. We use our system to efficiently and remotely capture a dataset of vibration responses for a plurality of everyday liquid containers. Then, we develop a transformer-based approach for analyzing the captured vibrations and classifying the container type and its hidden liquid level at measurement time. Our architecture is invariant to the vibration source, yielding correct liquid level estimates for controlled and ambient scene sound sources. Moreover, we show that the model can generalize to unseen container instances and fluid levels. We demonstrate our method by recovering liquid levels from various commonplace containers. Code and data will be made publicly available upon acceptance.