Developing robust and interpretable vision systems is a crucial step towards trustworthy artificial intelligence. One promising paradigm is to design transparent structures, e.g., geometric invariance, for fundamental representations. However, such invariants exhibit limited discriminability, limiting their applications in larger-scale tasks. For this open problem, we conduct a systematic investigation of hierarchical invariance, exploring this topic from theoretical, practical, and application perspectives. At the theoretical level, we show how to construct discriminative invariants with a Convolutional Neural Network (CNN)-like hierarchical architecture, yet in a fully transparent manner. The general blueprint, specific definitions, invariant properties, and numerical implementations are provided. At the practical level, we discuss how to customize this transparent framework into a given task. With the over-completeness, discriminative features w.r.t. the task can be adaptively formed in a Neural Architecture Search (NAS)-like manner. We demonstrate the above arguments with accuracy, invariance, and efficiency results on laboratory-style classification experiments. Furthermore, at the application level, our representations are explored in real-world forensic tasks on adversarial perturbations and generated content. Such applications reveal that our invariants exhibit competitive discriminability even in the era of deep learning. For robust and interpretable vision tasks at larger scales, hierarchical invariant representations can be considered as an effective alternative to traditional CNNs and invariants.