3D Gaussian Splatting (3DGS) has begun incorporating rich information from 2D foundation models. However, most approaches rely on a bottom-up optimization process that treats raw 2D features as ground truth, incurring increased computational costs and an excessive number of Gaussians. We propose a top-down pipeline for constructing compact and fast 3D feature fields, namely, \Ours{}. We first perform a weighted fusion of multi-view features with a pre-trained 3DGS. The aggregated feature captures spatial cues by integrating information across views, mitigating the ambiguity in 2D features. This top-down design enables a per-Gaussian autoencoder strategy to compress high-dimensional features into a 3D latent space, significantly balancing feature expressiveness and memory efficiency. Finally, we introduce an adaptive sparsification method that merges Gaussians to reduce complexity, ensuring efficient representation without unnecessary detail. Our approach produces a competitive 3D feature field using only about 10\% of the Gaussians compared to existing feature-embedded 3DGS methods.