Establishing dense correspondences is crucial yet computationally expensive in many multi-view tasks. Although the state-of-the-art dense matchers typically adopt a coarse-to-fine scheme to mitigate the computational cost, their efficiency is often compromised by the use of heavy models with redundant feature representations, which are essential for desirable results. In this work, we introduce adaptive refinement gathering that significantly alleviates the demand on such computational burdens without sacrificing too much accuracy. The pipeline consists of (i) context-aware offset estimator: exploiting content information for rough features to enhance the offset decoding accuracy. (ii) Locally consistent match rectifier: correcting erroneous initial matches with local consistency. (iii) Locally consistent upsampler: mitigating over-smoothing at depth-discontinuous edges. Additionally, we propose an adaptive gating strategy, combined with the nature of local consistency, to dynamically modulate the contribution of different components and pixels, enabling adaptive gradient backpropagation and fully unleashing the network's capacity. Compared to the state-of-the-art, our lightweight network, termed ArgMatch, achieves competitive performance on MegaDepth, while using 90% fewer parameters, 73% less computation time, and 84% lower memory cost.