Dichromatic Reflection Model (DRM), a widely used physical image formation model, has been extensively applied to specular highlight removal. However, traditional DRM solvers fail to effectively recover the missing content underneath specular highlights and are prone to incur visual artifacts. Additionally, existing deep learning-based methods do not exploit the underlying variables in DRM; instead, they primarily learn to translate an input image into its diffuse image (and specular residue image). As a result, their performance remains somewhat limited. To overcome these issues, we propose a neural DRM solver for specular highlight removal. Our pipeline for the solver consists of three networks: Highlight Detection Network (HDNet), Alpha-chrom Estimation Network (ACENet), and Refinement Network (RNet). Specifically, HDNet is first used to detect specular highlights. Meanwhile, leveraging multi-level contextural contrasted features from HDNet, ACENet estimates the underlying variables in DRM. Using these estimates, our new reconstruction models generate specular-free and specular residue images. To bridge the domain gap between color spaces, we additionally introduce RNet to refine the results. Extensive experiments on various datasets demonstrate that our neural solver is superior to previous traditional solvers as well as deep learning-based methods.