The Cosmic Microwave Background (CMB) radiation is a pillar of modern cosmology. This GHz-range signal gives rise to better understanding of the fundamental parameters of the universe, but requires sophisticated signal separation. While the astrophysics community has developed computational methods, the adoption of computer-vision methods for these tasks has been proposed by several groups. Results are difficult to compare, as the underlying datasets and evaluations are inconsistent and have not been made publicly available. We propose CMB-ML, a dataset and library that integrates dataset creation, model inference, and result evaluation into a pipeline. The library and links for data are available on GitHub at https://github.com/iccv-author-5412/cmb-ml.