Developing reliable defenses against patch attacks for object detectors has attracted increasing interest.However, we identify that existing defense evaluations lack a unified and comprehensive framework, causing inconsistent and incomplete assessment of current methods.To address this issue, we revisit 10 representative defenses and present the first large-scale benchmark, involving 2 attack goals, 13 patch attacks, 11 object detectors, and 4 diverse metrics.This leads to the first large-scale adversarial patch dataset with 94 types of patches and 94,000 images, which can also be used to improve existing defenses. We conduct comprehensive analyses to reveal new insights: (1) The difficulty in defending against naturalistic patches lies in the data distribution, rather than the commonly believed high frequencies. In light of this, we construct a large-scale dataset with diverse patch distributions to obtain stronger defenses, with 15.09\% AP@0.5 improvement.(2) A higher patch detection accuracy does not necessarily imply better defense performance.Instead, the average precision of the attacked object shows higher consistency.(3) Existing defenses can be substantially bypassed by adaptive attacks, and defenses that integrate complex/stochastic models or patch-level features are less vulnerable.We will open-source our dataset and code as well as keep integrating new attacks/defenses.