Fatigue cracking, either top-down or bottom-up, is one of the major distresses for asphalt pavements. As the binding agent, asphalt binder plays a critical role in the fatigue resistance of hot-mix asphalt (HMA). Accurate characterization and proper selection of fatigue-resistant asphalt binders could prolong the fatigue life of asphalt pavement. The fatigue indicator currently used in the Superpave® binder specification is based on the fatigue behaviors of asphalt binders within the undamaged, linear viscoelastic range. There have been some controversies on the validity of this parameter. In reality, HMA and asphalt binders develop cracks under repeated traffic loads. Therefore, a more fundamental approach is needed to characterize the fatigue behaviors at a stage of damage to understand the true fatigue mechanism of the asphalt binder. This paper presents the study of the application of viscoelastic continuum damage (VECD) mechanics, which has been successfully applied to HMA, to test results of asphalt binders in the shear mode under various loading conditions. The results indicated that VECD can be effectively applied to asphalt binders, present a good potential for characterizing asphalt binders and predict the contribution to fatigue resistance by means of a fundamental approach that parallels some of the advanced work regarding asphalt mixtures. This potential could allow for a unified model for fatigue of mixtures and binders.