This paper characterizes the viscoelastic response of asphalt matrix (AM) and hot-mix asphalt (HMA) mixtures under dry and moisture-damaged conditions. The AM and HMA mixtures were made with two types of aggregates and one asphalt grade. The hydrated lime was used as a binder additive to reduce the moisture sensitivity of the mixtures. A dynamic shear rheometer was used to conduct frequency sweep and creep tests at various temperatures with AM specimens. The indirect tensile load test was conducted to characterize the creep response, tensile strength, elastic and plastic, and fatigue properties of the HMA mixtures. The results indicated that the generalized creep compliance mechanical model can effectively characterize the viscoelastic response of the AM and HMA mixtures under both wet and dry conditions. The AM testing demonstrated that the lime modification significantly improves the viscoelastic properties of the moisture-damaged AM. The fatigue life of the HMA mixtures improved because of the decrease in the rate of accumulation of tensile plastic deformation as a result of the addition of hydrated lime to the mixtures.