The inadequacy of the existing ASTM E950 method for definition of the bias of an inertial profiler in the measurement of a pavement profile is analyzed. An improved bias calculation method is proposed. An analysis of the profile measurements and international roughness indexes (IRIs) of four pavement locations was conducted, and the results show that there is no statistical correlation between the profile biases and IRI bias. This analysis demonstrates that a profiler classified as having high accuracy in measuring profiles by ASTM E950 or other proposed bias criteria does not necessarily provide an accurate measurement of IRI. Therefore, it is recommended that the accuracy of a profiler in the measurement of IRI be evaluated independently when IRI, instead of profile, is used for a smoothness specification or other purposes. A criterion to evaluate profilers' accuracy to measure IRI is developed based on the concept of probability.