This paper advocates the importance of selecting appropriate measures of effectiveness (MOE) for evacuation planning and optimization purposes. In addition to evacuation time, which is often the choice MOE used by many practitioners, this paper also presents several increasingly more complex and useful MOEs that consider average evacuee travel time, delay, and temporal-spatial-based exposure risks. By studying a real-world transportation network in the event of a hypothetical nuclear power plant incident, the paper demonstrates that the superiority of one evacuation plan (or an optimization strategy) over others is highly dependent on the MOE selected for evaluation. Optimization efforts based on a single and ill-chosen MOE may not yield the best evacuation plan, as one might expect. The overall space-based or time- and space-based risk exposure may be increased if the optimization process is focused on travel time or some other time-based measure alone. Depending on the specifics of an emergency scenario (e.g., disaster type, risk components, and time constraints), an appropriate MOE should be chosen judiciously to make sure the optimization process for the evacuation plan is not wasted. This paper also demonstrates a multiobjective optimization approach for planning and designing an evacuation plan. With combined space-based risk and the travel time in route searching and traffic assignment, better destination assignment (and improved evacuation efficiency) may be obtained. The findings here should provide insights for future efforts toward assessing, improving, and optimizing evacuation plans.