Executive Summary

There is a concerted effort across the federal government to measure and track program performance. The development of an appropriate and manageable set of performance metrics is critical to the success of that effort. The Road Weather Management Program (RWMP) within the Office of Operations of the Federal Highway Administration (FHWA) at the U.S. Department of Transportation (USDOT) has been developing a set of performance metrics for its program since 2006. This report describes that effort and the results of applying metrics to evaluate its success against a set of goals established for the program.

Performance Measurement Goals

In 2005, the U.S. Congress passed the Safe, Accountable, Flexible, Efficient, Transportation Equity Act – A Legacy for Users (SAFETEA-LU) to fund the USDOT. Title V, Section 5308 of this act established a Road Weather Research and Development Program that has been managed by the RWMP. The RWMP’s activities and services are guided by a set of goals that are specified in Section 5308 of the current SAFETEA-LU legislation. Three significant goals are defined in the legislation to guide RWMP research and development:

1. Maximize use of available road weather information and technologies;
2. Expand road weather research and development efforts to enhance roadway safety, capacity, and efficiency while minimizing environmental impacts; and
3. Promote technology transfer of effective road weather scientific and technological advances.

Section 5308 specifically cited the National Research Council report Where the Weather Meets the Road¹ as a source for programmatic recommendations to help guide the work of the RWMP in meeting the SAFETEA-LU goals. The RWMP responds to as many of these programmatic recommendations as possible within the limits of the available resources. To measure and interpret the social, scientific, and organizational benefits that accrue from these RWMP projects, activities and services, a challenge is to identify and implement reasonable performance measures that track the attainment or progress towards the SAFETEA-LU goals.

Measurement Framework and Approach

The objective in creating a set of performance metrics for the RWMP is to understand the accomplishments and benefits of the program, its products, activities and services in terms of its success in meeting the three SAFETEA-LU goals. Most of these effects can be measured as direct consequences of the RWMP, but some that are attributable to the RWMP may also result from indirect effects channeled through other stakeholders. One of the main elements of the RWMP program has been to foster a collaborative research and applications agenda in the field of road weather management. Other federal agencies, state agencies and the private sector undertake activities independent of the RWMP that may affect progress toward achievement of the SAFETEA-LU goals. These activities can interact with the RWMP in a feedback loop that indirectly affects the performance of the RWMP’s programs and activities.

In order to more fully understand RWMP performance, it will be important to look at more than a snapshot of the data that reflect the program’s effect on the SAFETEA-LU goals. Progress toward goal achievement is expected to be incremental over time, and therefore, the selected metrics will need to be applied periodically, likely on an annual or bi-annual basis, to identify changes in RWM program performance.

The first step in identifying a manageable set of performance measures began with a literature review and a workshop with selected federal, state and private sector individuals knowledgeable about road weather and atmospheric weather, and experienced in developing performance metrics. A large list of candidate measures was widely distributed to over 250 national stakeholders who reviewed and commented upon the metrics, and offered their sense of program relevance and priority associated with each. This resulted in a refined short list of metrics that then were examined in greater detail, including more clearly defining each measure and specifying the data requirements to support each measure. The final report for Phase I describes the performance framework development process in more detail.²

These resulting 11 measures are shown in Table ES-1 under each of the 3 goal areas.

The RWMP addresses each of the three goals set forth under SAFETEA-LU by a combination of activities under four major objectives:

• Develop a national, open observing system that promotes data sharing to support weather observing and forecasting and transportation operations.
• Develop resources and training methods to assist state and local partners in deployment of weather management tools.
• Advance the state-of-the-practice by developing proactive solutions and disseminating information on adverse weather.
• Foster a collaborative, comprehensive, and dedicated surface transportation weather research program.

The RWMP has initiated several programs, projects and activities in the last five years. While each of these is at a different stage of project implementation, each has resulted in measurable outputs and outcomes for road weather management across the country. These accomplishments are expected to contribute directly to the attainment of the three SAFETEA-LU goals. In addition, the activities of the RWMP can serve as a catalyst to motivate and facilitate goal achievement through a variety of channels. For example, the RWMP sponsors workshops that inspire attendees to move in new directions with regard to more proactively incorporating weather into their own programs, and this in turn affects further progress toward achieving the SAFETEA-LU goals. The RWMP helps encourage the efforts of other federal agencies, state DOTs and private sector providers, and these other entities contribute to SAFTEA-LU goal achievement as well through their own independent initiatives and efforts.

RWMP Performance toward Achieving SAFETEA-LU Goals

Goal 1: Maximize use of available road weather information and technologies. The measures under Goal 1 primarily seek to address changes over time in the use of road weather information products and services. Deployment statistics data from 2004 to 2007 show increases in usage of road weather information in all aspects of operations.

• The number of states disseminating weather information to travelers has increased across various dissemination mechanisms from 2004 to 2007. In addition, the types of weather information disseminated have also substantially increased.
• A variety of control strategies were reported by states in 2007, ranging from 30 states reporting the use of technologies to manage diversions and closures during inclement weather to 5 states reporting the use of variable speed control during weather events.
• Since becoming a mature technology in 2005, the Maintenance Decision Support System (MDSS) has been utilized and tested by 30 agencies in 2008 in some manner with 5 agencies reporting operational use of an MDSS – a system intended to provide better information and decision-support to maintenance managers responsible for responding to winter weather conditions..
• Subscription to public and private road weather products and services has increased over the past three years.

The RWMP has had many direct and indirect contributions to the reported increase in use over the past three years by promoting the use of weather information in transportation operations around the country through a variety of activities including providing tools, conducting field tests and demonstrations, collaborating with private and public agencies for improved road weather information services, promoting best practices for weather information integration, weather-responsive traffic management, and development of the Clarus system and MDSS.

While increase in access and use has been clearly demonstrated, the quality and nature of use among agencies vary greatly and are uncertain from the deployment statistics. In other words, future performance measurement, in addition to tracking usage, needs to monitor quality of use when possible. The RWMP study on Baselining Current Road Weather Information³ can provide valuable information in this regard in addition to specific deployment evaluations.

Goal 2: Expand road weather research and development efforts to enhance roadway safety, capacity and efficiency while minimizing environmental impacts. The four measures under Goal 2 seek to quantify the extent to which agencies are engaging in activities that will enhance the safety, capacity, and economic advantages of addressing the impacts of adverse weather conditions on roads and travelers, and do that while also minimizing impacts to the environment. National level trends reflect positively on RWMP performance. Although many of the best practices and new technologies being promoted by the RWMP have only recently become available, state transportation agencies are eagerly adopting them. Best practices have existed prior to the RMWP, and the RWMP has sought to compile, promote and catalyze increased adoption and use of these techniques. Clear results are difficult to quantify in this early and selective nature of the deployments but the evidence to date suggests significant use and benefits. At the local levels of deployment, RWMP tools, products and services and the best practices that the RWMP supports have resulted in wide-ranging benefits in terms of safety, mobility, efficiency, productivity and customer satisfaction. Monitoring levels of use coupled with project evaluations clearly indicate that RWMP programs are having real benefits under this Goal 2.

Goal 3: Promote technology transfer of effective road weather scientific and technological advances. The four measures under Goal 3 seek to quantify the extent to which the RWMP has been able to engage a wide range of stakeholders and potential users of road weather technologies, communicate the benefits, promote usage, and successfully transfer operational tools and systems to states and agencies across the country. Measures of success include:

• Awareness and participation in the RWMP is very high among the states, and they report significant benefits from participation in the program.
• Participation in RWMP initiatives like Clarus and MDSS has continued to increase along with usage over the last four years.
• In-person and web-based training activities, road shows, and informational webinars sponsored by the RWMP are widely attended and appreciated.
• Close to 90 projects have been initiated in the area of Road Weather Management by private, public and academic groups that are involved with the RWMP.

Although indicated as the third goal of the RWM program, this goal reflects the first step in the process of raising awareness of the benefits of adopting RWM technologies and systems and directly contributes to the successes in Goal 1 and Goal 2. Much more progress along these lines can be expected in the future.

RWMP performance findings are summarized in Tables ES-2 to ES-4, and the available data have been collected either in direct support of each measure or indirectly through one or more indicators that are linked to the measure. The data vary in their ability to support the measure, and some of the indicators only offer weak linkage to their measure. These issues are discussed further in this report, and recommendations are made regarding future adjustments to the measures and indicators, as well as the need to identify new sources of data that can offer stronger support for the measures. Currently, the available data sources include the ITS-JPO Deployment Statistics and Benefit-Cost databases, RWMP program records, and transportation agency surveys and interviews conducted as part of this project.

The RWMP plays a variety of roles in its efforts to achieve the goals set for it under SAFETEA-LU, and these include initiating and implementing programs and projects that are designed to advance the frontiers of road weather research and development, providing training, engaging in partnerships and collaborative efforts, heightening awareness of the value of road weather information and its potential uses, and always seeking to encourage others to become more engaged in the application of road weather information and tools in support of advisory, control and treatment strategies in making our transportation systems safer and enhancing mobility. Both prior to the establishment of the RWMP, and since initiation of the range of activities and services described in this report, Federal, state and local agencies and the private sector have played, and will continue to play, a very significant role in the achievement of this program’s goals. While this report has sought to highlight the performance of the RWMP as illustrated by the findings shown in Tables 13-15, the attainment of the SAFETEA-LU goals is ultimately a widely collaborative effort in which the RWMP plays a leadership role.

Multiple measures and indicators, supported by a variety of relevant data sources are required to look at all available evidence in assessing the RWMP’s performance to date. This effort is only a first step to apply performance metrics in the RWMP. Applying these measures consistently over time will give a more complete picture of performance, particularly as the data improve and are collected more systematically for the purpose of supporting the metrics.

Conclusions

Performance measures offer a valuable tool to understand programmatic impacts, benefits, and potential for improvement. Stakeholders at the state and local level are eager to benefit from the activities, programs and services being offered by the FHWA Road Weather Management Program. At the same time, the RWMP is eager to see documented evidence that they are achieving the goals set out for the program and satisfying the needs of their customers. Reflecting early guidance from stakeholders, the eleven metrics examined in this paper are primarily directed at measuring the outcome benefits that the program is providing. But even the most carefully crafted performance measures can only approximate the concepts they seek to measure. The measures used to support the performance of the RWMP reflect a combination of quantifiable outputs (e.g., number of agencies that have acquired an MDSS, or the number of training programs conducted) and qualitative outcomes (e.g., the extent to which agencies are using MDSS more effectively throughout their jurisdiction, or the proactive incorporation of road weather information by transportation operators in decision making). In addition, other programs and factors can influence the measured outcomes, presenting a challenge to attribute observed data to the causal effects of the RWMP.

In addition, the responses of the state transportation agencies and stakeholders that are served by the RWMP offer another valuable indicator of program performance. State transportation agency stakeholders who were interviewed were asked to provide suggestions on how the RWMP could better support them. While a few respondents said they wanted more financial support from the RWMP, others sought more of what the program is already providing; namely, more opportunities to participate in programs such as the Clarus Initiative and MDSS, more long-distance learning opportunities and technology transfer given that states have severely constrained travel budgets, more direct engagement with the states in addition to channeling support through the private sector, and continued emphasis on ways to enhance and expand information flow and integrate weather into their operations. Overall, these stakeholders seemed pleased with the RWMP’s performance, with one respondent saying, “Keep doing what you’re doing.”

Based on an enhanced understanding of its past performance, the RWMP will be in a position to more effectively move the quality and benefits of the program forward. Even with this first application of the measures, clear progress toward attaining the SAFETEA-LU goals has been demonstrated, and the future forecast for improving performance of the RWMP appears bright.

¹ National Research Council. Where the Weather Meets the Road: A Research Agenda for Improving Road Weather Services, 2004.

² USDOT, FHWA. Road Weather Management Performance Metrics. Report No. FHWA-JPO-08-039, 2008.

³ FHWA. Baselining Current Road Weather. Report No. FHWA-JPO-09-055, 2009.