I. Introduction and Background

There is a concerted effort across the federal government to measure and track program performance. The development of an appropriate and manageable set of performance metrics is critical to the success of that effort. The Road Weather Management Program (RWMP), within the Office of Operations of the Federal Highway Administration (FHWA) at the U.S. Department of Transportation (USDOT), has been developing a set of performance metrics for its program since 2006. This report describes that effort and the results of applying metrics to measuring its performance against a set of goals established for the program.

Performance Measurement Goals

In 2005, the U.S. Congress passed the Safe, Accountable, Flexible, Efficient, Transportation Equity Act – A Legacy for Users (SAFETEA-LU) to fund the USDOT. Title V, Section 5308 of this act established a Road Weather Research and Development Program that has been managed by the RWMP. The RWMP’s activities and services are guided by a set of goals that are specified in Section 5308 of the current SAFETEA-LU legislation. Three significant goals are defined in the legislation to guide RWMP research and development:

1. Maximize use of available road weather information and technologies;
2. Expand road weather research and development efforts to enhance roadway safety, capacity, and efficiency while minimizing environmental impacts; and
3. Promote technology transfer of effective road weather scientific and technological advances.

Section 5308 specifically cited the National Research Council report Where the Weather Meets the Roadem⁴ as a source for programmatic recommendations to help guide the work of the RWMP in meeting the SAFETEA-LU goals. The RWMP responds to as many of these programmatic recommendations as possible within the limits of the available resources. To measure and interpret the social, scientific, and organizational benefits that accrue from these RWMP projects, activities and services, a challenge is to identify and implement reasonable performance measures that track the attainment or progress towards the SAFETEA-LU goals.

The RWMP contracted with a consultant team to identify potential output and outcome measures and select a reasonable and practical subset of those measures to quantify program performance. Eleven measures across the three goal areas were selected for implementation⁵. Data have been collected that reflect accomplishments through the implementation of a variety of RWMP activities undertaken in 2005-2009 to raise awareness, adoption and use of road weather information and technologies. These include, for example, the Clarus Initiative, Environmental Sensor Station Siting Guidelines, Maintenance Decision Support Systems (MDSS), Traffic Management Center Weather Integration, Vehicle Infrastructure Integration, and a number of other activities. These programs are intended to promote the use of road weather information in advisory, control, and treatment strategy decision-making supporting information to travelers, enhancing information and management systems for traffic operations, and improving weather-related decision support for highway maintenance activities.

The accomplishment and delivery of each of these program elements are intended to contribute to the attainment of the SAFETEA-LU goals. By measuring the progress towards the goal via the set of performance measures, and by attributing the impacts of the program elements to the attainment of these goals, the effectiveness of the RWMP can be assessed.

Measurement Framework

The objective in creating a set of performance metrics for the RWMP is to understand the accomplishments and benefits of the program, its products, activities and services in terms of its success in meeting the three SAFETEA-LU goals. Most of these effects can be measured as direct consequences of the RWMP, but some that are attributable to the RWMP may also result from indirect effects channeled through other stakeholders. One of the main RWMP program elements has been to foster a collaborative research and applications agenda in the field of road weather management. Other federal agencies, state agencies and the private sector undertake activities independent of the RWMP that may affect progress toward achievement of the SAFETEA-LU goals. These activities can interact with the RWMP in a feedback loop that indirectly affects the performance of the RWMP’s programs and activities. Thus, the framework for establishing viable performance metrics is complex, since the pathways by which the RWMP affects performance outcomes are themselves complex. Figure 1 illustrates these elements of the framework. It shows that, in order to more fully understand RWMP performance, it will be important to look at more than one snapshot of the data that reflect the program’s effect on the SAFETEA-LU goals. This will require a longitudinal assessment of how the RWMP’s programs and activities are causing changes in key metrics and indicators over time.

Progress toward goal achievement is expected to be incremental over time, and therefore, the selected metrics will need to be applied periodically, perhaps on an annual or bi-annual basis, to identify changes in RWM program performance.

Figure 1. RWMP Performance Measurement Framework

Performance in this study is measured in terms of the RWMP’s success in achieving each of the SAFETEA-LU goals. This is not an assessment of individual state Department of Transportation (DOT) performance in these areas. Each measure reflects strengths and weaknesses in terms of the efficiency and cost-effectiveness with which that measure can be operationalized, the data availability to support the measure, and the strength of the correlation of that measure with the goal performance it seeks to represent. The challenge presented by the presence of other external factors that also impact these goals is how to properly attribute attainment of the goal to the effects of the RWMP. An additional challenge is related to the early state of implementation of many of the RWMP’s program activities; that is, many state DOTs have not yet begun to adopt and implement these program components and those that have are typically only partially implementing components at this time. Hence, data are not expected to be available to measure full performance against these goals in many instances.

The identified performance measures are of two basic types. Output measures are more quantitative indicators of operational efficiency, such as tons of materials applied to a freezing road surface or the miles of roadway plowed over a period of time. Outcome measures represent impacts or benefits achieved from program activities that tend to be more qualitative and difficult to assess, such as the value of reductions in travel time or travel costs that can be attributed to the use of a decision-support tool. Outputs link most directly back to the inputs, while outcomes relate more to the programmatic goals. Both output and outcome measures are essential in assessing program performance. The metrics selected for use in the RWMP combine elements of both output and outcome measures.

Approach to Identifying and Refining Metrics

The steps followed to identify a manageable set of metrics for assessing the RWMP are illustrated in Figure 2. These steps follow the process through to institutionalizing and refining performance metrics in the program over time, thus interpreting performance assessment as a continuing and evolving activity.

The first step in identifying a manageable set of performance measures began with a literature review and a workshop with selected federal, state and private sector individuals knowledgeable about road weather and atmospheric weather, and experienced in developing performance metrics. A large list of candidate measures was widely distributed to a national set of stakeholders who reviewed and commented upon the metrics, and offered their sense of program relevance and priority associated with each. This resulted in a refined short list of metrics that then were examined in greater detail, including more clearly defining each measure and specifying the data requirements to support each measure.

Literature Review and Synthesis

The literature review included over 150 documents pertaining to measures being used throughout the federal government and private sectors, including the National Oceanic and Atmospheric Administration (NOAA) and the Federal Aviation Administration.

The literature review concentrated on: 1) literature describing evaluations of traveler information, including Advanced Traveler Information Systems (ATIS) and 511 services as well as Road Weather Management (RWM) systems and services, 2) literature describing methods of evaluating traveler information and RWM systems and services, 3) literature describing control and treatment strategies and the metrics appropriate to measuring performance associated with associated RWMP activities, and 4) reports on the “valuation of information,” as it applies to the economic benefits associated with RWM systems. Particular emphasis was placed on the outputs and outcomes of weather information on the following areas: traffic management (including traveler information), maintenance management, emergency management, transit management, transportation system performance, and driver performance.

The review examined the use of performance measures applied in other subject areas, for applicability to surface transportation weather programs. This included performance metrics currently used for products and services that are similar to RWMP products and services. Particular attention was placed on the NOAA National Weather Service’s (NWS) Assessment Program and the NOAA Regional Integrated Sciences and Assessments program (RISA), since that agency has a long history of performance metrics development as part of its activity assessments.

The literature identified Measures of Effectiveness (MOEs) used by organizations and programs to assess attainment of their organizations’ goals. The relationship of performance measures from similar programs and organizations was used to determine best practices, and the effectiveness of the performance measurement across programs and organizations was used to guide the selection of measures that would be likely to be useful for the RWMP.

Figure 2. Flowchart of the Identification and Application of Performance Measures

Similarities and differences employed for performance metrics development and application across different programs were examined as a way to identify metrics that might be applicable to the RWMP. The review showed that the use of performance measurement is a well-established practice at both the State and Federal levels with continued expansion underway. The theme that pervaded the literature was the importance of performance metrics associated with safety, effectiveness, efficiency, and customer satisfaction. In addition, the process of performance measurement addresses fundamental questions of conciseness, ease of data collection, ease of interpretation, cost effectiveness, and validity of findings. Consideration is given to ensure that the process developed aims to continually improve programs through constant examination of outcomes and avoiding the pitfall of examining outputs only for program evaluation.

Each of the publications, reports and articles reviewed discussed efforts to assess performance in terms that could be classified into one or more of the following RWMP objectives:

• Enhanced observation capabilities: extended availability of weather observations through increased spatial coverage, number of observed weather elements and quality of observed data.
• Advanced state-of-the-practice: improvements in the implementation and operation of technologies and methods.
• Coordinated research program: enhanced management of research activities within agencies and across multiple agencies.
• Training and outreach: increased awareness of new knowledge or technology and methods for more effectively using technology through education.

This illustrates substantial consistency with the RWMP and its guiding SAFETEA-LU goals and provides a useful pathway toward selecting candidate metrics that would be suitable for assessing the performance of the RWMP. The measures identified in the literature were mapped to these four outcome categories, which facilitated compilation of the initial list of performance measures to then assess more carefully against the needs of the RWMP.

Stakeholder Review and Refinement of Preliminary Measures

Initially about 120 output and outcome measures were identified through the literature review. Next a workshop was held at which the preliminary measures were presented and discussed with USDOT personnel and selected stakeholders to obtain feedback and refine the measures before distributing them more widely for additional comment. The workshop was held with members of the RMWP team, representatives from other modes of the USDOT, NOAA, State Departments of Transportation, representatives of the Institute of Transportation Engineers and other stakeholders. An initial list of the draft measures was provided to the participants prior to the workshop. The comments and changes made at the workshop led to a reduced and refined set of about 65 measures that were then circulated on-line in the form of a Request for Information (RFI) survey to over 250 public and private sector stakeholders nation-wide for comments and recommendations. Respondents were identified from State DOTs, weather and road weather working groups, NOAA, and other agencies and individuals known to have an interest or involvement in the RWMP. These respondents included providers and users of road weather information, such as State DOT policy, maintenance, and operations personnel; transit (all modes) policy; maintenance and operations personnel; transportation and meteorological professional organizations; and other organizations cited in Section 4308(b) of SAFETEA-LU. The providers included both public agencies, who consume road weather information for their internal use as well as for delivery to the public, and private sector road weather service providers. The road weather information users included both public and commercial travelers.

Respondents were asked to indicate whether they thought each measure was “good” as is, “fair” and needed improvement, or “poor” and should be eliminated. They were asked to provide comments if they felt the metric needed improvement or should be dropped. They were also given the opportunity to add any new metrics that they thought were missing from the list. Out of this process, eleven measures were selected based on their relevance to the RWMP, endorsement by stakeholders, data availability, and ease of implementation. These 11 measures are shown in Table 1 under each of the 3 goal areas.

Most of the eleven measures shown in Table 1 are multi-dimensional, that is, performance in the measure is defined not by a single number but by a matrix of indicators. In order to effectively operationalize several of these measures using available data, selected indicators were identified that sought to support these various dimensions. For example, Measure 1.1 has associated with it four separate indicators. In a few instances, a data source can directly support the measure, such as with Measures 2.1 and 2.3.

Chapter II summarizes the programs, activities and services of the RWMP and the outputs and outcomes it seeks to accomplish. Then, Chapters III, IV and V discuss how these RWMP activities are directed to the attainment of each of the three SAFETEA-LU goals. For each of the performance measures and supporting indicators, direct and indirect effects of these RWM program activities are identified and data presented to illustrate, or “track”, what that measure or indicator can tell us about program performance with regard to goal attainment.

Chapter VI provides an overview summary of the program performance that has been identified and tracked, and Chapter VII discusses the need to be forward-looking and the importance of conducting an annual assessment of changes in program performance over time in order to more fully capture the effectiveness of the RWMP in terms of goal attainment outcomes.

Finally, Chapter VII offers conclusions derived from the performance assessment conducted to date, fully recognizing that this is but a first step in a road weather program that in many respects has just begun to deploy its critical component project, activities and services.

⁴ National Research Council. 2004. Where the Weather Meets the Road: A Research Agenda for Improving Road Weather Services. Washington, D.C.: The National Academies Press.

⁵ U.S. Department of Transportation, Federal Highway Administration. (2008). Road Weather Management Performance Metrics. Report No.: FHWA-JPO-08-039. EDL No.: 14420. (April).