As the framework described in Goal 1, 2 and 3 show, multiple measures and indicators, supported by a variety of relevant data sources are required to look at all available evidence in assessing the RWMP’s performance to date. This is only the first step for the application of performance metrics in the RWMP. Applying these measures consistently over time will give a more complete picture of performance, particularly as the data improve and are collected more systematically with the purpose of supporting the metrics.
The framework takes advantage of many of the existing sources of data collection, minimizing the data collection costs to the program. Several sources of data are particularly important to the program
- ITS Deployment Statistics. To continue monitoring the performance of the RWMP, it is critical that the program work with the ITS-JPO to use the ITS Deployment Survey as a primary data source for quantifying the level of use and adoption of RWMP supported programs, services and activities. Having relevant questions included in the continuing national surveys can provide the RWMP with a majority of the data for the indicators identified particularly on the level of use of RWMP products, services and supported strategies.
- Project Evaluations and Benefits Database. Of more importance from an RWMP performance standpoint is evaluation of strategies that the RWMP directly or indirectly affects and monitoring the benefits and the levels of use. Many of the RWM programs are at research stages or being deployed by a few sites nationwide and are being evaluated. These evaluations need to be compiled as part of the performance measurement. The ITS-Benefits Database maintained by the ITS-JPO is a useful repository for compiling evaluation results.
- Data from the RWMP. These program data are the best source for many of the indicators identified. This is especially true for the output type measures in Goals 1 and 3 regarding participation of state agencies in RWMP activities. The RWMP should continue to collect information on participation but also have an evaluation component as part of their outreach activities focusing on quality and benefits that the stakeholders are deriving from participation in RWMP activities.
- Focused RWMP sponsored surveys/interviews of state DOTs. Such survey can capture the awareness, acceptance and use of specific RWMP products and activities at a level of detail not possible by the more general ITS Deployment Statistics surveys. By creatively framing questions, the survey can be used to obtain a detailed assessment of specific programs. An example of such a survey is the Baseline Weather Information Surveys carried out by the RWMP that focused on the quality attributes of weather information. The baselining work will continue every two years and will provide information relating changes in the quality characteristics of ESS to road weather, including usage by transportation agencies. The biggest challenges with the online survey are lack of adequate responses, and the identification of the survey participants. The lack of adequate responses is more critical especially since several measures require national summaries.
Ideally, performance measurement will be carried out annually or bi-annually, focusing on improvements that can be assessed against this baseline. By maintaining consistency in the assessment of performance across the years, a more complete picture of RWMP performance can be obtained. It is also possible that newer indicators emerge to show additional aspects of progress toward the goal, and these will need to be included as part of the performance measurement.
