ITS - Intelligent Transportation Systems Report ITS Home Page

Final Report

Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System



1.0 Introduction

1.1 Purpose of this Document

This document presents the findings of the national evaluation of the 511 telephone traveler information system "Model Deployment" led by the Arizona Department of Transportation (ADOT). The United States Department of Transportation (U.S. DOT) National 511 Model Deployment supported a wide range of enhancements to the existing ADOT 511 system, including the addition of several new types of information and significant redesign of the user interface.

On July 21, 2000, the Federal Communications Commission assigned 511 as the nationwide traveler information telephone number and granted responsibility for it to government transportation agencies. Since that time, approximately 27 statewide and regional 511 systems have been implemented throughout the United States, including the current Arizona statewide 511 system.

The Federal Highway Administration (FHWA) and Federal Transit Administration (FTA) jointly sponsored the Arizona 511 Model Deployment. The Model Deployment sought to demonstrate the potential of 511 services to bring together data from various sources and provide useful information to travelers through a state-of-the-art telephone interface. The national evaluation assessed the extent to which the Arizona 511 project satisfied the objectives of the Model Deployment, which were to:1

The national evaluation of the 511 Model Deployment is intended to generate findings that will help shape U.S. DOT approaches to 511 and that will be of use to others implementing and operating 511 systems. The national evaluation is the primary mechanism for documenting the performance of the Model Deployment and the lessons learned. This report presents an overview of the Model Deployment, the evaluation approach, and the findings of the evaluation.

1.2 Overview of the Model Deployment

The Arizona Department of Transportation (ADOT) led the 511 Model Deployment in partnership with other transportation agencies in the Phoenix and Tucson regions. The 511 Model Deployment implemented a number of key enhancements to the previous statewide 511 system. That prior system became operational in March 2002 when ADOT converted their ten-digit telephone information system, which had been operational for several years, to 511. The planned enhancements to the 511 system included:

ADOT received the award of the Model Deployment in July 2002 and immediately began the planning and design process. Starting in August of that year, meetings of the 511 Task Force (the name of the 511 Model Deployment committee) were held periodically. A draft Program Management Plan document was completed in November 2002 that includes an overall schedule and high-level scopes of work for the various enhancements. At that same time, a draft System Requirements Document was completed that identifies the requirements for the various enhancements. Implementation of the system began in early 2003. The enhanced 511 system was rolled out in mid-December 2003, with many of the planned system enhancements included, such as several new data types, a complete redesign of the menu system, and implementation of voice recognition. The one-year Model Deployment operational period extended from January through December 2004.

Battelle Memorial Institute is conducting the national Model Deployment evaluation. Battelle was given notice to proceed in September 2002. Battelle is working in partnership with the local evaluator, Dr. Mark Hickman, Assistant Professor at the University of Arizona. Dr. Hickman is consulting to ADOT, providing the required local evaluation of the Model Deployment.

1.3 Overview of the Evaluation Approach

1.3.1 Evaluation Objectives

Nationally, implementation of 511 is still in its early stages. Twenty-seven 511 systems have been set up in the U.S. to date, but many other locations are still considering or planning 511 deployments. Clear "best practices" have yet to emerge in a number of technical and institutional areas. Key questions include how much transit information should be made available directly on 511 versus transferring callers to existing transit agency customer information systems; how to interface with other states' 511 systems in border regions; how much of the cost of 511 should public agencies pay and what sorts of private sector partnerships may be possible; and how to accommodate additional information while keeping menu systems easy to use. The Model Deployment and this evaluation are intended to address these questions.

Specific objectives of the U.S. DOT evaluation of the 511 Model Deployment are to:

1.3.2 Evaluation Phasing

The evaluation was conducted in three phases. Table 1-1 identifies the phases, the associated time frames, and the major milestones associated with each evaluation phase. Phase I consisted of the evaluation planning and collection and analysis of baseline (pre-enhancement) data. Phase II consisted of the analysis of the Model Deployment enhancement process and included participation in the local 511 project meetings, two rounds of stakeholder interviews and two rounds of lessons-learned stakeholder workshops. Phase III consisted of the collection and analysis of post-enhancement data, covering a full year of post-enhancement 511 operations. Phase III data collection included a variety of 511 system data collected on a monthly basis (call volumes, phone bills, phone system server log files, etc.) and a survey of 511 users.

Table 1-1. Evaluation Phasing and Milestones
Evaluation Phase Dates
Start		 Dates
End		 Milestones
Activity/Deliverable		 Milestones
Date
Phase I – Planning and Baseline Analysis August 2002 February 2004 Evaluation Plan October 2003
Phase I – Planning and Baseline Analysis August 2002 February 2004 Detailed Test Plan October 2004
Phase I – Planning and Baseline Analysis August 2002 February 2004 Interim Analysis Report February 2004
Phase II – Enhancement Process Analysis August 2002 February 2005 First Round Stakeholder Interviews December 2003 – January 2004
Phase II – Enhancement Process Analysis August 2002 February 2005 First Round Lessons Learned Workshop February 2004
Phase II – Enhancement Process Analysis August 2002 February 2005 FHWA Interim Enhancement Process Briefing February 2004
Phase II – Enhancement Process Analysis August 2002 February 2005 Second Round Stakeholder Interviews December 2004 – January 2005
Phase II – Enhancement Process Analysis August 2002 February 2005 Second Round Lessons Learned Workshop February 2005
Phase III – Post Enhancement Data Analysis January 2004 May 2005 One-Year Enhanced 511 Operations Period January 2004 – December 2004
Phase III – Post Enhancement Data Analysis January 2004 May 2005 User Survey October-November 2004
Phase III – Post Enhancement Data Analysis January 2004 May 2005 Complete Post-Enhancement Data Collection January 2005
Phase III – Post Enhancement Data Analysis January 2004 May 2005 Draft Evaluation Report March 31, 2005
Phase III – Post Enhancement Data Analysis January 2004 May 2005 Final Evaluation Report May 31, 2005

1.3.3 Model Deployment Objectives and Hypotheses

In addition to capturing information on deployment and management issues, lessons learned, and costs, the evaluation attempted to test a number of specific hypotheses. The hypotheses were developed by the evaluation team based on the planned 511 enhancements. The hypotheses fall into three of the National ITS Goal Areas: customer satisfaction, mobility, and efficiency. The hypotheses that were planned for testing are listed below. Key hypotheses are shown in bold type.

Usage Hypotheses
Customer Satisfaction Hypotheses
Mobility Hypotheses
Efficiency Hypotheses

1.3.4 Evaluation Analyses

Evaluation analyses were developed to test the hypotheses presented in Section 1.3.3 and to support the analysis of the enhancement process (e.g., lessons learned). The analyses are organized into individual "tests," with each test focusing on particular types of data. The tests and their relationship to the major focal points of national Intelligent Transportation System (of which 511 systems are an example) evaluations are shown in Table 1-2. In addition to these formal tests, a cost analysis was performed and various other types of supporting data were collected and analyzed to aid in the interpretation of test results and to identify management and deployment issues and lessons learned.

Table 1-2. Evaluation Tests
Analysis Area Tests
Usage (1)		 Tests
User Survey		 Tests
Stakeholder Interviews
Customer Satisfaction X X  
Mobility X X  
Efficiency X X X
Management and Deployment Issues     X
(1) The analysis of system usage included consideration of system data
content (inputs from the various agencies that supply data to the system)
and the reliability (e.g., system downtime) and availability (phone line utilization) of the system.

Table 1-3 identifies the general division of responsibilities for the evaluation activities between Battelle and the Local Evaluator, Dr. Mark Hickman of the University of Arizona. For each test, activities were very closely coordinated between Battelle and Dr. Hickman to ensure that all evaluation needs were met.

Table 1-3. Evaluation Lead and Support Roles by Evaluation Activity
Evaluation Team Member Usage Logs Test User Survey Test Stakeholder Interviews Test Cost Analysis
Battelle Support Lead Lead Lead
University of Arizona Lead Support Support Support

The usage analysis consists of three main components. The primary component of the test is the analysis of 511 system usage data based on data gathered from phone system server log files and, in the case of the baseline analysis, electronic phone bill records. The secondary component consists of an analysis of the performance of the 511 system in terms of system downtime and phone line availability/utilization.

The user survey featured a post-enhancement survey of 511 users, conducted in late October and early November 2004, about 9 months after the roll out of the enhanced 511 system. A percentage of calls into the 511 system were intercepted and callers were recruited to participate in a brief survey, which was administered later in a separate call to users who agreed to be surveyed. Approximately 400 completed surveys were obtained. The survey included questions focusing on users' utilization of the system (frequency of use, what sorts of trips, at what stage in the trip, cell phone versus landline, etc.), their satisfaction with the system (ratings of information types, ratings of accuracy/timeliness, ratings of overall satisfaction, perception of benefits, favorite and least favorite aspects, etc.), and recommended improvements. A decision was made not to undertake a comparison of the enhanced service with the original service because the service had changed substantially and because of the impracticality of trying to find a sufficient number of users of both versions of the 511 service. Thus, the objective of surveying users was to measure their satisfaction with the enhanced system.

The stakeholder interviews were conducted in two rounds. The first occurred in January 2004, shortly after the roll out of the enhanced 511 system. The second round occurred in January 2005, just after the end of the one-year Model Deployment operational period. The interviews were a primary mechanism for collecting lessons learned and other data related to the enhancement process. Interview results were also used in the testing of efficiency-related hypotheses and to support the analysis of costs.

1.4 Scope and Organization of the Final Report

This report presents the results of the national evaluation of the 511 Model Deployment along with context-setting information on the evaluation approach (this section), the Arizona context for traveler information (Section 2.1), and a description of the specific Model Deployment enhancements as planned, and as implemented and available for evaluation (Section 2.2). Section 3.0 describes the data collection approaches associated with the evaluation analyses.

The results of the usage analysis and user survey are presented in Sections 4.0 and 5.0, respectively. Section 6.0 summarizes the results of the hypothesis testing, drawing upon all of the evaluation analyses. Sections 7.0 and 8.0 present the results of the enhancement process and cost analyses, respectively.

The final section, Section 9.0, summarizes major findings and lessons learned. Section 9.0 also presents a number of recommendations for consideration by 511 deployers.


1 511 Model Deployment Solicitation, Federal Register: January 16, 2002 (Volume 67, Number 11), Federal Highway Administration, Federal Transit Administration.

< Executive Summary | Table of Contents | 2.0 >


FHWA Home | Feedback
FHWA
United States Department of Transportation - Federal Highway Administration