Final Report
Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System
Multi-Modal 511 Traveler Information System
Executive Summary
This document presents the findings of the national evaluation of the 511 telephone traveler information system "Model Deployment" in Arizona. The United States Department of Transportation (U.S. DOT) National 511 Model Deployment supported a wide range of enhancements to the existing statewide Arizona 511 system, including the addition of several new types of information and significant redesign of the user interface.
On July 21, 2000, the Federal Communications Commission assigned 511 as the nationwide traveler information telephone number and granted responsibility for it to government transportation agencies. Since that time, 27 statewide and regional 511 systems have been implemented throughout the United States, including the Arizona system.
The Federal Highway Administration (FHWA) and Federal Transit Administration (FTA) jointly sponsored the Arizona 511 Model Deployment. The Model Deployment sought to demonstrate the potential of 511 services to bring together data from various sources and provide useful information to travelers through a state-of-the-art telephone interface. The national evaluation documented costs and lessons learned, measured the accomplishment of the implementers' hypothesized project impacts, and assessed the extent to which the Arizona 511 project satisfied the following U.S. DOT objectives for the Model Deployment:
- "Push the envelope" on the production and dissemination of quality traveler information.
- Utilize a voice recognition user interface and a menu that balances comprehensive information with ease-of-use.
- Provide information automatically, with no operator needed, and provide users the ability to request information for specific segments of roadways, such as "I-10 in Phoenix" rather than providing them information only at the less specific "route level", that is, "I-10" for the entire state.
- At a minimum, provide a comprehensive set of basic, multi-modal traveler information including roadway congestion, incidents and construction; major transit service disruptions; special events; and abnormal road weather conditions.
Key Recommendations to 511 Deployers
A list of lessons learned is presented later in this Executive Summary and discussed in greater detail in the body of the report. This section highlights the most significant of those lessons learned in the form of recommendations to 511 deployers. The recommendations are as follows:
- If specific, near-term impacts are important, prioritize and focus your 511 implementation. The Arizona Model Deployment consisted of a very broad, ambitious slate of enhancements, none of which were explicitly identified as more or less important. The Model Deployment approach reflected ADOT's traditional "evolutionary-opportunistic" approach to incrementally enhancing their 511 system with the long-term objective of a generally improved system. 511 deployers who have specific performance levels in mind or who are trying to effect specific changes in 511 usage or travel behavior are strongly encouraged to utilize a directed, "problem-solving" approach. Such an approach would identify a small, manageable and measurable list of specific objectives and then concentrate resources in those few areas.
- Voice recognition is not "plug and play"; expect to spend considerable resources on its implementation. Although voice recognition technology is widely deployed, its successful application to 511 systems is challenging. Statewide, multi-modal 511 systems strain the limits of the technology due to the very large number of potential utterances that the system must detect. Although basic implementation of a voice recognition system is relatively straightforward, troubleshooting, modifying, and testing the system to achieve acceptable performance are very resource intensive. Because of the complicated nature of voice recognition and the changes in technology, agencies that do not have up-to-date technical expertise in-house may consider "turn-key" firms that specialize in voice recognition services. Deployers should expect to devote considerable attention to the testing and refinement of a 511 voice recognition system before it goes on-line.
- Pursue a vigorous, targeted 511 marketing program. Marketing is essential to building awareness and stimulating usage of 511 systems. This Model Deployment demonstrated that dynamic message signs as part of a marketing campaign are highly effective in reaching en-route travelers. It is especially important to market any significant new features or information content enhancements to existing 511 systems so that users know about them. This Model Deployment indicated that simply adding new features and information is not sufficient to stimulate usage. Targeted marketing to the user subgroups who would be interested in the new information is especially important.
- In the early planning stages, thoroughly investigate institutional and technical issues associated with data input by partner agencies. Many of the Model Deployment enhancements that were not completed or not fully successful depended on data entry by new 511 partners. Several of these agencies committed to this role in principle during the very early stages of the Model Deployment (during preparation of the funding proposal) but for various reasons did not fulfill that role. If dependent on key data from other agencies, 511 deployers are strongly encouraged to thoroughly investigate the issues that could impact the ability of partners to deliver data as envisioned and to resolve key issues before moving forward.
- Incorporate mechanisms for capturing user feedback for system evaluation. Because many of the benefits of traveler information systems such as 511 are qualitative, it is important to be able to capture users' experiences as a way of evaluating the system's performance. Although ADOT was initially reluctant to interrupt callers as part of a user survey process, the evaluators and system designers were able to develop and implement a procedure that minimally inconvenienced callers, yet permitted capturing valuable information about the ways that users interacted with the system. This information not only revealed how callers were using the system but also where improvements were needed to make the service easier to use and provide better travel information.
Description of the Model Deployment
The Model Deployment was led by the Arizona Department of Transportation (ADOT) and assisted by a 511 Task Force composed of Tucson and Phoenix area Metropolitan Planning Organizations, airport and transit operators, and other county and municipal transportation agencies. The enhanced 511 system became operational in mid-December 2003.
Table ES-1 identifies the 18 planned Model Deployment enhancements and their operational status relative to the one-year, January – December 2004, post-deployment evaluation period. The enhancements fall into four general categories:
- New information content including arterial streets, airports, etc.
- A redesigned user interface, including conversion from touch-tone (keypad) input to voice recognition and from user selection of roadway information by entire roadway (e.g., "I-10") to roadway segments (e.g., "I-10 in Phoenix").
- Enhanced 511 marketing consisting of advertisements on ADOT dynamic message signs (DMS) statewide, printed materials, radio advertisements, and 50 static roadside signs deployed statewide.
- Partnership with a commercial information provider for fee-based, premium 511 service.
As noted in Table ES-1, 9 of the planned 18 Model Deployment enhancements were deemed by the Evaluation Team to be fully operational and suitable for evaluation within the evaluation period. None of the planned 18 Model Deployment enhancements were considered by the deployers to be more critical than any others, and there was, therefore, no explicit prioritization of the enhancements. However, when it became clear that not all of the enhancements could be implemented concurrently, efforts were focused on the basic user interface enhancements—voice and segment-based road information—which suggests that these may have been viewed as more important. Work continues on all of the unimplemented enhancements with the exception of the partnership with a private premium service provider, which has been dropped. No qualified private partner responded to the ADOT request for partnership proposals.
| Enhancement | Fully Complete & Operational within the Evaluation Period | Completed & Operational After the Evaluation Period | Not Currently Complete & Fully Operational | Comments |
|---|---|---|---|---|
| Information Content | ||||
| Tucson and Phoenix Local Street Data | ADOT's plan to increase their capture of arterial street incidents through more intensive monitoring of law enforcement scanners was carried out. However, inputs by cities and counties have been negligible. Inputs by the Tucson region are pending (training has been completed as have necessary map updates); Phoenix area agencies are making a limited number of inputs. | |||
| Transit Major Service Disruptions and Call Transfer Capability | The original plan called only for Phoenix and Tucson Transit, but the menu system and call transfer capability have been extended to include all regional/rural transit providers statewide. | |||
| Phoenix Bus Rapid Transit Estimated Arrival Times | Transit currently unable to provide data. Enhancement not implemented and unavailable for evaluation. | |||
| Phoenix Arterial Street Travel Times | Data collection system completed and now being tested; not available for evaluation. | |||
| Data Quality Enhancements | Enhanced 511 data entry operator training and 511 message preview function to see how entries will be conveyed on 511. | |||
| Phoenix and Tucson Airport Information | Phoenix has used the 2-minute message recording capability but Tucson has not. | |||
| Grand Canyon National Park Information | All necessary technical elements are in place (menu changes and voice-recording capability). During the evaluation period, the Park was not entering any information but now intends to. | |||
| Arizona Office of Tourism Call Transfer | An unplanned enhancement requested by the Office of Tourism. | |||
| Segment Weather Information | Came on-line after completion of evaluation. | |||
| Downtown Phoenix Special Events and Parking | There has been no discernable change in data capture techniques or the type and volume of information in the system. | |||
| Sharing Data with Other States (i.e., Utah) | Under development. Not available for evaluation. | |||
| User Interface | ||||
| Regional Roadway "Quick Reports" | ||||
| Roadway Segment-Based Reporting | ||||
| Roadway Info. Accessible Via Roadway Name | ||||
| Voice Recognition | ||||
| Other | ||||
| Premium Service Partnership | No viable private partnership proposal was received. | |||
| Marketing | The only incomplete elements are the 50 static road signs throughout the state, which are being installed now. | |||
| Improved System Performance Monitoring | Not all data analysis and reporting functions have yet been utilized but many new capabilities have been established and many of them have been used by ADOT. | |||
Evaluation Approach
The evaluation was conducted on behalf of the FHWA by Battelle Memorial Institute in partnership with the University of Arizona. Volpe National Transportation Systems Center served as Evaluation Program Manager on behalf of the FHWA ITS Joint Program Office. The evaluation included four major analyses:
- Usage – Several types of system data, including detailed computer logs of each call into the 511 system, were used to analyze the following usage parameters: call volumes and patterns, including during several special events (e.g., wildfire, winter storm, the DMS ad campaign, and month encompassing the conversion to the enhanced system); call frequency; call durations; wire line vs. cell phone calls; caller geographic location; menu selections; call transfers; system capacity; and unrecognized caller inputs. Data were analyzed over a one-year period and, where baseline data allowed, pre- and post-enhancement data were compared.
- User Survey – 411 users of the 511 system were surveyed to gauge their satisfaction with the enhanced system and reaction to specific features. Users were intercepted by a live survey recruiter during their call to the 511 system and the interview was conducted in a separate, follow-up phone call.
- Enhancement Process – Lessons learned associated with the design, implementation, and operation of the enhanced 511 system were developed through interviews and workshops with 511 Task Force members.
- Costs – Detailed cost information was collected for the baseline and enhanced 511 system, categorized by phase (i.e., planning, implementation and operation), type (i.e., hardware and software/engineering), and enhancement. The total capital (i.e., non-recurring) cost to enhance the system was about $1.4 million, much greater than the approximately $355,000 investment to establish the pre-enhanced system. Operations and maintenance costs for the first post-enhanced year were about $293,000, compared to the pre-enhanced $140,000.
Results
Overall, the Model Deployment succeeded in dramatically improving the Arizona 511 system, although it was not fully successful in a number of respects. Through both the accomplishments and shortcomings of the Model Deployment, a number of insights have been gained that will be useful to ADOT and their 511 partners as well as other 511 deployers around the country. This section summarizes major accomplishments, challenges, lessons learned, and findings regarding hypothesized project impacts.
Major Accomplishments
Major accomplishments of the Model Deployment include:
- Conversion to a voice recognition user interface; touchtone remains an option.
- Conversion to segment-based roadway reporting.
- In response to user requests, addition of regional roadway "Quick Report" congestion and incident summaries.
- Addition of call transfer options to airports, the Arizona Office of Tourism, and all of Arizona's rural/regional transit operators.
- The addition of a 2-minute voice recording capability for transit and airport operators for providing announcements via 511.
- Significant enhancement of 511 coverage, to full statewide coverage.
- No significant 511 system down-time as enhancements were implemented.
In addition to these system enhancements, the Model Deployment facilitated a number of other broader benefits, including:
- Solidification of a 511 philosophy of customer service and continuous quality improvement at ADOT, including many new tools to support on-going system performance monitoring (detailed system data reports) and customer service (e.g., caller comment line).
- Implementation of the first formal Arizona 511 marketing campaign, including a very effective week-long, 24 hour-a-day dynamic message sign (DMS) advertising (Figure ES-1).
- By demonstrating a long-term commitment to the 511 system and establishing it as a truly multi-modal, statewide source of integrated, multi-agency information, the 511 Task Force has created a focal point for, and invigorated, multi-agency ITS operations coordination. Most notably, the Model Deployment resulted in a major increase in the time and attention that the ADOT Traffic Operations Center Information Technologies (IT) Manager—who essentially served as Deputy Model Deployment Project Manager—devotes to the 511 system. One of the most important parts of this greatly expanded role and attention is that the IT Manager now regularly attends the multi-agency AZTech (i.e., Phoenix region ITS) Advanced Traveler Information Systems (ATIS) Committee meetings and his involvement has reinvigorated regional traveler information coordination efforts.

Figure ES-1. 511 Marketing via Dynamic Message Sign
In several important respects, the Model Deployment was well received by users. About 70% of surveyed callers were satisfied with information content. When interpreting user feedback, it is important to note that the Arizona 511 user base is changing rapidly. Call volumes during the one-year evaluation period increased 74% over the prior year (Figure ES-2) and much of the increase was from new 511 users. The week-long 511 DMS advertising was very effective in stimulating calls to the system. Daily call volumes increased three-fold during that time and just over one-third of surveyed callers reported seeing the DMS advertisements (note the May 2004 spike in Figure ES-2). By the time of the caller survey almost six months later, one-third of surveyed users were new callers, that is, the 511 call during which they were intercepted for survey recruitment was their first ever call to Arizona 511 (Figure ES-3).
Comparing the experience of new callers to 511 with that of previous callers provides useful insight into how well the service is performing and how likely it is to retain callers over time. It should be noted that 511 is not heavily utilized even by most repeat callers, among whom half reported that they made three or fewer calls in the last month and virtually all of them were seeking road information. Within that context repeat callers report high levels of satisfaction with the information they receive and cite several specific benefits in using the 511 system; for example, saving time/arriving on time and avoiding traffic congestion due to accidents or construction delays were each cited by about 20% of repeat callers. While most first-time callers also expressed overall satisfaction with the service, they were less likely to articulate specific benefits than were repeat callers and, indeed, 31% said they received no benefit from using 511 the first time they called. In addition, when asked about improvements to 511 that they would recommend, first-time callers tended to want more types of improvements than did repeat callers. On the other hand, repeat callers placed a higher premium on two specific areas for improvement: speech recognition and more details on traffic congestion and delays. While 95% of first-time callers said they would use the service again, surely on future calls to 511 the service must begin to deliver clear benefits to them for them to be converted to regular users.

Figure ES-2. Monthly Call Volumes

Figure ES-3. Percent of Callers by How Long Ago They First Used 511 (n=404)
Challenges
The major challenges or shortcomings of the Model Deployment are that many of the planned enhancements were not completed within the evaluation period (several of these have since been completed), some 511 partner agencies have not taken significant advantage of their new capabilities to provide information via 511, new information types are not being used significantly by most users, and many users are dissatisfied with the performance of the voice recognition system.
As indicated in Table ES-1, half of the 18 planned enhancements were not fully operational in time for the evaluation, or were not being utilized in a manner that would support evaluation. With the exception of the private partnership, which was cancelled, efforts are continuing on all of the unimplemented enhancements. Delays resulted from a variety of factors not uncommon to complex technology implementations, including assorted technical challenges and limited agency staff resources. That is, ADOT found that there was a limit to how many enhancements they could simultaneously pursue.
Several enhancements were only partially implemented. In a few of these cases, ADOT added the capability for data entry but the partner agency did not, during the evaluation period, take advantage of it. This was the case with the Grand Canyon National Park, Tucson airport, and Phoenix-area municipal traffic information enhancements. In the case of municipal traffic agencies, they indicated that they lack the senior agency support necessary to devote resources to collecting and entering this information into 511.
With regard to user reactions to the Model Deployment, there are two primary concerns. First, very few users consulted the new information available on the 511 system (i.e., tourism, airports, and roadway Quick Reports). Combined, these items accounted for less than 8% of all menu selections over the one-year evaluation period. The vast majority of menu selections—92%—were for the two basic types of pre-enhancement information, roadways and transit. Of those selections, nearly all (91%) were for roadways. These findings are at least in part a result of the fact that, during the evaluation year, little 511 marketing promoted the new types of information to travelers who would have been interested, and, therefore, most were presumably unaware of its availability. They may also be related to the quality or depth of the information available on 511, although very small sample sizes for users of these non-roadway options prevented definitive conclusions. It may also relate to the availability of other options for non-roadway information. In the case of transit, the low utilization relative to roadways reflects very low transit usage—1.2% of all person trips in the Phoenix region.
The second concern from a user perspective is the relatively high levels of dissatisfaction with the performance of the voice recognition feature. About 35% of repeat callers who choose to use voice recognition as their primary means for accessing the service said they were dissatisfied with the feature, and 49% of new callers were dissatisfied. The difficulties with voice recognition are reported on both cell and landline phones and both while holding a receiver and speaking hands free. The performance of the voice recognition system is clearly not acceptable to a large number of users. System data provide support to the perception of voice recognition problems. Over the one-year operational period, 37% of all calls included at least one user input that could not be interpreted by the system. That number was 58% during the first month of voice recognition operation. The voice recognition feature was the most commonly recommended area for improvement among both new and repeat callers.
Lessons Learned
The major lessons learned of the Model Deployment consist of the following:
- Invest in formal planning and design documents (e.g., system requirements). ADOT indicated that the significant efforts that went into developing the Program Plan and System Requirements Definition documents during the design phase were instrumental to the successes of the Model Deployment.
- Solicit user input during design and system refinement. Prior to the Model Deployment ADOT had very little input from 511 users. As part of the Model Deployment design effort, several focus groups were conducted with travelers. ADOT found these focus groups very illuminating and one specific enhancement—the regional roadway Quick Reports—came about based directly on focus group input. During the first few months of enhanced system operation 511 caller comments were very useful in identifying and correcting voice recognition system problems.
- Plan for voice recognition development and refinement to be resource intensive. A great deal of effort, significantly more than anticipated, was devoted to the voice recognition implementation. The costs for this enhancement, about $560,000, were 61% higher than estimated. The vast number of individual user utterances to which the system must respond, the many Spanish and Native American place name location references, and the significant revision and expansion of the transit portion of the menu all contributed to the magnitude of the voice recognition implementation and refinement effort.
- Conduct marketing targeted to potential users of new information types. Even with a large percentage of new users each month, as is the case with the Arizona 511 system, most users do not consult new, non-roadway menu options (i.e., tourism and airports). Targeted marketing of these resources appears to be a necessary, if not necessarily sufficient, activity to stimulate usage of this information.
- Consider DMS marketing for reaching roadway information users. Although the lasting impact of short-term DMS marketing is unclear, short-term impacts appear dramatic. During the week-long, 24-hour/day, statewide DMS ad campaign in Arizona, daily 511 usage increased three-fold. Predictably, given the en-route exposure to DMS, the percentage of cell phone calls also spiked—to 83%, up from the post-enhancement pre-DMS campaign level of about 30%.
- Don't assume a regional, multi-modal 511 systems will "replace" transit customer information lines. Transit operators believe 511 lacks necessary capacity and that many transit requests can only be handled by customer service agents. They are reluctant to divert callers from the staffed information lines they've spent considerable resources to establish awareness of among their customers. Phoenix and Tucson transit plan to implement their own interactive voice response systems, although they are open to increasing integration with the 511 system.
- Vigorously build support among local agencies for their input of roadway information. Most local agencies have limited congestion and incident information and often lack the staff resources to enter it into a 511 system. Substantial efforts should be anticipated to secure support for 511 data collection and entry among the senior municipal and county agency leaders who provide the resources and technical staff who will be responsible for data entry. Efforts should be made to link 511 systems directly with agencies' construction and incident databases so as to minimize the need for "dual entry" by local agencies (i.e., entering information once into their own system and a second time into the 511 system). Efforts to integrate public safety computer-aided dispatch (CAD) systems with 511 databases should also continue.
Hypothesis Testing Results
A number of hypothesized project impacts were tested as part of the evaluation. Table ES-2 presents the results for the key hypotheses. Some hypothesized impacts were fully supported by the evaluation and others were partially supported or not supported, consistent with the mix of successes and challenges described on the preceding pages.
| Project Objective | Hypothesized Project Impacts | Evaluation Findings |
|---|---|---|
| Increase usage of the 511 system. | The addition of a number of new data types will contribute to increased usage of the 511 system. [NOT SUPPORTED] |
|
| Increase usage of the 511 system. | Usage of the 511 system will increase as a result of enhanced marketing. [SUPPORTED] |
|
| Contribute to high levels of customer satisfaction with the 511 system. | Users will view the information available on the 511 system as comprehensive and multi-modal. [PARTIALLY SUPPORTED] |
|
| Contribute to high levels of customer satisfaction with the 511 system. | Users will be satisfied with the quality of the information on the 511 system. [SUPPORTED] |
|
| Contribute to high levels of customer satisfaction with the 511 system. | Enhancements to the user interface, including voice recognition, segment-based reporting, and "Quick Reports" will contribute to customer satisfaction. [PARTIALLY SUPPORTED] |
|
| Promote efficiency of the 511 system. | The efficiency of information dissemination will be promoted through enhanced arterial street data capture, data entry operator training, and data quality control procedures. [PARTIALLY SUPPORTED] |
|
Concluding Observations
ADOT's approach to 511 has traditionally been evolutionary and opportunistic—they incrementally enhance the system over extended periods of time with the timing and nature of enhancements significantly impacted in the short-term by funding opportunities and partner interests. This approach was carried into the Model Deployment.
FHWA objectives for the Model Deployment were fairly broadly defined—essentially to "push the envelope" in as many ways possible—and the few specific requirements were not particularly challenging (i.e., certain types of information content were specified but the depth and quality of information was not.) The timeline was also tight—basically providing about a year for planning, design and implementation of a wide range of major enhancements.
Rather than suggesting mediocre success, the outcome of the Model Deployment is really a function of the confluence of ADOT's traditional evolutionary, opportunistic approach, a very ambitious slate of enhancements, and a relatively tight schedule. The fact that several of the enhancements have been completed since the end of the formal evaluation period (e.g., segment road weather information) and that work proceeds on all of the others with the exception of the premium service partnership, also indicate that the success of the Model Deployment—if measured strictly by enhancements completed-is largely a function of perspective.
511 deployers and the agencies that contribute 511 funding should have a shared, explicit understanding of their objectives and how realistic they are in light of resource constraints and the deployers' 511 strategy. If short-term unequivocal successes in many areas are required, those expectations and associated measures of success should be clearly communicated and the deployer should gear their approach accordingly.