ITS - Intelligent Transportation Systems Report ITS Home Page

Final Report

Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System



Executive Summary

This document presents the findings of the national evaluation of the 511 telephone traveler information system "Model Deployment" in Arizona. The United States Department of Transportation (U.S. DOT) National 511 Model Deployment supported a wide range of enhancements to the existing statewide Arizona 511 system, including the addition of several new types of information and significant redesign of the user interface.

On July 21, 2000, the Federal Communications Commission assigned 511 as the nationwide traveler information telephone number and granted responsibility for it to government transportation agencies. Since that time, 27 statewide and regional 511 systems have been implemented throughout the United States, including the Arizona system.

The Federal Highway Administration (FHWA) and Federal Transit Administration (FTA) jointly sponsored the Arizona 511 Model Deployment. The Model Deployment sought to demonstrate the potential of 511 services to bring together data from various sources and provide useful information to travelers through a state-of-the-art telephone interface. The national evaluation documented costs and lessons learned, measured the accomplishment of the implementers' hypothesized project impacts, and assessed the extent to which the Arizona 511 project satisfied the following U.S. DOT objectives for the Model Deployment:

Key Recommendations to 511 Deployers

A list of lessons learned is presented later in this Executive Summary and discussed in greater detail in the body of the report. This section highlights the most significant of those lessons learned in the form of recommendations to 511 deployers. The recommendations are as follows:

  1. If specific, near-term impacts are important, prioritize and focus your 511 implementation. The Arizona Model Deployment consisted of a very broad, ambitious slate of enhancements, none of which were explicitly identified as more or less important. The Model Deployment approach reflected ADOT's traditional "evolutionary-opportunistic" approach to incrementally enhancing their 511 system with the long-term objective of a generally improved system. 511 deployers who have specific performance levels in mind or who are trying to effect specific changes in 511 usage or travel behavior are strongly encouraged to utilize a directed, "problem-solving" approach. Such an approach would identify a small, manageable and measurable list of specific objectives and then concentrate resources in those few areas.
  2. Voice recognition is not "plug and play"; expect to spend considerable resources on its implementation. Although voice recognition technology is widely deployed, its successful application to 511 systems is challenging. Statewide, multi-modal 511 systems strain the limits of the technology due to the very large number of potential utterances that the system must detect. Although basic implementation of a voice recognition system is relatively straightforward, troubleshooting, modifying, and testing the system to achieve acceptable performance are very resource intensive. Because of the complicated nature of voice recognition and the changes in technology, agencies that do not have up-to-date technical expertise in-house may consider "turn-key" firms that specialize in voice recognition services. Deployers should expect to devote considerable attention to the testing and refinement of a 511 voice recognition system before it goes on-line.
  3. Pursue a vigorous, targeted 511 marketing program. Marketing is essential to building awareness and stimulating usage of 511 systems. This Model Deployment demonstrated that dynamic message signs as part of a marketing campaign are highly effective in reaching en-route travelers. It is especially important to market any significant new features or information content enhancements to existing 511 systems so that users know about them. This Model Deployment indicated that simply adding new features and information is not sufficient to stimulate usage. Targeted marketing to the user subgroups who would be interested in the new information is especially important.
  4. In the early planning stages, thoroughly investigate institutional and technical issues associated with data input by partner agencies. Many of the Model Deployment enhancements that were not completed or not fully successful depended on data entry by new 511 partners. Several of these agencies committed to this role in principle during the very early stages of the Model Deployment (during preparation of the funding proposal) but for various reasons did not fulfill that role. If dependent on key data from other agencies, 511 deployers are strongly encouraged to thoroughly investigate the issues that could impact the ability of partners to deliver data as envisioned and to resolve key issues before moving forward.
  5. Incorporate mechanisms for capturing user feedback for system evaluation. Because many of the benefits of traveler information systems such as 511 are qualitative, it is important to be able to capture users' experiences as a way of evaluating the system's performance. Although ADOT was initially reluctant to interrupt callers as part of a user survey process, the evaluators and system designers were able to develop and implement a procedure that minimally inconvenienced callers, yet permitted capturing valuable information about the ways that users interacted with the system. This information not only revealed how callers were using the system but also where improvements were needed to make the service easier to use and provide better travel information.

Description of the Model Deployment

The Model Deployment was led by the Arizona Department of Transportation (ADOT) and assisted by a 511 Task Force composed of Tucson and Phoenix area Metropolitan Planning Organizations, airport and transit operators, and other county and municipal transportation agencies. The enhanced 511 system became operational in mid-December 2003.

Table ES-1 identifies the 18 planned Model Deployment enhancements and their operational status relative to the one-year, January – December 2004, post-deployment evaluation period. The enhancements fall into four general categories:

As noted in Table ES-1, 9 of the planned 18 Model Deployment enhancements were deemed by the Evaluation Team to be fully operational and suitable for evaluation within the evaluation period. None of the planned 18 Model Deployment enhancements were considered by the deployers to be more critical than any others, and there was, therefore, no explicit prioritization of the enhancements. However, when it became clear that not all of the enhancements could be implemented concurrently, efforts were focused on the basic user interface enhancements—voice and segment-based road information—which suggests that these may have been viewed as more important. Work continues on all of the unimplemented enhancements with the exception of the partnership with a private premium service provider, which has been dropped. No qualified private partner responded to the ADOT request for partnership proposals.

Table ES-1. Model Deployment Enhancements and Status
Enhancement Fully Complete & Operational within the Evaluation Period Completed & Operational After the Evaluation Period Not Currently Complete & Fully Operational Comments
Information Content
Tucson and Phoenix Local Street Data   square bullet   ADOT's plan to increase their capture of arterial street incidents through more intensive monitoring of law enforcement scanners was carried out. However, inputs by cities and counties have been negligible. Inputs by the Tucson region are pending (training has been completed as have necessary map updates); Phoenix area agencies are making a limited number of inputs.
Transit Major Service Disruptions and Call Transfer Capabilitysquare bullet  The original plan called only for Phoenix and Tucson Transit, but the menu system and call transfer capability have been extended to include all regional/rural transit providers statewide.
Phoenix Bus Rapid Transit Estimated Arrival Times  square bulletTransit currently unable to provide data. Enhancement not implemented and unavailable for evaluation.
Phoenix Arterial Street Travel Times  square bulletData collection system completed and now being tested; not available for evaluation.
Data Quality Enhancementssquare bullet  Enhanced 511 data entry operator training and 511 message preview function to see how entries will be conveyed on 511.
Phoenix and Tucson Airport Informationsquare bullet  Phoenix has used the 2-minute message recording capability but Tucson has not.
Grand Canyon National Park Information square bullet All necessary technical elements are in place (menu changes and voice-recording capability). During the evaluation period, the Park was not entering any information but now intends to.
Arizona Office of Tourism Call Transfersquare bullet  An unplanned enhancement requested by the Office of Tourism.
Segment Weather Information square bullet Came on-line after completion of evaluation.
Downtown Phoenix Special Events and Parking  square bulletThere has been no discernable change in data capture techniques or the type and volume of information in the system.
Sharing Data with Other States (i.e., Utah)  square bulletUnder development. Not available for evaluation.
User Interface
Regional Roadway "Quick Reports"square bullet   
Roadway Segment-Based Reportingsquare bullet   
Roadway Info. Accessible Via Roadway Namesquare bullet   
Voice Recognitionsquare bullet   
Other
Premium Service Partnership  square bulletNo viable private partnership proposal was received.
Marketing square bullet The only incomplete elements are the 50 static road signs throughout the state, which are being installed now.
Improved System Performance Monitoringsquare bullet  Not all data analysis and reporting functions have yet been utilized but many new capabilities have been established and many of them have been used by ADOT.

Evaluation Approach

The evaluation was conducted on behalf of the FHWA by Battelle Memorial Institute in partnership with the University of Arizona. Volpe National Transportation Systems Center served as Evaluation Program Manager on behalf of the FHWA ITS Joint Program Office. The evaluation included four major analyses:

Results

Overall, the Model Deployment succeeded in dramatically improving the Arizona 511 system, although it was not fully successful in a number of respects. Through both the accomplishments and shortcomings of the Model Deployment, a number of insights have been gained that will be useful to ADOT and their 511 partners as well as other 511 deployers around the country. This section summarizes major accomplishments, challenges, lessons learned, and findings regarding hypothesized project impacts.

Major Accomplishments

Major accomplishments of the Model Deployment include:

In addition to these system enhancements, the Model Deployment facilitated a number of other broader benefits, including:

A photograph shows signs over the highway that include an electronic message board to provide alerts and instructions to motorists.

Figure ES-1. 511 Marketing via Dynamic Message Sign

In several important respects, the Model Deployment was well received by users. About 70% of surveyed callers were satisfied with information content. When interpreting user feedback, it is important to note that the Arizona 511 user base is changing rapidly. Call volumes during the one-year evaluation period increased 74% over the prior year (Figure ES-2) and much of the increase was from new 511 users. The week-long 511 DMS advertising was very effective in stimulating calls to the system. Daily call volumes increased three-fold during that time and just over one-third of surveyed callers reported seeing the DMS advertisements (note the May 2004 spike in Figure ES-2). By the time of the caller survey almost six months later, one-third of surveyed users were new callers, that is, the 511 call during which they were intercepted for survey recruitment was their first ever call to Arizona 511 (Figure ES-3).

Comparing the experience of new callers to 511 with that of previous callers provides useful insight into how well the service is performing and how likely it is to retain callers over time. It should be noted that 511 is not heavily utilized even by most repeat callers, among whom half reported that they made three or fewer calls in the last month and virtually all of them were seeking road information. Within that context repeat callers report high levels of satisfaction with the information they receive and cite several specific benefits in using the 511 system; for example, saving time/arriving on time and avoiding traffic congestion due to accidents or construction delays were each cited by about 20% of repeat callers. While most first-time callers also expressed overall satisfaction with the service, they were less likely to articulate specific benefits than were repeat callers and, indeed, 31% said they received no benefit from using 511 the first time they called. In addition, when asked about improvements to 511 that they would recommend, first-time callers tended to want more types of improvements than did repeat callers. On the other hand, repeat callers placed a higher premium on two specific areas for improvement: speech recognition and more details on traffic congestion and delays. While 95% of first-time callers said they would use the service again, surely on future calls to 511 the service must begin to deliver clear benefits to them for them to be converted to regular users.

Monthly call volumes. Line chart plotting number of calls over a 16-month period for two data sets. The pre-enhancement data set runs from September 2002 to August 2003 and trends below 20,000 calls through November, rises to 100,000 in December reflecting a major winter storm, falls to just above 20,000 in January, then hits 60,000 in  February, about 45,000 in March and then trends along the 20,000 mark through August. The post-enhancement data set runs from January through December 2004. It starts slightly above 40,000 in January, increases slightly in February and then drops steadily to just above 20,000 in April. It peaks to just under 160,000 in May, reflecting a week-long advertising campaign, drops to about 50,000 in June, and increases to 70,000 in July. It trends just above 40,000 through October, and increases to just under 60,000 for the rest of the year.

Figure ES-2. Monthly Call Volumes

Percent of callers by how long ago they first used 511 (n = 404). Pie chart in five segments. New caller accounts for 33 percent; 1 month accounts for 13 percent; 2 to 6 months accounts for 33 percent, 7 to 12 months accounts for 8 percent, and greater than 12 months accounts for 13 percent of callers

Figure ES-3. Percent of Callers by How Long Ago They First Used 511 (n=404)

Challenges

The major challenges or shortcomings of the Model Deployment are that many of the planned enhancements were not completed within the evaluation period (several of these have since been completed), some 511 partner agencies have not taken significant advantage of their new capabilities to provide information via 511, new information types are not being used significantly by most users, and many users are dissatisfied with the performance of the voice recognition system.

As indicated in Table ES-1, half of the 18 planned enhancements were not fully operational in time for the evaluation, or were not being utilized in a manner that would support evaluation. With the exception of the private partnership, which was cancelled, efforts are continuing on all of the unimplemented enhancements. Delays resulted from a variety of factors not uncommon to complex technology implementations, including assorted technical challenges and limited agency staff resources. That is, ADOT found that there was a limit to how many enhancements they could simultaneously pursue.

Several enhancements were only partially implemented. In a few of these cases, ADOT added the capability for data entry but the partner agency did not, during the evaluation period, take advantage of it. This was the case with the Grand Canyon National Park, Tucson airport, and Phoenix-area municipal traffic information enhancements. In the case of municipal traffic agencies, they indicated that they lack the senior agency support necessary to devote resources to collecting and entering this information into 511.

With regard to user reactions to the Model Deployment, there are two primary concerns. First, very few users consulted the new information available on the 511 system (i.e., tourism, airports, and roadway Quick Reports). Combined, these items accounted for less than 8% of all menu selections over the one-year evaluation period. The vast majority of menu selections—92%—were for the two basic types of pre-enhancement information, roadways and transit. Of those selections, nearly all (91%) were for roadways. These findings are at least in part a result of the fact that, during the evaluation year, little 511 marketing promoted the new types of information to travelers who would have been interested, and, therefore, most were presumably unaware of its availability. They may also be related to the quality or depth of the information available on 511, although very small sample sizes for users of these non-roadway options prevented definitive conclusions. It may also relate to the availability of other options for non-roadway information. In the case of transit, the low utilization relative to roadways reflects very low transit usage—1.2% of all person trips in the Phoenix region.

The second concern from a user perspective is the relatively high levels of dissatisfaction with the performance of the voice recognition feature. About 35% of repeat callers who choose to use voice recognition as their primary means for accessing the service said they were dissatisfied with the feature, and 49% of new callers were dissatisfied. The difficulties with voice recognition are reported on both cell and landline phones and both while holding a receiver and speaking hands free. The performance of the voice recognition system is clearly not acceptable to a large number of users. System data provide support to the perception of voice recognition problems. Over the one-year operational period, 37% of all calls included at least one user input that could not be interpreted by the system. That number was 58% during the first month of voice recognition operation. The voice recognition feature was the most commonly recommended area for improvement among both new and repeat callers.

Lessons Learned

The major lessons learned of the Model Deployment consist of the following:

Hypothesis Testing Results

A number of hypothesized project impacts were tested as part of the evaluation. Table ES-2 presents the results for the key hypotheses. Some hypothesized impacts were fully supported by the evaluation and others were partially supported or not supported, consistent with the mix of successes and challenges described on the preceding pages.

Table ES-2. Evaluation Results for Key Hypothesized Impacts
Project Objective Hypothesized Project Impacts Evaluation Findings
Increase usage of the 511 system. The addition of a number of new data types will contribute to increased usage of the 511 system. [NOT SUPPORTED]
  • The overall call volume in the post deployment period increased by 74% compared to the same period before the enhancement. However, the old information categories accounted for 91% (Roads) and 2.4% (Transit) of all information requests in the post-enhancement period. The new information categories accounted for 4.3% (Quick Reports), 1.3% (Airports), and 0.4% (Tourism) of all information requests. Thus, it cannot be concluded that new data contributed substantially to increased usage.
Increase usage of the 511 system. Usage of the 511 system will increase as a result of enhanced marketing. [SUPPORTED]
  • During the week-long DMS marketing campaign, call volume increased by a factor of 3.1. Ninety-six percent of the callers during the campaign were new users—they had not used the system in the last five months—and 86% were calling from wireless phones. The DMS marketing campaign, though short-lived, was extremely effective in publicizing the 511 service to highway users.
  • Based on the survey results, the other marketing activities that were implemented (i.e., distribution of materials at the state fair and freeway opening ceremonies and radio advertisements over an approximately two-week period) had much less of an impact. Nine percent cited the radio ads, and only 2% of respondents cited ADOT as their source of awareness.
Contribute to high levels of customer satisfaction with the 511 system. Users will view the information available on the 511 system as comprehensive and multi-modal. [PARTIALLY SUPPORTED]
  • Surveyed callers overwhelmingly (90%) used the system to obtain road information for the trip they were taking. Very few (<1%) accessed transit, airport, and tourism information.
  • 97% of surveyed repeat users viewed 511 as covering the areas and routes in which they are interested. However, only 26% agreed that 511 covers all the bus services of interest. With so very few callers using bus information, the measure cannot be considered reliable.
  • Only 5 repeat callers had ever used the airport and only 6 had used tourism information. This number was too few to draw conclusions about customer satisfaction with these content areas.
Contribute to high levels of customer satisfaction with the 511 system. Users will be satisfied with the quality of the information on the 511 system. [SUPPORTED]
  • 71% of surveyed callers expressed overall satisfaction with information they received for the trip they were taking. For each of the high-level menu items selected, satisfaction with the quality of information was even higher.
  • For all the times they've used 511 the majority of surveyed repeat callers expressed satisfaction with most all the road content. The exception was regional Quick Reports, which was rated a 46% satisfaction level.
  • 82% of surveyed repeat callers perceived traffic information on 511 to be accurate and timely.
  • In comparing 511 to radio, a source used by 72% of surveyed callers, 43% of callers felt that that the quality of traffic information on 511 was better, 29% said it was about the same, and 28% thought the radio was better.
Contribute to high levels of customer satisfaction with the 511 system. Enhancements to the user interface, including voice recognition, segment-based reporting, and "Quick Reports" will contribute to customer satisfaction. [PARTIALLY SUPPORTED]
  • Although voice recognition is used by the majority of callers, it has its drawbacks and receives lower satisfaction ratings than phone buttons. First-time callers are less satisfied with voice recognition than repeat callers. The principal reasons for dissatisfaction were that 511 did not understand the spoken request, 511 gave the wrong information, and background noise caused interference. The problems were not related exclusively to either cell phones or hands free mode, based on survey results.
  • Segment-based reporting for road information was a feature with which 84% of repeat callers expressed satisfaction.
  • Regional Quick Reports received mixed reaction for callers. Only 8% of callers used the feature for the surveyed trip, but 73% of them were satisfied with the information they received. On the other hand, most repeat callers had at some time tried Quick Reports, but only 46% of them found them satisfactory. It appears that callers prefer to access specific roads of interest rather than go to the regional summaries.
Promote efficiency of the 511 system. The efficiency of information dissemination will be promoted through enhanced arterial street data capture, data entry operator training, and data quality control procedures. [PARTIALLY SUPPORTED]
  • The volume of arterial street information input by ADOT increased dramatically as a result of their intensified monitoring of police scanners. Annual entries of this type increased from 234 (2% of the total) to 2,763 (14%).
  • Information input by Phoenix area cities and counties increased negligibly, from pre-enhanced average of 15 entries per year to 45 entries in the post-deployment year of operations.
  • No information was input by Tucson area city and county agencies during the post-enhancement period. These agencies achieved the ability to do so only late in the one-year operational period (2004). They intend to begin doing so sometime in 2005.
  • Interviews with ADOT data entry personnel and other ADOT 511 staff indicate that significant changes have been made that they view as enhancing data quality. Refinements were made to roadway information location references. A key new preview function was added allowing data entry operators to see how their input will be translated to a 511 message. The data entry operator's manual has been revised and training has been conducted on new procedures and features.

Concluding Observations

ADOT's approach to 511 has traditionally been evolutionary and opportunistic—they incrementally enhance the system over extended periods of time with the timing and nature of enhancements significantly impacted in the short-term by funding opportunities and partner interests. This approach was carried into the Model Deployment.

FHWA objectives for the Model Deployment were fairly broadly defined—essentially to "push the envelope" in as many ways possible—and the few specific requirements were not particularly challenging (i.e., certain types of information content were specified but the depth and quality of information was not.) The timeline was also tight—basically providing about a year for planning, design and implementation of a wide range of major enhancements.

Rather than suggesting mediocre success, the outcome of the Model Deployment is really a function of the confluence of ADOT's traditional evolutionary, opportunistic approach, a very ambitious slate of enhancements, and a relatively tight schedule. The fact that several of the enhancements have been completed since the end of the formal evaluation period (e.g., segment road weather information) and that work proceeds on all of the others with the exception of the premium service partnership, also indicate that the success of the Model Deployment—if measured strictly by enhancements completed-is largely a function of perspective.

511 deployers and the agencies that contribute 511 funding should have a shared, explicit understanding of their objectives and how realistic they are in light of resource constraints and the deployers' 511 strategy. If short-term unequivocal successes in many areas are required, those expectations and associated measures of success should be clearly communicated and the deployer should gear their approach accordingly.


< Acknowledgements | Table of Contents | 1.0 >



FHWA Home | Feedback
FHWA