Final Report

Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System

9.0 Conclusions

This section presents the overall conclusions of the Model Deployment. Preceding sections included detailed discussions of the conclusions based on individual evaluation analyses and hypotheses (Sections 4.0 - 8.0). This section focuses on conclusions relative to the four major objectives of the evaluation, which are to:

Provide an independent review of the performance of the Model Deployment, including the extent to which it accomplishes the national objectives (which are summarized in Section 1.1).
Document how the Model Deployment was implemented, including system costs and how technical and institutional issues (especially cross-modal and interstate) issues were resolved.
Provide ADOT and the other partners with feedback that will allow them to improve the effectiveness of the system.
Deliver lessons learned that will inform the U.S. DOT 511 effort and that will be of use to agencies operating and planning to implement 511 systems.

9.1 Performance of the Model Deployment and Accomplishment of National Objectives

The objectives for the 511 Model Deployment set forth by the U.S. DOT fall into three areas. First, the Model Deployment was to "push-the-envelop" in telephone traveler information, in terms of innovative techniques and the quality of the service. Second, the user interface was to effectively balance comprehensiveness with brevity and usability, and to require no direct contact with human operators. Third, the 511 system was to provide information, including but not limited to the following: information related to current traffic, public transportation, roadway incidents and construction, special events, and weather or road surface conditions. This section summarizes evaluation findings in each of these three areas.

9.1.1 Innovation and Quality

Overall, the Model Deployment dramatically improved on the previous Arizona 511 system, greatly enhancing the quality of the product available to travelers and establishing the means to maintain and further enhance the quality and effectiveness of the system. Thus, with regard to system structure, the Model Deployment is a success. Highlights of the structural enhancements realized through the Model Deployment include:

Introduction of user input into the design and ongoing operation of the system, providing a critical source of customer feedback (the comment line), both as input to the Model Deployment and for ongoing quality improvement.
Replacement of the cumbersome, roadway-based touch tone menu system with a voice recognition system featuring roadway-segment and region-based reporting (Quick Reports).
Introduction of several new types of information, fundamentally changing it from a heavily roadway-oriented to a more multi-modal information resource.
Ability to deliver much more information of the type desired by design focus group participants: arterial street, incident, congestion, and regional traffic summary (Quick Report) information.
Creation of a philosophy of "customer service" and "continuous quality improvement," including the necessary system performance monitoring tools and framework for continuing enhancement.

In addition to the accomplishments that were realized during the Model Deployment, work continues on several fronts to fully realize the original, ambitious Model Deployment vision. This work focuses on the numerous planned new data types that were not successfully integrated by the time of the evaluation: segment weather, estimated bus arrival times, arterial street travel times, and information on bordering states.

Finally, the overall 511 system effectiveness has increased substantially in several areas, some directly related to the Model Deployment and some benefiting from the associated overall increase in focus on the system. These gains include significant expansion of the coverage area to essentially statewide coverage and maintenance of adequate capacity and system availability (i.e., minimal downtime) in the face of major system renovation and significantly increased call volumes.

In terms of dramatically improving the previous 511 system, the Arizona 511 Model Deployment has been extremely successful, thus meeting the U.S. DOT model objective to improve quality. However, in regard to "pushing the envelope" and creation of a truly innovative system, results to date are mixed. This conclusion is based on the following three considerations.

First, the enhancement of the user interface, institution of the first real Arizona 511 marketing campaign, incorporation of user input, and addition of links to several types of new data represent dramatic improvements in the Arizona 511 system. However, relative to state-of-the-practice 511 systems around the country, such as the systems in Virginia, Washington State and Maine, they have brought the Arizona system into the mainstream of current 511 approaches.

Second, many of the truly innovative, "envelop-pushing" aspects of the Model Deployment were not completed within the evaluation time frame (other states' data, segment weather information, arterial street travel times, and estimated bus arrival times). ADOT and its 511 partners are continuing to vigorously pursue these enhancements (several have been completed since the evaluation period ended) but the ultimate success of a number of them is still uncertain. One of the areas where innovation was anticipated failed completely—the premium service partnership with a commercial information provider. It was the only planned enhancement to be dropped entirely, after no viable partner and business model could be identified.

Third, some of the 511 enhancements that have been implemented are not yet being used by any significant number of users, and/or are not universally well regarded by users. The usage and survey analyses indicate that although a number of new data types (many consisting of call transfers) have been introduced, very few callers are using them. From a usage perspective, the system remains highway-centric. Of course, this may change significantly over the next couple of years, as the impacts of the ongoing marketing campaign are felt, including growing national recognition of 511 as a resource, and with the continued enhancement of the system. The survey analysis also indicates that although 71% of users are satisfied with the system, there is some lingering concern with the quality of the voice recognition system, specifically in its ability to correctly interpret user utterances. Over one third (35%) of callers who primarily use the voice recognition option (rather than the touch tone option) said they were dissatisfied with the feature. This sentiment is shared by several (although not all) of the agency participants in the 511 Model Deployment.

9.1.2 User Interface

As described in Section 9.1.1, the Model Deployment dramatically enhanced the Arizona 511 user interface. The system now provides the option of voice recognition, which, although not yet perfect, is operational and is well received by a majority of users. Roadway information is now available at the segment and regional level, a dramatic improvement over the old system that provided information only at the roadway level. The previous method required callers to listen to or skip through every event on a given roadway over the entire state until they found the information they were looking for. The regional traffic reports—"Quick Reports"—provide summary traffic information for specific regions, such as portions of the Phoenix area. Perhaps most importantly, most 511 users are generally satisfied with the user interface, finding it easy to navigate and understandable. This suggests that so far, the Model Deployment has been successful in adding many new menu options without making the system onerous to use. Of course, this perception could change if and when more users seeking non-roadway information begin using the system in significant numbers.

These successes in accomplishing user interface-related Model Deployment objectives are mitigated by two factors. First, many users are not fully satisfied with the voice recognition system. Second, the analysis of unrecognized user inputs (Section 4.1.9) suggests that many user inputs are not comprehended by the system.

9.1.3 Information Content

The enhanced 511 system includes all of the targeted data types identified by the U.S. DOT and information content was enhanced through the Model Deployment. Improvements in information content include dramatic increases in the amount of roadway incident and congestion information, the addition of a call transfer option to the Arizona Office of Tourism, pre-recorded information from the Phoenix airport, and call transfer options to the Phoenix and Tucson airports and transit operators throughout the state. These accomplishments are, however, somewhat lessened by a number of factors:

Some key new data types planned for the Model Deployment were not completed within the evaluation time frame, including several of the most innovative and challenging: estimated next bus arrival times, segment weather information, arterial street travel times, arterial street information overall, and bordering states' data. Of these enhancements, segment weather information has, since the end of the evaluation period, been successfully implemented.
Most 511 callers are not using the new data.
A couple of information content enhancements could be considered only partially complete or effective. For example, there has been no notable increase in the amount of Downtown Phoenix special event or parking information, although it is hard to draw conclusions since the deployment plan did not identify specific enhancements. Other examples include information for Grand Canyon National Park and Tucson's transit system and airport. Although all three organizations now have the means to input information into the system, they are not taking much advantage of it. In the case of transit and the airport, this may be because the organizations do not feel that they have notable specific information to disseminate.

9.2 The Enhancement Process

This section includes two components. The first presents conclusions relative to the overall effectiveness of the enhancement process, including scope (success in implementing planned enhancements), schedule and budget considerations. The second section highlights a number of specific technical and institutional issues that were encountered in the enhancement process.

9.2.1 Overall Effectiveness

Overall, the Model Deployment process was partially effective. Successes include:

Completion of many enhancements, including some key ones like voice recognition and the addition of a couple of new data types.
Strengthening of ADOT relationships with some partner agencies.
Reinvigoration of statewide and Phoenix region ITS partnering, especially in the area of traveler information.

Shortcomings of the process include:

Several incomplete enhancements (as of the conclusion of the evaluation period), including some of the most innovative and nationally significant ones (e.g., other states, arterial street travel times, bus arrival times, segment weather).
Costs for several enhancements significantly exceeded estimates, although ADOT made up short-falls with their own funds.
Several partners did not actively participate (at least during the evaluation period), including local traffic agencies that did not contribute the local street information that was expected.
No private premium service partner was identified (although an adverse outcome, this is not necessarily a shortcoming of the process.)

Most of the project participants view the preceding shortcomings of the enhancement process as regrettable but not unexpected given the scale and ambition of the project. This perspective also reflects the fact that efforts continue on almost all of the delayed enhancements.

9.2.2 Specific Issues

Institutional

A comprehensive list of issues is presented in Section 7.0. Among the most significant institutional issues were:

It was difficult to engage some partners before the basic enhanced system was deployed in December 2003, and some partners never did fully engage (at least within the evaluation period.)
Inability or unwillingness of cities and counties to enter a significant volume of local street roadway information (incident, congestion and construction). Agencies indicated that they lacked the staff resources to enter the data into HCRS. It is also the case that most cities and counties have a very limited amount of accurate incident and congestion information.

Technical

Several technical challenges were encountered. With the exception of those related to the user interface, these issues directly contributed to the delays which prevented a number of enhancements from being completed within the Model Deployment time frame. Technical issues include:

Discovering that Utah's roadway information was categorized differently and at a much higher level, and therefore would be difficult to synchronize within ADOT's much more elaborate coding scheme.
Inability to export Phoenix bus estimated arrival time data from the proprietary automatic vehicle location system, either to other parts of the transit operation or to the 511 system.
Difficulties in identifying a workable communications scheme to relay arterial street travel time information (based on vehicle-matching using license plate readers) from the field to a central processing location.
Challenges in mapping the National Weather Service's 2-kilometer grid weather data to the much longer roadway segments used in the 511 system.
The unexpectedly extensive enhancement of HCRS base maps necessary to accommodate local street data to be entered by the City of Tucson.
Computer network security challenges encountered in providing non-ADOT agencies secure access to the ADOT computer system where HCRS resides.
Segmenting interstates that serve both heavy intercity and urban area demand in a way that would be intuitive to users, both those interested in information on the roadway throughout the state and those interested in information for the roadway only within a given urban area.
The process of implementing and refining the voice recognition system proved more challenging than anticipated—"more of an art than a science." Specific technical challenges included the effort to build the large library of human voice-recorded roadway location references (necessary as the system moved from text-to-speech to concatenated speech) and the significant fine-tuning and de-bugging of the voice recognition system. For example, it was discovered that the system tended to interpret many extraneous background noises as the utterance "eight" and therefore often mistakenly provided information on Interstate 8.

9.3 Suggested Improvements to the Arizona 511 System

Through the evaluation process, a number of suggested improvements to the Arizona 511 system have been identified. The need for a number of these improvements became clear to most participants based on the results of the enhancement process, and some are being pursued. Other suggestions are based on the results of the usage and survey analysis and may not be as widely perceived among the Model Deployment participants. Suggested improvements consist of the following:

Further Refine the Voice Recognition System – The concerns on the part of several participants and the evaluation team that the voice recognition needs further improvement were supported by the results of the user survey. More than a third (35%) of repeat callers surveyed who primarily used the voice recognition interface (as opposed to the touch tone option) said they were dissatisfied with it. It may be useful to perform controlled testing of the voice recognition system with real 511 users, to observe first hand the problems they encounter and why they encounter them.
Continue to Pursue Unimplemented Enhancements – None of the experiences of the enhancement process or any findings of the evaluation suggest that any of the unimplemented enhancements were ill conceived and should be entirely dropped. ADOT and the 511 partners are encouraged to continue their efforts to implement these enhancements, informed by the results of this evaluation. As they pursue non-highway related enhancements (e.g., estimated bus arrival times) they should keep in mind that current users do not appear very interested in such information. Therefore, the addition of this information should be accompanied by marketing efforts targeting the intended users of such information, i.e., non-traditional 511 users.
Continue to Work to Increase Arterial Street Information – Creating the mechanism for city and county data entry directly into HCRS was an important Model Deployment accomplishment. However, additional efforts are obviously needed in order to stimulate data entry. ADOT is working with regional ITS partners on this issue and should continue to do so. An approach suggested by one of the few cities that has input information (City of Glendale) was to start with a smaller, more manageable goal of capturing all planned event information in the Phoenix area. Later efforts could advance to real-time information, which is even more challenging, given that most cities and counties do not have this information for their roads. Those efforts should probably include outreach to law enforcement. One prong of the overall effort in the area of arterial street information could focus on building support among senior agency leadership, since many technical staff point to a lack of resources as one of the obstacles to data entry. ADOT's efforts to identify an automated approach to importing planned event information from city and county data systems directly into HCRS could also help address this challenge.
Continue to Utilize the New Performance Monitoring Tools – The Model Deployment has greatly increased the amount of data available to ADOT to monitor the performance of the system and to guide ongoing enhancements. This includes the caller comments feature and the ability to generate reports providing many of the data presented in the usage analysis portion of this evaluation, including menu selections by topic. ADOT should take advantage of these new tools. It is also recommended that ADOT consider adopting the performance measures recommended by the national 511 Deployment Coalition.
Plan on Additional Surveys and/or Focus Groups in the Future – The caller comment feature is a good tool but does not replace the sort of input that can be obtained through surveys and interviews. ADOT and the 511 partners should strive to include periodic surveys and/or focus groups as part of their long range 511 monitoring and enhancement program. Other statewide surveys conducted by ADOT or Metropolitan Planning Organizations may provide an opportunity to ask a few basic questions about 511, which if nothing else could help gauge overall awareness and the impact of ongoing marketing. Future surveys or focus groups could also explore some of the apparent inconsistencies observed in this evaluation. For example, Quick Reports were identified by design focus group participants as a very desirable feature but are hardly used.
Continue the 511 Marketing Program, Focus It, and Leverage Partners – Although ADOT is clearly continuing some aspects of the 511 marketing program, to some extent it appeared that they may have considered the Model Deployment marketing as something of a special "one-time" effort. It is recommended that ADOT and their partners continue marketing 511. Those efforts should include a focus on the non-highway data types that are not currently being utilized and the associated user submarkets, such as transit riders and tourists. Although the 511 partners have provided some assistance in marketing, considerably greater efforts on their part are possible and can play a major role in reaching non-highway information users. Also, the usage analysis and the caller survey indicated that dynamic message signs were an effective marketing outlet. ADOT should consider periodically repeating the statewide 511 message postings, perhaps quarterly, as a way to remind long-term residents and inform tourists and new residents. With Tucson area cities and counties now able to input information, and with the massive reconstruction of I-10 looming, increased marketing in Tucson on the part of ADOT and/or Tucson-area 511 partners also seems warranted.
Reconsider Expectations Relative to Commuter Versus Recreational Use – One of ADOT's broader, informal objectives of the Model Deployment was to make the system more commuter oriented. They traditionally have viewed the system as oriented primarily toward intercity recreational travel, presumably based on the historically limited amount of urban area incident and congestion information available on the system and the relatively high weekend call volumes. This evaluation indicates that the system was and is utilized fairly equally for commuting and intercity/recreational trip purposes. This suggests that ADOT may have underestimated the commuter usage of the system or had expectations for even a truly commuter-dominated system. The balanced usage is in fact a positive finding, indicating that the system has value for different types of travelers and different trip purposes. From that perspective, expectations that the system should or can become more commuter-oriented may be ill founded. It is also the case that recurring traffic congestion is not as extreme anywhere in Arizona as it is in cities like New York, San Francisco and Los Angeles, and therefore the Arizona 511 system may not demonstrate the heavy levels of commuter usage typical of those systems.

9.4 Lessons Learned of National Significance

This section summarizes the evaluation conclusions most relevant to other 511 deployers and the national 511 community, including federal agencies. Conclusions are presented in two areas: those stemming from the enhancement process and cost analyses and those from the usage and survey analyses.

9.4.1 Enhancement Process and Cost-Related Conclusions

Invest in Formal Planning and Design Documents (e.g., system requirements) – The ADOT 511 project manager noted that the time and effort spent early on to develop the Program Management Plan and System Requirements Definition document were instrumental in the project successes. He also indicated that outside consultant program management support was critical (the program management consultant led the development of the two documents, among many other support activities).
Nurture Partner Agency Relationships and Commitments Throughout the Process – A number of agencies who had pledged to participate in the Model Deployment did not play an active role during the first year of post-enhancement operations. ADOT's attempts to involve these agencies, which were sincere but somewhat limited, were unsuccessful in stimulating their participation. This suggests that when involving new partners in a 511 operation traditionally associated with a single agency, extensive outreach may be needed to stimulate meaningful participation by all partners.
Solicit User Input During Design and System Refinement – Prior to the Model Deployment ADOT had essentially no user input regarding the 511 system. Focus groups conducted as part of the design process were very useful, both in expanding ADOT's appreciation of how real users relate to 511 and identifying specific issues and preferences. The focus groups resulted in the implementation of "Regional Quick Reports," regional summaries of traffic conditions, an enhancement that was not part of the original Model Deployment plan. These focus groups also revealed that there was perception that the 511 system contained little congestion and incident information, which may have helped stimulate the increase in this type of data that has been observed.
Consider the Potential Advantages of In-House 511 Operation if In-House Expertise is Available – ADOT noted that the fact that they operate the 511 system themselves, as opposed to out-sourcing it to a commercial operator, has provided them a great deal of flexibility and control in modifying the system. They also point out that although the initial costs for an in-house system are higher, they believe the recurring costs are lower. Based on their experience, they encourage other 511 deployers to consider in-house operation. However, one of the key ingredients underlying ADOT's success with the in-house approach is that they have had experienced technical personnel able to devote extensive time and effort to the system. Because of the complicated nature of voice recognition and the changes in technology, agencies that do not have up-to-date technical expertise in house may consider "turn-key" firms that specialize in voice recognition services.
Budget Resources for Data Entry Training and Quality Control When Relying on Partner Agencies' to Input Data – Utilization of other agencies (those not fundamentally responsible for the 511 operation) to enter their data into the system is, in theory, a highly efficient way to quickly increase the data content on a system. Getting the agencies to actually do so, however, can be difficult. Even if the agencies do input the data as planned, there is some risk that they may not fully comply with data entry procedures, or they may be inconsistent in judgment calls, such as the prioritization given to various types of incidents (an inherently subjective process). In short, a decentralized approach to data collection sacrifices a measure of consistency and control that is possible when data is input from a central location. This suggests that when a decentralized approach is used, training and explicit data entry and data quality procedures are especially important.
Consider the Possible Advantages of Retaining Some Human Operator Role in Data Entry – ADOT believes that in order to ensure data quality and consistency, human data entry operators need to play a central role. They feel that a fully automated system cannot provide acceptable quality and consistency. The ADOT 511 project manager noted that the Arizona 511 system features considerable human data input and that a continuing significant level of human operator involvement is viewed as necessary in order to maintain data quality.
Plan for Voice Recognition Development and Refinement to be Resource Intensive – Plenty of time should be allowed for this activity. Implementation and refinement of the voice recognition capability required more time and effort than was anticipated. Specific challenges had to do with voice-recording the large number of road names and roadway location references (e.g., place names) associated with the conversion from a text-to-speech approach to a concatenated speech approach, and the significant fine-tuning and de-bugging of the voice recognition system. For example, it was discovered that the system tended to interpret many extraneous background noises as the utterance "eight" and therefore often mistakenly provided information on Interstate 8. 511 deployers lacking specific in-house voice recognition expertise should plan to utilize a contractor experienced in this area to support their in-house efforts or utilization of a "turn-key" firm to provide the full 511 voice recognition service.
Build In a Call Intercept Capability – Deployers should periodically conduct caller surveys to monitor customer satisfaction and obtain other useful information about callers' use of 511. Since a live intercept is the most effective means for obtaining a representative sample, a built-in survey capability will facilitate live intercepts and automated surveys without the time and expense of retrofitting the system.
Plan for the Possibility that Costs for Significant Upgrades to 511 Systems May Exceed Costs for the Basic, Initial Deployment – Cost considerations will likely vary considerably by system. Therefore, it may not be possible to generalize the Model Deployment findings. In the case of the Arizona 511 system enhancement, the cost of enhancing the system far exceeded the initial cost to develop it (assuming the cost of the HCRS data engine are excluded); costs associated with the menu design and voice recognition were significantly (61%) higher than estimated; and the cost to operate the system during the post-deployment year (about $293,000) represented a significant proportion (17%) of the total cost of the Model Deployment.

9.4.2 Usage and Survey Analysis-Related Conclusions

Target Marketing to Users of New Information – For a traditionally roadway-oriented system, the addition of multi-modal information is not enough to stimulate significant usage of that information. The analysis of call logs indicates that 91% of all information requests during the post-enhancement period were for roadway information. The user survey results were consistent with this finding. Not unexpectedly, when a 511 system has an established roadway-oriented user base, simply adding multimodal information and carrying out general (radio ad) and highway-oriented marketing (DMS) is not enough to attract significant numbers of transit, airport, or tourist information users.
DMS Marketing is an Effective Way to Reach Roadway Travelers – Of the several marketing activities conducted by ADOT, a week-long, 24/7 statewide DMS campaign had by far the greatest impact. DMS was cited by 34% of survey respondents. The other marketing activities (ADOT appearances at the State Fair and other events; a two-week period of radio advertisements) were much less effective, with 2% and 9% of respondents, respectively citing these activities. The impact of the DMS campaign was also reflected in call volumes, which skyrocketed during the period. During the DMS campaign, call volume increased three fold, with 96% of users being new to the system. Expectedly, DMS was especially effective in reaching en-route highway travelers. A much higher percentage of (83% versus the annual average 53%) of 511 calls were made by cell phones during the DMS advertising. These findings suggest that deployers can cost effectively take advantage of their own assets, such as DMS, to raise awareness of 511.
Don't Assume A Regional, Multi-Modal 511 System will Replace Transit Agency Telephone Information – Major national questions about how 511 systems will relate to transit customer information telephone systems (both IVRs and human operators) remain. Progress was made in the Phoenix area in determining how the systems will mesh (and the general answer is that both systems will have transit information) but a definitive relationship/overall strategy has not been identified.
Vigorously Build Support Among Local Agencies for their Input of Roadway Information – Arterial street data capture is a major challenge. Most cities and counties do not have good real-time information (congestion and incidents). They are often resistant to inputting their planned event information, because they already enter it into their own systems and do not want to perform "dual entry", or because they do not do it at all and do not have the time to start doing it. ADOT has managed to significantly increase their capture of arterial street information by more intensely monitoring police scanners, but they recognize this approach, in itself, as inadequate. It is labor intensive and does not always garner all the desired information, especially notification of incident clearance. ADOT looks for the upcoming addition of a state police computer-aided dispatch work station to significantly improve the situation.
Build Support for 511 by Emphasizing User-Reported Benefits – Among repeat 511 users, the most commonly cited benefits were saving time/arriving on time (21%); avoiding traffic congestion due to accidents or construction delays (20%); and ability to change route based on information on traffic or road conditions (15%). Only 1% of repeat users indicated that 511 made their trip more relaxing or easier, suggesting that 511 information is valued for its specific use in travel decisions. Attracting first time users and getting them to try the system again is, obviously, critical to long-term customer satisfaction and realization of 511 benefits. This evaluation clearly indicates that the more callers use the system the greater their appreciation of benefits. So, it is critical that their first experience is satisfactory enough to warrant a repeat use—benefits will snowball from there. Nearly all (95%) first-time callers surveyed were willing to try the system again. Also, although definitive conclusions were not possible, the analysis of system usage during the initial roll out of voice recognition—a time when many complaints were received—suggests that callers, existing or new, were not quick to dismiss the (at that time flawed) 511 system.
Don't Assume Transit Information on 511 Will Impact Mode Choice – 511 users appear resistant to mode choice changes, although this may be significantly influenced by the fact that few transit users appear to be using the Arizona 511 system. At least among traditional travelers (auto users), it does not appear that having some transit information on 511 is likely to contribute to use of transit.
Understand that for Many or Most 511 Calls, No Change in Travel Plans Will Result – Drivers do make changes in their travel plans based on 511 information, but such changes are the exception; in most cases the information on 511 does not impact their travel plans. It is not clear whether this is because the information shows the intended travel route as trouble free, or whether roadway conditions sufficiently severe enough to warrant a change are uncommon. The specifics of the Phoenix travel environment could have much to do with these findings. When callers decide to make a change to their trip in response to 511 information, the most frequent changes are taking a different route (12%), changing lanes (12%) and slowing down or changing speeds (9%).

< 8.0 | Table of Contents | Appendix A >

FHWA Home | Feedback