Final Report
Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System
6.0 Testing the Hypotheses
This section presents the results of the testing of specific evaluation hypotheses (identified in Section 1.3.3), summarizing relevant findings from the various evaluation analyses. The results are presented in Table 6-1. A total of 12 hypothesized impacts were tested. Six of those impacts are identified as "key" (shown in bold in Table 6-1).
One hypothesis, which was related to arterial street travel times, could not be tested because the enhancement was not completed within the evaluation period. Of the remaining 11 hypotheses that were testable, four were fully supported, two were partially supported, and five were not supported.
Hypotheses are grouped into four areas, corresponding to the four main project objectives pertaining to increasing usage, contributing to high levels of customer satisfaction, promoting mobility and access, and promoting efficiency. Gains were made in three of the four objectives; that is, at least one hypothesis was supported by the evaluation findings. The exception was the area of mobility and access. This is not unexpected since impacts in this area are "secondary" in that they depend not only on successfully implementing new data and features, but also on how and to what extent those features are used by callers. Those factors are not easily influenced in the short term. ADOT and the project partners never believed measurable benefits in these areas were likely, although they do view them as very desirable long term objectives. Generally, within the areas that might impact mobility and access, ADOT and the partners focused primarily on the first, fundamental step: getting quality information into 511.
Table 6-1. Hypothesis Testing Results
("Key" hypotheses are shown in bold type)
| Project Objective |
Hypothesized Project Impacts |
Evaluation Findings |
| Increase usage of the 511 system. |
The addition of a number of new data types will contribute to increased
usage of the 511 system. [NOT SUPPORTED] (KEY) |
- The overall call volume in the post deployment period increased by 74% compared to the same period before the enhancement. However, the old information categories accounted for 91% (Roads) and 2.4% (Transit) of all information requests in the post-enhancement period. The new information categories accounted for 4.3% (Quick Reports), 1.3% (Airports), and 0.4% (Tourism) of all information requests. Thus, it cannot be concluded that new data contributed substantially to increased usage.
|
| Usage of the 511 system will increase as a result of enhanced marketing. [SUPPORTED] (KEY) |
- During the week-long DMS marketing campaign, call volume increased by a factor of 3.1. Ninety-six percent of the callers during the campaign were new users (had not used the system in the last five months) and 86% were calling from wireless phones. The DMS marketing campaign, though short-lived, was extremely effective in publicizing the 511 service to highway users.
- Based on the survey results, the other marketing activities that were implemented (distribution of materials at the state fair and freeway opening ceremonies and radio advertisements over an approximately two-week period) had much less of an impact. Nine percent cited the radio ads, and only 2% of respondents cited ADOT as their source of awareness.
|
| The enhanced 511 system will retain more users. [SUPPORTED] | - Based on system usage data, the overall percentage of repeat users increased significantly from 10 to 19% before the DMS marketing campaign to approximately 27 to 37% after the campaign. In light of the increasing call volumes, the relative low percentage of repeat users indicates that the user base of the system is expanding.
- 95% of first-time users that were surveyed said they would use the system again.
|
| Contribute to high levels of customer satisfaction with the 511 system. |
Users will view the information available on the 511 system as comprehensive
and multi-modal. [NOT SUPPORTED] (KEY) |
- Surveyed callers overwhelmingly (90%) used the system to obtain road information
for the trip they were taking. Very few (<1%) accessed transit, airport,
and tourism information.
- Too few users utilized the transit information to allow any valid assessments of their satisfaction to be made.
- Only 5 repeat callers had ever used the airport information and only 6 had used tourism information. This number was too few to draw conclusions about customer satisfaction with these content areas.
|
| Users will be satisfied with the quality of the information on the
511 system. [SUPPORTED] (KEY) |
- 71% of surveyed callers expressed overall satisfaction with information they received for the trip they were taking. For each of the high-level menu items selected, satisfaction with the quality of information was even higher.
- For all the times they've used 511 the majority of surveyed repeat callers expressed satisfaction with most all the road content. The exception was regional Quick Reports, which was rated a 46% satisfaction level.
- 82% of surveyed repeat callers perceived traffic information on 511 to be accurate and timely.
- In comparing 511 to radio, a source used by 72% of surveyed callers, 43% of callers felt that the quality of traffic information on 511 was better, 29% said it was about the same, and 28% thought the radio was better.
|
| Enhancements to the user interface, including voice recognition, segment-based reporting, and "Quick Reports" will contribute to customer satisfaction. [PARTIALLY SUPPORTED] (KEY) |
- Although voice recognition is used by the majority of callers, it has its drawbacks and receives lower satisfaction ratings than phone buttons. First-time callers are less satisfied with voice recognition than repeat callers. The principal reasons for dissatisfaction were that 511 did not understand the spoken request, 511 gave the wrong information, and background noise caused interference. The problems were not related exclusively to either cell phones or hands free mode, based on survey results.
- Segment-based reporting for road information was a feature with which 84% of repeat callers expressed satisfaction.
- Regional Quick Reports received mixed reaction from callers. Only 8% of callers used the feature for the surveyed trip, but 73% of them were satisfied with the information they received. On the other hand, most repeat callers had at some time tried Quick Reports, but only 46% of them found them satisfactory. It appears that callers prefer to access specific roads of interest rather than go to the regional summaries.
|
| Promote mobility and access by providing new information pertaining to transit, major destinations, and arterial street travel times. |
The addition of transit information, downtown Phoenix information, and tourism information will promote mobility and access. [NOT SUPPORTED] | - Transit and tourism information was used by so few callers (<1%) that no connection can be made to mobility and access.
- Downtown Phoenix information was implemented as one of the Quick Report regional summaries and not as a separate item focusing on events and parking in downtown. Because only 1% of callers asked for the Phoenix Quick Report, no connection can be made about this specific feature for promoting mobility and access.
|
| Providing users with arterial street travel times will allow them to avoid congestion and reduce travel time and travel time variability. [NOT TESTED] | - The arterial street travel time enhancement was not implemented in time for the evaluation, and, therefore, could not be tested.
|
| 511 usage will expose travelers to transit information and encourage consideration of transit as an alternate mode. [NOT SUPPORTED] | - Only 1% of the callers reported that they changed mode based on the information from 511 for the trip they were taking. In addition, only 4% of callers indicated they would be more likely to take the bus due to information on 511, when asked without regard to the specific trip they were taking. Private vehicle use is the dominant mode of travel (92%) among 511 callers. The information they've heard on 511 has not yet been compelling enough for any appreciable portion to consider bus as an alternate mode.
|
| Promote efficiency of information dissemination by providing an easily understood menu system, improving data quality, and increasing system capacity.
|
Acceptable system availability/reliability will be maintained through the enhancement process and after the enhancement. [SUPPORTED] | - 100% of the callers agreed that they could get through to 511 without any busy signals. Thus, necessary availability was maintained.
- Line capacity was never exceeded after the Model Deployment. During all but a few peak periods each year, only a small fraction of available capacity was utilized.
|
| Menu system enhancements will minimize the number of unrecognized caller inputs. [NOT SUPPORTED] | - Percentage of information requests that could not be comprehended by the 511 system was highest, at 25%, in January 2004. By May 2004, the system was at the lowest error rate of the year, at 13.8% of all requests. This number gradually increased to nearly 20% by the end of the year. The average error rate over the one year post-enhancement period was 17.8% of all requests.
- In total, slightly over one-third of all calls to the 511 system generated errors. This percentage decreased over the course of the post-implementation period, starting with almost 54% of calls including errors in January 2004 dropping to 30% to 37% over the last nine months of the year.
|
| The efficiency of information dissemination will be promoted through
enhanced arterial street data capture, data entry operator training, and
data quality control procedures. [PARTIALLY SUPPORTED] (KEY) |
- The volume of arterial street information input by ADOT increased dramatically as a result of their intensified monitoring of police scanners. Annual entries of this type increased from 234 (2% of the total) to 2,763 (14%).
- Information input by Phoenix area cities and counties increased negligibly (from pre-enhanced average of 15 entries per year to 45 entries in the post-deployment year of operations).
- No information was input by Tucson area city and county agencies during the post-enhancement period. These agencies only achieved the ability to do so late in the one-year operational period (2004). They intend to begin doing so sometime in 2005.
- Interviews with ADOT data entry personnel and other ADOT 511 staff indicate that significant changes have been made that they view as enhancing data quality. Refinements were made to roadway information location references. A key new preview function was added allowing data entry operators to see how their input will be translated to a 511 message. The data entry operator's manual has been revised and training has been conducted on new procedures and features.
|
