Final Report

Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System

5.0 511 Users and User Perspectives

This section presents the results of the telephone survey of 511 callers described in Section 3.2. The section addresses the following questions:

• When did callers start using 511 and how frequently do they call?
• Who uses 511?
• What is their occasion of use?
• What type of phone are callers using to access 511?
• What content do they seek?
• How satisfied are callers with the information they obtain?
• How does the service benefit them and impact their travel?
• What is their experience with the user interface?
• What improvements would they like to see to the service?

The reader is reminded that the results presented in Section 5.0 are weighted data. As noted previously in Section 3.2, and further elaborated in Appendix B, weighting is a standard survey practice used to reduce bias and improve the precision of estimates. Thus, the number of respondents (n) in the tables below is the weighted frequency for a specific survey question. Appendix C presents the unweighted and weighted frequencies for each question.

5.1 Frequency of Use of 511

The proportion of callers who were new to the service at the time of the survey was large. Figure 5-1 shows that a third of the callers report using it for the first time, and 13% had started using the service within the past month. Only 13% had first used 511 over a year ago, even though 511 has been available in Arizona since March 2002. Although the enhanced service was launched in December 2003, it was not heavily advertised until May 2004. This might explain why almost half (46%) of callers had started since then.

The number of unique individuals using 511 can be important to the future of the service. As long as usage of the service continues to grow, having a high percentage of new callers is a healthy sign, because some of them will become repeat callers. If the overall service plateaus or declines in overall usage, this might signify that new callers are trying the service but are not being converted to regular users at a significant rate. As noted in Section 4, the overall trend has been upward, which bodes well for the turnover in users of the service. Except for peaks of usage related to extreme weather conditions, calls made to 511 in the fall of 2004 were about 20,000 higher per month than during the same period in 2003.

Figure 5-1. Percent of Callers by How Long Ago They First Used 511 (n=404)

There is another reason for noting the proportion of new and repeat callers. Previous research¹¹ suggests that new users of a traveler information service may differ from repeat users in how they use the service, in their expectations, and in user characteristics. Thus, differences between repeat and new users will be noted where appropriate in Section 5.0.

Figure 5-2 shows the distribution of percent of repeat callers according to the number of calls they made to 511 in the previous month. Over half (53%) made one to three calls, 31% made four to ten calls, and 16% made eleven or more calls. As noted in Section 4.1.2, there are several reasons that the survey-reported percentage of new users is lower than that found in the usage analysis, including the fact that repeat users were much more strongly represented in the survey sample.

When callers were asked how they first heard about 511 (Table 5-1), the electronic signs (dynamic message signs) used by ADOT to promote the enhanced system for a short while in the spring of 2004 were most frequently mentioned (34%). Friends and co-workers influenced 15% of the respondents to use 511, and websites, principally ADOT's, accounted for an additional 11%. Radio was cited by 9% of the respondents. This could reflect the short radio ad campaign conducted by ADOT in May and early June 2004 or possibly local news coverage of the 511 system (some television and newspaper coverage was observed at the enhanced system roll out in late 2003 and early 2004). Interestingly, 511 road signs were cited by 3% of the respondents even though they had not been deployed by the time of the survey. Perhaps some callers had become aware of 511 by seeing road signs in neighboring Utah, or perhaps they were thinking about the dynamic message signs when they said road signs.

Figure 5-2. Percent of Repeat Callers by Number of Calls Made to 511 in Last Month (n=277)

Only 5% of repeat callers reported using 511 in a different state. At the time of the survey, 511 was available in 24 locations in the U.S., including the neighboring state of Utah. Utah was one of the most frequently mentioned of the sixteen states identified by respondents. All but two of the states named are west of the Mississippi, which may reflect regional patterns of travel by these 511 callers. However, respondents' recall of exactly where they had used 511 was not very reliable. Of the sixteen states named, only nine had 511 service at the time of the survey. Nevertheless, the results indicate that a small portion of the callers are aware of 511 being available outside Arizona and have used it. The finding can be viewed as an indication that 511, if not yet a national brand, is starting to be recognized by callers as more than a service of a single state.

5.2 Who Uses 511?

This section examines the characteristics of Arizona 511 users in terms of their place of residence and demographic and socioeconomic attributes. In general, the data reveal that residents of Phoenix are the dominant users of the service, but out-of-state residents and those from more rural parts of Arizona are well represented among first-time users. Men and women use 511 in about the same numbers. Measures of age, education, and income reveal important differences between repeat users and those using 511 for the first time. First-time users tend to be older, less educated, and have somewhat lower income than repeat users. These overall characteristics of 511 users are discussed in more detail below.

The distribution of all respondents' zip codes of residence on the day of the survey (Table 5-2) shows that a considerable majority of callers live in the Phoenix metropolitan area as defined by Maricopa County (73%), a higher concentration than the 61% of the state's population that live there.¹² However, when residence of repeat users is compared to first-time users, striking and statistically significant differences in the distribution of callers by geographic residence were observed (p-value < 0.0001¹³). Eighty-five percent of repeat users are from the Phoenix area versus 48% for first-time callers (p-value < 0.0001). Tucson accounts for very few users among either group. Of the 52% of callers that are non-Phoenix first-time users, 6% are from Tucson, 23% are from other parts of Arizona (i.e., locations other than Tucson) and 23% are from out of state. Arizonans who reside outside Phoenix or Tucson may have recently become aware of 511 and tried it for the first time. Out-of-state callers may be vacationers or "snowbirds" making their annual trip to Arizona.

Turning to gender and age of users, men hold a slight majority among both repeat and first-time users, 52% and 54% respectively, but the difference between the two groups is not statistically significant (p-value = 0.829). More important than gender are differences in the age profile of the two groups, where there are statistically significant differences (p-value < 0.0001). As shown in Figure 5-3, first-time users tend to be somewhat older than repeat users. Indeed, 24% of first-time users are 65 or older, compared to only 2% among repeat users (p-value < 0.0001).

Figure 5-3. Percent of Callers by Age Group

Household income and education of the surveyed individual were the socio-economic characteristics measured in the survey, and repeat users tended to have higher incomes and education levels than did first-time callers (Table 5-3). There is a significant difference for income (p-value = 0.007), but not for education (p-value = 0.154). The median education level of callers is two years of college, but only 7% of first-time users have postgraduate degrees compared to 15% among repeat users (p-value = 0.0545). Nearly twice as many first-time callers (41% vs. 21%) have a household income less than $40,000 (p-value = 0.0036), although slightly more (but not statistically significant) first-time callers have a household income exceeding $100,000 (p-value = 0.2048). Lower income along with higher age and somewhat lower education levels may reflect a higher concentration of retirees among the first-time users, although employment status was not included in the survey. Conversely, the lower age and higher income of repeat callers suggest that they tend to be working age and probably more likely to use 511 on a regular basis for work-and school-related travel.

5.3 Occasion of Use and Type of Phone Used

This section characterizes the occasion of use of 511 in Arizona by describing the caller's situation when he or she placed a call to obtain information. The data include the caller's location, trip purpose, and mode of travel. This section also profiles the type of phone—landline or cell phone—callers are using to access 511.

As shown in Table 5-4, a significant 61% majority (95% confidence interval of 50% to 72%¹⁴) of 511 callers placed their calls from a private vehicle as either a driver or passenger. Another significant 30% of callers (95% confidence interval of 24% to 37%) used 511 from home. A surprisingly small, though statistically significant, 6% (95% confidence interval of 4% to 9%) accessed the service while at work, suggesting that workers may not be checking on travel conditions for their commute home before setting out, but instead are calling from their vehicles.

Trip purpose, as shown in Figure 5-4, reveals that 511 is providing callers with information for a broad range of trip purposes. Slightly more than half (52%) of the calls were for information for trips for family or personal reasons, and for social, religious, or recreational purposes. The remaining 48% of calls were for information related to commute or other trips for work or school. The split in use of 511 based on trip purpose is consistent with the system usage data based on day of week presented in Section 4. Data in Section 4 showed that about half of the calls were placed from Monday through Thursday and exhibited the morning and afternoon peaks typical of commute patterns. The other half of calls occurred on Friday afternoon and evening and on Saturday and Sundays—indicative of weekend travel not related to work and school.

Figure 5-4. Callers by Purpose of Trip When Called 511 (n=378)

Another aspect of occasion of use of 511 is the timing of the call. The question of interest is whether users are calling 511 pre-trip to plan their trip based on what they hear about travel conditions and other information, or are calling 511 while en route to make trip adjustments if the information so warrants. The survey revealed that a 65% majority (95% confidence interval of 58% to 72%) of 511 users made their calls while traveling (Table 5-5), which suggests that real-time information would be important to most 511 users. An additional 23% called the same day before setting out, and only 11% used 511 to plan a trip a day or more in advance, perhaps for a long-distance trip to check potential construction delays or rural weather conditions. However, there is no significant difference in the percentage of callers who called within 15 minutes before, more than 15 minutes but on the day of the trip, and a day or more in advance of trip.

When asked about their mode of travel for the trip they were taking (Figure 5-5), the vast majority of callers were either drivers (87% with a 95% confidence interval of 83% to 92%) or passengers (5% with a 95% confidence interval of 3% to 8%) in a private vehicle. Commercial vehicle operators accounted for 8% of callers (95% confidence interval of 4% to 12%), with two-thirds of them driving "semi's." Arizona's 511 system is used by very few transit riders, as only 1% of the callers reported their trip was by bus (not a statistically significant percentage), which confirms the findings in Section 4.0 that road information is the type of information that callers are accessing.

As shown in Table 5-6, 56% of all callers used cell phones to access the 511 service, although that was not a significant majority (95% confidence interval of 50% to 62%). On the other hand, relative to land line callers the distribution of cell phone use differed significantly between repeat and first-time callers (p-value <0.0001). Among repeat callers, 65% indicated they were using a cell phone for the trip when intercepted for the survey. Moreover, when asked about all the calls they have made to 511, 81% of repeat users said that they usually call from a cell phone. Cell phone usage among repeat callers correlates with the high percentage of callers who said they called from a private vehicle (61%) and while driving (65%). In contrast, among first-time callers, only 37% were using a cell phone when they called 511 at the time of the survey. This may be because users' first exploratory call to 511 is more often a general familiarization call, made from home, and not necessarily pertaining to a specific trip. It may also be related to the fact that first-time callers tend to be older and to have somewhat lower incomes than repeat callers, which could indicate lower cell phone ownership.

Figure 5-5. Mode of Travel of Trip (n=382)

The survey data imply that cell phone usage is higher than that seen in the wireless vs. wireline usage analysis in Section 4.0, which shows only about 40% of the calls to 511 were from cell phones during the month of November 2004 when the survey was in the field. One explanation might be that first-time callers, who reported lower usage of cell phones (p-value < 0.0001), had a higher incident of refusal for participation in the survey.

5.4 Content Selected

Callers were asked to identify the sections of the 511 menu that they accessed for information for the trip they were taking. The modal split in content selected is pronounced (Table 5-7). Ninety percent of callers selected road information, 8% selected the regional quick reports on roadway conditions, and less than 1% asked for information on buses for the trip they were taking when surveyed. All the bus inquiries were for Phoenix Valley Metro rather than Tucson Sun Tran or other bus systems around the state. When callers were asked if they had ever sought certain types of content on 511, not for just this trip, the numbers rose by only one to three points, and the overall pattern remained the same. Clearly, at the time of the survey, the Arizona 511 service was being used almost entirely for information on roads. These findings are consistent with those of the usage analysis (Section 4.0). Factors which could account for modest usage of 511 for transit, airport, and tourism information are low levels of advertising of these features, seasonality of demand in the case of tourism, and possibly callers' dissatisfaction with the information provided in these areas, leading to low repeat use. The issue of satisfaction is taken up in Section 5.5, but it should be noted that the number of respondents using these content areas is too small for the results to be considered reliable.

For callers requesting road information (Table 5-8), the interstates and freeways accounted for 70% of all roads that callers mentioned. The leading roadways were I-10, a major route through both Phoenix and Tucson, and I-17, the major north-south route between Phoenix and Flagstaff. U.S. 60, between Phoenix and the state border to the east, was the only other road that received mentions of 10% or more by callers.

5.5 Satisfaction with 511 Service

A key objective of the evaluation is to measure user satisfaction with the 511 service. The enhancements to the service discussed in Section 2.2 were expected to improve the service in ways that would be perceived positively by the traveling public. Thus, a substantial portion of the survey was devoted to assessing satisfaction through various means. One type of measurement was to ask callers about their satisfaction for the particular trip they were taking when surveyed. Callers rated on a scale of 1 to 5 their satisfaction with the information they received from 511 in general and for the specific content areas they selected from the 511 menu. In addition to the particular trip, repeat callers were asked about their satisfaction for all the past calls they had made to 511. Another satisfaction measure involved the comparison of 511 to radio traffic reports, another popular form of traveler information. Satisfaction was also measured by using a series of statements to which they could agree or disagree, and whether they would use the service again and recommend it to a friend. This section discusses the findings from these various measurements.

5.5.1 Satisfaction with Information for the Trip Taken

Table 5-9 presents callers' level of satisfaction with the quality of information they received for the menu options they selected for the trip they were taking when intercepted for the survey. For all the content areas, over 70% of callers were satisfied with the information they received for the particular trip. For both road information and Quick Reports, over 20% of callers expressed dissatisfaction with the information. The potential sources of that dissatisfaction are explored later in this section. (The responses for use of bus, airport, and tourism information were too few (? 2) to calculate meaningful percentages, and, thus, are not shown.)

5.5.2 Repeat Callers Satisfaction with 511

As noted in Section 5.1, two-thirds of the surveyed callers had used 511 prior to the survey. Based on their weeks or months of experience with 511, repeat callers can provide valuable insights into how 511 is performing for those users who have made the decision to keep using the service, and, thus, Section 5.5.2 focuses on repeat callers' satisfaction with the service.

Table 5-10 examines levels of satisfaction with specific aspects of menu items that repeat callers may have used throughout their entire history with 511. Whereas satisfaction with information for the specific trip taken when the caller was intercepted was examined in Table 5.9, Table 5.10 considers all previous calls and the repeat callers' satisfaction with the quality of the information received. Many of the information types listed in Table 5.10 were added in the enhancement process, such as quick reports, bus, airport, and tourism information. Assessing how well those features are performing in the eyes of users is one objective of the evaluation.

Because the vast majority of repeat callers to 511 are seeking road information, their satisfaction with this part of the menu is very important to the success of the service. In general, most repeat callers said that they were very satisfied or somewhat satisfied with the road-related information. Repeat callers were especially pleased with the information on traffic incidents, congestion, and construction. For these content areas satisfaction levels (very and somewhat) were 74% (66% to 81%), 78% (70% to 85%), and 86% (81% to 91%) respectively, which were all a statistically significant majority. Road information also garnered high marks from repeat callers when they were asked to express agreement or disagreement with the following statements:

• "The traffic information I get from 511 is accurate and timely." 57% agreed and 25% strongly agreed for a total of 82% (95% confidence interval of 76% to 88%).
• "511 covers the areas and routes I'm interested in." 50% agreed and 47% strongly agreed for a total of 97% (95% confidence interval of 95% to 99%).¹⁵
• "I prefer to get information on segments of roads, like I-10 from Phoenix to Tucson, rather than for an entire road, like all of I-10 in Arizona." 30% agreed and 54% strongly agreed (84% with a 95% confidence interval of 76% to 92%).

Quick report summaries were one road-related feature for which a large percentage of repeat callers did not indicate satisfaction (54% with 95% confidence interval of 43% to 65%), but most of those (46%) were simply ambivalent about the feature. It is interesting to note that the quick reports were added to the service based upon interest voiced by focus group participants during the design stage of the service in 2002. Perhaps a sizeable portion of users of road information prefer to access specific roads of interest, like the route of their daily commute, rather than try to get the information through the quick report summaries.

Another area of ambivalence was weather-related roadway information, for which 37% (95% confidence interval of 27% to 47%) of the repeat callers using that information were either ambivalent or dissatisfied (32% were neutral and 5% were dissatisfied.) As noted in Section 2.2, the roadway segment weather enhancement was not implemented, although some road weather information has continued to be provided through HCRS. Fortunately, weather concerns are not the primary reason for calling 511. In a related question, repeat callers were asked to rate their agreement with the following statement: "I call 511 most often when the weather is bad." Only 27% (95% confidence interval of 20% to 35%) agreed or strongly agreed with that statement.

The survey revealed that Arizona 511 is used very little to obtain information related to buses. There are several possible explanations for the low usage: travelers' preference for their own vehicles, the extent and quality of information on buses, or lack of awareness among travelers, especially transit users, about the availability of information. With only five repeat callers reporting use of the information on bus service disruptions, their 100% satisfaction level cannot be considered a reliable indicator. Of the four repeat callers who had used the feature to transfer from 511 to a bus agency operator, three were neither satisfied nor dissatisfied with that feature. Other questions in the survey provide some additional insight about repeat callers who have accessed bus information on 511. Five users responded to the following statements:

• "511 covers all of the bus services I'm interested in." One agreed or strongly agreed, three were neutral, and one stated he/she did not use that aspect of the service.
• "On 511 I can quickly get through to a live operator to help plan my trip." One agreed, one strongly disagreed, and the rest had not used that feature.

In the area of tourism and airport information, only six repeat callers reported using those features, and, thus, the results should be interpreted with caution. None of the repeat callers expressed dissatisfaction with these features, but they expressed higher satisfaction with airport information than tourism information. Of the five that responded to satisfaction levels with airport information, three were somewhat satisfied and one was strongly satisfied. Three of the five respondents on Grand Canyon information were satisfied, but only one of six expressed satisfaction with the ability to transfer to the Arizona Office of Tourism. One possible explanation for dissatisfaction with the tourism transfer option is that, having heard the opening greeting identify "tourism information" as one of the 511 options, they may have expected actual tourism content on the 511 system itself.

5.5.3 Comparison of 511 to Radio Reports

Radio is the most frequently used source of traffic information throughout the U.S.¹⁶ It serves as a benchmark from which advanced traveler information services such as 511 can be measured. Indeed, many Arizona 511 users also report using radio traffic reports. Seventy-two percent of repeat 511 callers reported they used radio traffic reports as a source of travel information, and their median usage was 13.4 times per month. Figure 5-6 shows callers' comparison of the quality of information on 511 with radio traffic reports for those using both radio and 511. Forty-three percent felt that radio was worse than 511, only 28% thought radio was better, and 29% felt it to be about the same quality of information. Thus, one can conclude that, if not quite a majority, many callers find 511 a better alternative to radio, and a substantial portion think that it is no worse. It is not clear why 511 users who find 511 superior to radio would continue to use radio as a source of traveler information. It may be that they do not specifically seek out information via radio but are still exposed to radio traffic reports through their regular radio listening or are "information seekers" who like to have as much information as possible, preferably from different sources.

Figure 5-6. Repeat Callers' Comparison of 511 and Radio Traffic Reports

5.5.4 Willingness to Use Service Again and Recommend to Friends

Willingness to use the 511 service again is another reflection of satisfaction with the service. Table 5-11 shows that almost all callers, repeat and first-time callers alike, reported that they are likely to phone 511 again. They are also willing to recommend the service to a friend. For both these measures, first-time callers were slightly less likely to do so than repeat callers, though this was not statistically significant (p-value = 0.0996). This is not surprising because most repeat callers have shown a greater commitment to the service through their repeated usage. The fact that nearly all first-time users would use the system again in spite of the fact that 31% cited no particular benefit (see Section 5.6) may suggest that first-time callers are willing to withhold final judgment until they've had more experience with the system. It may also suggest that they derived some diffuse, non-specific benefit.

Although few users indicated that they would not use the system again or recommend it, the reasons given by those who said no can be instructive, and these are shown below.

The reasons for not calling 511 again included:

• Got no information
• Information not timely
• Information from Highway Patrol is better
• Information not accurate
• Voice recognition didn't work well
• Hard to use or navigate
• Dangerous
• Won't be going to that area again
• Didn't help; Wasted my time
• No information for sites outside the valley.

Explanations of why they would not recommend 511 to a friend included:

• Inconsistency
• Not helpful
• Difficulty in using
• Needs improvement
• Not enough information outside Phoenix
• Doesn't work
• Too confusing
• Didn't get any information
• Prefer to talk to a person
• Hard to navigate
• Not reliable; not what caller was experiencing on road
• Not satisfied enough to recommend
• Only useful on the road. At home, suggest the Web.
• Not good enough.

Although those who would not use the service again or recommend it to a friend are a small minority, the source of their dissatisfaction is helpful for understanding where the service may be falling short for some users. These results could be helpful as changes and upgrades to the service are considered in the future.

5.6 Benefits and Impacts of Using 511

The ultimate objective of 511 is to have an impact on the callers who receive the information. What benefits did callers believe they had received from 511 in Arizona? What, if any, changes did they make after receiving the information?

To assess the benefits, the respondents were asked to identify benefits they perceived from using 511. The question was open-ended, in that respondents were not provided with a predetermined list of benefits and asked to respond. Instead, the benefits were volunteered by the respondents themselves. Table 5-12 shows the types of benefits that the respondents said they derived from 511, based upon the study team's classification of the verbal responses. The approach was to identify the principal benefit that the respondent was articulating in the response to the question. When the caller's response included more than one thought about benefits, the study team selected the dominant theme of the comment. For example, "rerouting myself and getting to my destination on time" was designated as emphasizing the time-saving aspect of having the information.

The most notable aspect of Table 5-12 is that repeat callers and first-time callers differ markedly in the benefits they say they received from 511 (p-value <'0.0001). Not surprisingly, the more experienced repeat users identified certain benefits in higher proportions than the new users. Even more significant is the fairly large percentage (31% with 95% confidence interval of 23% to 42%) of first-time callers who perceived no benefit at all. However, as indicated in Table 5-11, the lack of a specific perceived benefit does not appear to impact most first-time users' willingness to try the system again or recommend it to a friend.

Among the perceived benefits, having the ability to access current road and weather information was cited most frequently by both repeat (33%) and first-time users (28%). The implication is that other sources of traveler information are either not as accurate or accessible as 511. Having the information leads to other benefits that users identified. The ability to either save time or arrive on time was recognized by 21% of repeat users versus only 6% of first-time callers (p-value = 0.0014), suggesting that only with the experience with 511 gained by repeat usage over several weeks or months can those benefits be fully appreciated. Similarly, avoiding traffic congestion and changing routes were identified by 20% and 15% of repeat callers respectively and only 2% and 5% of first-time callers (p-values of < 0.0001 and 0.0172, respectively). A few callers emphasized the "serenity" benefit of 511, feeling it made travel more relaxing or easier. Finally, in the categories of "other" and "satisfied," callers did not cite specific benefits but, instead, made comments about the user interface or simply expressed general satisfaction with the service. These types of comments were more prevalent among first-time callers than the repeat callers.

5.6.1 Impact of 511 on Travel Decisions

To assess the impact of information from 511 on travel decisions, the survey provided callers with a list of possible changes they had made as a result of calling 511 for the particular trip when they were intercepted for the survey. Table 5-13 presents the types of changes and the percent of respondents who made the change. Respondents could report more than one change to their trip, and the total changes mentioned across all respondents numbered 168. Based on the small percentage of respondents making any particular change, one can conclude that most callers did not feel compelled to change their plans based on information on 511. This should not be construed negatively, because under most circumstances travel conditions can be considered "normal," being free of incidents or other adverse conditions that would warrant a change in plans. For those callers who did make a change, taking a different route was the most frequent type of change mentioned (12% of callers). It is not known whether the decision to change route was made en route or before setting out. Callers changed their behavior while on the road by changing lanes (12%), slowing down or changing speed (9%), or making stops along the way (4%). Pre-trip changes were less frequent, but included change in departure time either earlier (2%) or later (2%) or canceling the trip entirely (1%). No respondents indicated that they took a different bus than the one planned.

By providing multimodal information to travelers, one potential benefit of 511 systems is facilitating use of transit and attracting travelers away from their private vehicles. For the particular trip the caller was making when intercepted, only 1% of respondents reported a change in mode. To explore whether callers would consider taking a bus based on traveler information, repeat callers were queried about their agreement with the following statement:

"I am more likely to take the bus due to information on 511."

Those who disagreed (34%) and strongly disagreed (22%) with this statement significantly outnumbered those who agreed (3%) or strongly agreed (<1%) (p-value <0.0001). These results could be interpreted as simply an entrenched commitment to use of private vehicles. An alternative explanation could be that most callers to 511 have not yet heard information that is so dire to cause them to consider switching from their normal mode of travel.

5.7 Customer Experience with the 511 User Interface

This section deals with the 511 caller's experience with the mechanics of 511. This includes the interactive voice response (IVR) feature with voice recognition, as well as menu design, ability to understand the information, and ability to reach 511 without busy signals.

Three aspects of the user experience were explored though questions that were statements to which the respondents expressed their level of agreement. These are shown in Table 5-14. Only repeat callers responded to these questions in the survey.

A significant 71% majority of callers (95% confidence interval of 63% to 79%) are able to navigate easily through the 511 menu. However, 21% express a level of disagreement. Whether their concern is a result of the structure of the menu or perhaps difficulties with the IVR cannot be determined from this question.

Callers appear to have no problem with busy signals on 511. One enhancement to the 511 service was the expansion of the capacity of the system to serve 96 simultaneous calls, and that expansion has proved satisfactory to all callers surveyed.

Ninety-four percent of repeat callers (95% confidence interval of 90% to 98%) indicate that they can easily understand the information being delivered on 511. On the other hand, as noted below, concern was expressed about the system understanding the caller.

5.7.1 Voice Recognition Feature

One major enhancement of Arizona's 511 service was the conversion of the system to voice recognition. The expectation was that enabling callers to speak their requests would be an easier and more convenient feature compared to the previous push button (touch tone) interface. After voice recognition was launched in December 2003, ADOT received considerable negative feedback through its 511 comment line. Consequently, ADOT devoted more effort to improving the IVR, and also made the alternative touch tone interface option more comprehensive and prominent. The survey, occurring ten months after voice recognition was instituted, provided an excellent opportunity to assess callers' experience with voice recognition and phone button access.

Callers were asked what method—voice recognition, phone buttons, or both—they used to make menu selections for the trip they were taking and how satisfied they were with the method. Table 5-15 shows that while voice recognition was used exclusively by repeat callers more than 5 to 1 (224 vs. 38) and by first-time callers over 2 to 1 (81 vs. 32), the experience with phone button access tended to be more satisfactory. For example, of repeat callers using one method exclusively only 30% said they were very satisfied with voice recognition whereas 72% said they were very satisfied with phone buttons (p-value < 0.0001). Among repeat callers using both methods, a significant percentage of callers (34% with 95% confidence interval 14% to 62%) were satisfied with voice recognition, but 66% (95% confidence interval of 38% to 86%) were dissatisfied. The overall propensity of callers to use voice recognition regardless of their satisfaction with it may be because it is the default option.

First-time callers generally used only one method, apparently not choosing to experiment with the alternative method on their first call to 511. Among this group, 46% of voice recognition users expressed satisfaction, but nearly one-half expressed dissatisfaction, whereas among phone button users 90% were satisfied. Although a smaller proportion rated phone buttons very satisfactory than did repeat callers, the difference was not significant (p-value = 0.1051).

As indicated in Section 5.2, 24% of first-time callers were age 65 and over. It is interesting to note that in a focus group study of Arizona seniors conducted in 2004,¹⁷ seniors using 511 for the first time were frustrated with the voice recognition feature. They reported that it did not understand their requests or gave them the wrong road information. The difference in dissatisfaction with voice recognition between first-time callers versus repeat callers found in the survey, although not statistically significant, may be a reflection of the large proportion of seniors among the new-user group (given the findings of the earlier focus group study).

One inference to be drawn from these results is that as callers become more experienced in using the 511 system, some of the initial difficulties with voice recognition that first-time callers might have encountered go away. Nevertheless, even among repeat callers, a sizeable portion still find the voice recognition feature unsatisfactory in some way.

Callers who were somewhat or very dissatisfied were asked why. Their explanations were recorded during the telephone survey and later coded by the study team into major types of reasons. As shown in Table 5-16, the major complaint was that the system misunderstood the caller's request, and related to that, gave the wrong information. Those two reasons accounted for 56% of repeat callers and 82% of first-time callers who were dissatisfied with the voice recognition feature. Another reason cited by a third of the repeat callers was that voice recognition was especially sensitive to background noise. Such sensitivity could lead to the system not understanding the request, of course. A small number of callers felt the menu options were unsatisfactory in some way. For example, one first-time caller knew the desired road by name and not by number, but his/her perception was that the system asked for the number. The system in fact says "you can say the roadway name or number." A number of "other" type of explanations were offered by first-time callers, such as "it seemed difficult to use" or "it was okay but I got cut off," but such remarks were less frequent among repeat callers who had a more definitive complaint based on longer experience with the system.

To further explore the nature of callers' difficulties with voice recognition, those who had expressed dissatisfaction were asked under what circumstances their problems occurred. Table 5-17 reveals that the majority of problems occurred on cell phones (versus landline) and while holding the receiver (versus speakerphone). However, a quarter of the repeat callers, by virtue of their weeks or months of experience with the voice recognition features, reported having problems on both types of phone and talk modes. Thus, there would appear to be either fundamental problems with the voice recognition technology, a need for further education of 511 callers on how to use it most effectively, or both. Focus groups or other types of research with actual users might help pinpoint the problems and potential solutions.

5.8 Potential Improvements to 511

The survey provided an opportunity to identify areas of improvement by asking respondents to rate several aspects of the 511 service which could be enhanced or improved. Shown in Table 5-18 are five specific improvements about which respondents were questioned and the priorities—high, medium, and low—that callers assigned to five potential improvements. In addition to these questions, respondents could also volunteer other areas of improvement that were important to them, and their recommendations are reported in Table 5-19.

The results reveal a consistency in the responses of repeat and first-time callers, with the five types of improvements ranked in the same order by both groups. The top-ranked requested enhancement was improvements to the speech recognition feature. Note though, that the percentage of repeat callers requesting this enhancement was not statistically significantly higher than those requesting more detailed information on general traffic congestion levels and delays (p-value = .2579), and among first-time callers, the percentage was not statistically significantly different from those who requested more detailed information on general traffic congestion levels and delays and those who requested the addition of more roads (p-value = 0.0922). The least sought improvement was information on bus arrival time. Eighty-three percent of repeat callers and 75% of first-time callers rated that improvement low in their priorities, which was not statistically different (p-value = 0.0783). It is interesting to note that the bus arrival time feature and the information on roads in neighboring states were enhancements originally planned for the Arizona 511 Model Deployment, but they have not yet been deployed.

From the standpoint of satisfying the priorities of current 511 customers, these findings provide direction for future enhancements to the service. On the other hand, if 511 is to be as multimodal as possible and reflect the information needs of other segments of travelers (bus riders in particular) who might not be using 511 very much right now, some enhancements should be considered that may not be high priorities among current 511 callers. With transit riders are virtually absent among current callers, it is not surprising that the addition of bus arrival time would be a low priority.

Respondents also were given the opportunity to volunteer other improvements that they were interested in seeing made to 511. Their verbatim comments were coded into several general categories as shown in Table 5-19. Respondents were also asked to assess whether the suggested improvement was a high, medium, or low priority. Eight-five percent of the repeat callers who volunteered an improvement considered the improvements in Table 5-19 to be high priorities as did 98% of the first-time callers (p-value = 0.0223).

Whereas the repeat callers most often mentioned (35%) improvements related to information on traffic and road conditions, such as more timely or more detailed information, the most frequently volunteered (25%) improvement among first-time callers was the ability to talk to a live operator. Perhaps their expectation in calling the service for the first time was that they would be getting information from a live operator, or they would have the option of transferring to an operator (for traffic and road information) if they had difficulty with the system or wanted more detail. Setting callers' expectations appropriately would appear to be an important activity for future marketing.

5.9 Conclusions

The conclusions to be drawn from the results of the telephone survey of 511 callers are:

• A significant portion of callers—33%—are new users of the 511 service. With an overall upward trend in usage, the large percentage of new callers is a healthy sign as some of them will convert to repeat callers if their experience with 511 is satisfactory.
• Only 21% of those surveyed had used the system prior to the May 2004 marketing campaign in which ADOT used its dynamic message signs to promote the service for a few days. Those signs were the most frequently mentioned source of their first awareness of 511.
• Phoenix residents dominate usage of 511 (73%), and few callers (2%) are from Tucson. The low level of usage by Tucson residents is not surprising. As noted in Section 2.2, 511 includes relatively little information specific to Tucson, and there has been little to no Tucson-focused marketing.
• Over half of repeat callers make three or fewer calls per month to 511, indicating that usage of the service is not dominated by commuters seeking travel conditions on their daily commute. Indeed, more than half of the survey respondents were making a trip not for work or school but for other purposes.
• First-time callers tend to be older, less educated, and have somewhat lower income than repeat users, perhaps indicative of a high proportion of retirees among the new-user group. The repeat callers are more likely to be working age and more technology-savvy and perhaps more motivated to learn to use 511 effectively for the commute and other trip-making.
• The situational profile of usage indicates that a majority of calls are being placed en route on a cell phone in a private vehicle. The content callers are seeking is almost exclusively (90%) road-related information, mostly for interstates and freeways (as opposed to state or U.S. highways).
• Overall satisfaction with content on 511 is high, more than 70% among both repeat and new callers. Because most callers are seeking road-related information, the high positive satisfaction ratings for most road-related content speaks well of the enhancements made in that area. One surprising result was that 54% (95% confidence interval of 43% to 65%) of repeat callers were ambivalent about the value of the quick report regional summaries or dissatisfied, which suggests that there may be some more work to do on how that feature is implemented or promoted.
• Experienced users feel more strongly than inexperienced users about the benefits of using 511. Repeat callers perceived benefits in higher proportion than first timers. Moreover, for the call for which they were intercepted, 31% of first-time callers saw no benefits at all versus 7% among repeat callers. To repeat callers, the benefits included the ability to access road and weather information, saving time or arriving on time, avoiding traffic congestion, and the ability to change routes based on information received. Clearly, repeated usage has led callers to appreciate how 511 can assist them in their ongoing travel decisions, whereas new users are drawing on only one call to make the judgment about benefits. With repeated usage, they too should realize how 511 can assist their travel. Thus, it appears that callers need to have a longer term view to realize the full benefits of 511. Indeed, first-time callers seem willing to give 511 another chance, as 95% said they were likely to call again.
• Callers are using 511 very little for bus information. Current usage represents only about one percent of all callers. This is in part a result of the fact that transit usage in Arizona is low (e.g., about 1.2% of all person trips in the Phoenix region) and that there was no significant marketing targeted to transit users. While the small number of respondents prohibits extrapolation of the results beyond the sample itself, it appears that those five respondents have not made full use of the bus information on the service, such as the ability to access a live operator through the call transfer feature. The question must be asked if the low usage of 511 bus content is due to the lack of marketing among bus passengers, the quality of bus information on 511, or some other factor.
• As with bus information, only one or two callers who were surveyed accessed tourism and airport information, respectively. While one could argue that the low interest in tourism information might be due to a seasonal fluctuation, only six repeat callers indicated they had ever used the tourism content. Only five repeat callers had used the airport information. While no dissatisfaction was expressed by callers who had used the airport and tourism information, there are several possible explanations for the low usage levels. It could be that 511 callers are not interested in the information, they do not look to 511 as the source for obtaining the information, or there has been insufficient promotion of these features to reach the audience who might find the information of value.
• The 511 service fares well in comparison to radio traffic reports, typically the most common source of information with which travelers are familiar. Even though 72% of repeat callers used traffic reports on the radio, 43% regard 511 as better than radio reports, and only 28% regard 511 as worse.
• Under normal circumstances, information on 511 will tell callers that they will not need to deviate from their planned route. When callers decide to make a change to their trip in response to information they obtain on 511, the most frequent changes are taking a different route, changing lanes, and slowing down or changing speeds.
• Resistance to changing mode of travel remains strong no matter what information is provided on 511. For most callers, changing mode of travel would mean switching from private vehicle to bus or some other alternative. Less than 4% would consider taking the bus based on information on 511; and over 50% of callers indicated that information on 511 was unlikely to cause them to take a bus. This view might reflect an entrenched preference for their current mode, or it may be simply that information on 511 was not sufficiently compelling for them to change their mode of travel.
• Except for the voice recognition feature, callers are generally satisfied with the user interface. They can navigate easily, have no problems with busy signals, and are able to understand the information being delivered to them on 511.
• Even though voice recognition is used by repeat callers more than 3 to 1 and by first-time callers over 2 to 1, the experience with phone button access tends to be more satisfactory. As callers become more experienced in using voice recognition, some of the initial difficulties that first-time callers may have encountered go away to some extent. Nevertheless, 35% of repeat callers who choose to use voice recognition as their primary means for accessing the service said they were dissatisfied with the feature. The difficulties with voice recognition are reported on both cell and landline phones and both while holding a receiver and speaking handsfree. Thus, there are either fundamental problems with the voice recognition technology, a need for further education of 511 callers on how to use it most effectively, or both.
• Repeat and first-time callers are consistent in the improvements they would like to see on 511, and voice recognition tops their list, followed by more detailed information on traffic. Bus arrival time was the least sought improvement, and information on neighboring states also was given low priority by a majority of callers. These findings suggest that future enhancements to the service could focus on the highest priority improvements to satisfy current callers. On the other hand, to attract a more multi-modal user base, the bus, airport, and tourism content cannot be ignored, even though current callers appear to have little interest in these areas.

ITS Home | US DOT Home | Privacy | Feedback

United States Department of Transportation - ITS