Final Report
Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System
Multi-Modal 511 Traveler Information System
4.0 Usage Patterns
This section discusses 511 usage based on an analysis of system data. The analysis contains two components. The first considered overall usage patterns, including call volume distributions (by time, day, type of phone, call geographic location), line utilization, and contents accessed. The changes in usage pattern before and after the enhancement are discussed, excluding enhancements not previously available. The second component examines post-enhancement usage under several specific scenarios. Those scenarios were identified based on events that could result in significant impacts on 511 usage, such as forest fires, marketing activities, major traffic incidents, and winter storms.
4.1 General Usage Patterns
4.1.1 Call Volumes
Distribution of daily call volumes during the post-enhancement period (year 2004) is presented in Figure 4-1. The 511 system recorded 670,369 calls in this period, averaging 1,832 calls per day. By comparison, the pre-enhancement period (September 2002 through August 2003) experienced an average of 1,055 calls per day. This represents a growth of 74% in daily overall call volume. As elaborated in Section 4.2.1, the increase in call volumes is attributable to the statewide dynamic message sign (DMS) marketing campaign that occurred in May 2004. The extremely high volumes during the week-long campaign, and the somewhat increased volumes that persisted over the months after the DMS campaign, drove the overall annual increase. There is no indication that the content or user interface enhancements to the 511 system increased usage of the system. In fact, the post-enhancement call volumes for the four-month period prior to the DMS campaign (January – April 2004) are actually about 2% lower than those for the same period immediate prior to the Model Deployment (January – April 2003). The absence of an increase in call volumes following the introduction of new content is not surprising because no 511 marketing whatsoever was conducted prior to the May DMS campaign and almost none of the marketing conducted over the course of the year focused on the new data content.
Of particular note in Figure 4-1 is a number of dates where the call volumes are well over 2,000 calls per day, including several dates in January (3rd), February (3-5, 21-24, 27-29), March (4-6) and again in November (21-22, 28-29) and December (4-6, 29-31). These volumes are correlated with snow in northern and eastern Arizona. Other exceptional dates in June (5, 12-13) and July (16-17) correspond to wildfires in Arizona. The very large spike in call volumes that occurred in early May (with highest daily call volume of 17,265, approximately 9.4 times the annual average daily volume) corresponds to the marketing campaign on dynamic message signs throughout the state. Many of the daily spikes in call volumes are investigated further in Section 4.2, 511 Usage in Special Occasions.
Figure 4-1. Post-Enhancement Daily Call Volumes
Figure 4-2 compares the monthly call volumes of the pre- and post-enhancement periods. The highest volume, in May of the post-enhancement period, was contributed by ADOT's week-long promotion of 511 on dynamic message signs statewide. Other than the spike in May, the general trend is an increase in call volume toward the end of 2004, ending the year with two to three times the usage of the pre-enhancement period.
Call Volumes by Time of Day and Day of Week
Figure 4-3 presents the call volume distributions by day of week for pre- and post-enhancement periods. The data indicate a significant increase in the percentage of call volumes on weekends. In the pre-enhancement period, call volumes were relatively flat throughout the week, with the weekly peak occurring on Tuesday. In the post-enhancement period call volumes show a more pronounced peak on the weekend. Post-enhancement call volumes average over 2,500 calls per day (about 20% of all calls) for Saturday and over 2,000 calls per day (about 15% of all calls) for Sunday. Post-enhancement weekday volumes peak on Fridays at around 1,800 calls per day, with the lowest volumes on Tuesdays (about 1,400 calls per day), the opposite of the pre-enhancement phase.
Figure 4-2. Monthly Call Volumes
Figure 4-3. Percentage of Call Volume by Day of Week
The post-enhancement results suggest that the Arizona 511 system is used more frequently during weekends for non-commute trips than before the enhancements. Moreover, in comparison with the pre-enhancement period, it shows that in overall call volume during the post-enhancement period came from the weekend travelers than the weekday travelers.
Figures 4-4, 4-5, 4-6, and 4-7 present hourly call volumes for Monday-Thursday, Friday, Saturday, and Sunday, respectively. Figure 4-4 shows that during weekdays, call volumes peak in the morning between 7 and 9 a.m. and in the evening between 3 and 5 p.m. The call volumes remain fairly steady during the middle of the day. This generally conforms to the commute traffic patterns in the Phoenix and Tucson areas.
In Figure 4-5, on Fridays the volume distribution is more accentuated for the evening peak period. There is a slight peak in the morning, but the highest calling volumes occur during the hours of 2 to 6 p.m. There is also a fairly high volume well into the evening (7 to 10 p.m.) during which volumes are comparable to those of the morning peak.
Figure 4-4. Hourly Call Distributions During Weekday (Post-enhancement)
Figure 4-5. Hourly Call Distributions on Friday (Post-enhancement)
Figure 4-6 shows peak call volumes during the morning hours (8 to 11 a.m.), continued heavy use throughout the day (12 to 5 p.m.) and declining volumes through the evening. The patterns of Sunday call volumes (Figure 4-7) are similar to those of Saturday, with the exception that the heavy call volumes sustain longer into the evening, perhaps indicative of weekend recreational travelers returning home.
Figure 4-6. Hourly Call Distributions on Saturday (Post-enhancement)
Figure 4-7. Hourly Call Distributions on Sunday (Post-enhancement)
The call patterns discussed above are similar to those observed during the pre-enhancement period in some respects. In particular, the strong orientation of weekday calling toward peak travel hours (morning and evening), with a steady demand during the mid-day period, are very similar to that observed in the pre-enhancement period. However, the overall patterns of call volumes shows a slight shift toward weekend use. The strong orientation of calls on Fridays toward the evening peak, the heavy demand on Saturday mornings, and the relatively level demand throughout the day on Sunday are characteristic of heavy use for weekend travel.
4.1.2 Call Frequency and Repeat Callers
Frequency of Use
In the examination of usage patterns, it is useful to understand how often users called 511. Table 4-1 shows the number of calls made from the same phone number in a month averaged across the post-enhancement period. It shows 291,245 unique calling numbers yielding a total of 670,369 calls during the one-year post-enhancement period, which equates to approximately 2.3 calls per phone number. This is slightly higher than in the pre-enhancement period, with approximately 2.0 calls per phone number, indicating a slight increase in call frequency. Figure 4-8 shows the call frequency by month in the post-deployment period which generally conforms with the overall call frequency patterns shown in Table 4-1.
| Number of 511 calls from the same phone number per month | Percentage of Calls Pre-Enhancement |
Percentage of Calls Post-Enhancement |
|---|---|---|
| 1 | 72.2% | 68.0% |
| 2-3 | 21.7% | 24.5% |
| 4-5 | 3.4% | 4.7% |
| 6-7 | 1.0% | 1.4% |
| >7 | 1.8% | 1.4% |
| Total unique phone numbers | 7,261 (3 months)8 | 291,245 (12 months) |
| Total 511 calls | 14,682 (3 months) | 670,369 (12 months) |
One confounding factor in this analysis is that the use of private branch exchange (PBX), which are common to offices, could potentially obscure this analysis by showing the same external number for all company phones tied to an exchange (PBX affects only the wireline calls). In this case, the observed increased call frequency may reflect higher use from a single phone, thus demonstrating legitimately increased call frequency, or it may reflect increased usage by multiple users, from a common PBX phone line.
Figure 4-8. Call Frequency by Month (Post-Enhancement)
New and Repeat Callers
An analysis was conducted to examine the number of new callers in each month during the post-enhancement period. As shown in Table 4-2, the end of the month of December was used as the baseline (the new system went on-line on December 17, 2003) and new phone numbers were identified by comparing with all numbers accumulated over the previous months. Repeat callers are defined as those who made more than one call in the post-enhancement period. This analysis is useful in gauging 511's ability to retain repeat users (as an inverse function to the percentage of new users).
Table 4-2 shows that the enhanced system attracts significant (64 to 96%) new users every month. The highest percentage of first-time callers (96%) was recorded in the month of May, which corresponds to the week-long dynamic message sign (DMS) marketing campaign. It suggests that DMS is a very effective tool in promoting the awareness and first use of 511. The percentage of new users after the DMS campaign hovered between 64% and 73%, although overall calls increased, suggesting that 511 generated more repeat callers than the period before the DMS campaign, when new users accounted for 81% to 95% of all callers. The slightly upward trend of the percentage of new users in November and December could be due to the influx of winter visitors.9
| Month | Total Calling Numbers | New Callers | Percent New |
|---|---|---|---|
| December 2003 | 6,902 | 6,902 | 100.0% |
| January 2004 | 17,287 | 16,453 | 95.2% |
| February 2004 | 19,199 | 16,983 | 88.5% |
| March 2004 | 13,380 | 11,097 | 82.9% |
| April 2004 | 10,548 | 8,510 | 80.7% |
| May 2004 | 77,652 | 74,598 | 96.1% |
| June 2004 | 20,849 | 13,795 | 66.2% |
| July 2004 | 29,009 | 21,151 | 72.9% |
| August 2004 | 18,157 | 11,543 | 63.6% |
| Septemeber 2004 | 19,309 | 12,376 | 64.1% |
| October 2004 | 19,576 | 12,512 | 63.9% |
| November 2004 | 19,876 | 13,772 | 69.3% |
| December 2004 | 26,230 | 18,713 | 71.3% |
| Total | 291,072 | 231,503 |
The percentage of new users identified in this analysis (about two-thirds) is considerably higher than the one-third finding from the user survey (see Section 5-1). There are several factors contributing to the discrepancy:
- A disproportionate share of veteran users agreed to participate in the survey.
- The usage analysis, by necessity, counted unique calling numbers rather than actual unique callers. Individual users accessing the system from various phone numbers (e.g., phone, home, office) would appear as multiple users.
- Some 2004 callers who were actually veteran users may have appeared as new users in the usage analysis because they were not reflected in the (by necessity) limited baseline pool that was utilized (i.e., they had called before but it was pre-December 2003).
4.1.3 Call Durations
Average call duration increased approximately 41% in the post-enhancement period, from 63 seconds to about 93 seconds. Figure 4-9 shows the distribution of call durations in 10-second increments, during the post-enhancement period. Over half of the calls were completed within 80 seconds, and 75% were completed within 130 seconds (slightly over 2 minutes). Ninety-four percent were completed within 4 minutes (240 seconds).
Figure 4-9. Call Durations (Post-Enhancement)
About 15% of post-enhancement calls were completed within 20 seconds, compared to 51% in the pre-enhancement period. This could bode well or poorly for the Model Deployment, depending on the nature of these calls. If very short calls are those where the caller hangs up in frustration before getting specific information, this reduction is positive. If short calls are those where skilled users quickly work through the menu system and obtain their information, then the findings are ambiguous. If it's taking callers longer to access the same amount of information, then this is obviously not a positive development. If users are collecting more information per call, that would be a positive outcome.
Based on analysis of content accessed, it is estimated that slightly over 9% (or 1 in 11) of the calls during the post-enhancement period did not remain on the line past the initial greeting. Comparable data from the pre-enhancement period is not available.
4.1.4 Wireless Versus Wireline
This analysis is based on thousand-block (prefix) number assignment data, which assigns phone numbers in a block of one thousand to wireless and wireline service providers, last published in October 2003. The prefix assignment has been relaxed since the FCC ruled to allow number portability (i.e., changing a wireless number to a wireline service, and vice versa; as well as keeping the same wireless number with a different service provider) effective November 2003. Therefore, the accuracy of this analysis depends on the magnitude of changes between the wireless and wireline registrations of phone numbers in the state of Arizona. It is estimated that the impacts of number portability on this analysis is limited, because most phone companies started marketing campaigns in mid-2004 and only focused on the portability among wireless providers.
Given the resource-intensive thousand-block number matching analysis technique that was necessitated for this analysis, it was only feasible to examine Arizona registered phone numbers. Those numbers constituted approximately 69% of all calls to the 511 system. Non-Arizona registered numbers accounted for the remaining 31%.
During the post-enhancement period, the percentage of calls from Arizona-registered telephone prefixes averaged about 53% wireless and 47% wireline. Notably, average wireless call percentage (for all of 2004) of 53% is much higher than that of the pre-enhancement period (20%). A significant portion of the increase may be related to the very large number of calls fielded during the week-long DMS marketing campaign, a very high percentage of which (84%) were from cell phones. Figure 4-10 shows the percentage of wireless calls by month after enhancement. Wireless use was hovering between 22% and 29% prior to the DMS campaign and after the campaign tapered back to just slightly higher than the pre-enhancement level, finishing the year at about one-third of the total calls.
If it was possible to take into account the 31% of 511 calls originating from non-Arizona registered phone numbers, it is quite likely that the percentage of wireless calls would be significantly higher. This is because it is likely that many of these calls are made by Arizona visitors using cell phones registered in their home area.
Figure 4-10. Percentage of Wireless Calls by Month
The average post-enhancement percentage of wireless calls, at 53%, corresponds closely to the 56% obtained in the user survey (see Section 5.0), but is significantly higher than the approximately 40% that occurred during the specific time of the survey (November 2004). The explanation may be that the survey included a disproportionate share of repeat callers, which the survey identified as having significantly higher cell phone use (65%) than first-time users (about 37%).
4.1.5 Call Geographic Location
The distribution of call locations is approximated using data from ADOT 511 phone bills during the three-month period August-October, for the both the pre- (2003) and post-enhancement (2004) periods. This analysis provides only an approximation because it was not possible to analyze all of the call data. The phone bills that were used (from Qwest) cover much of the state, including Phoenix and Flagstaff, but omit Tucson, a number of rural areas, and out-of-state calls. Data for these calls, served by a different provider (AT&T) were not available in a format that could be efficiently analyzed.
Table 4-3 presents the percentage of calls by origin (only locations with 0.5% of total calls or more are shown) for the pre- and post-enhancement periods. Expectedly, call origins correlate closely with population concentrations, with the largest population center, Phoenix, accounting for the greatest percentage of calls before and after the Model Deployment. The percentage of calls from the Phoenix region identified in this analysis is lower than the Phoenix area residency findings in the user survey (73%; see Table 5-2). The difference is likely a combination of the fact that the survey sample contained far fewer first time users than repeat users, with the Phoenix residency of the former being much closer (48%) to the results here, and the fact that some callers residing in Phoenix make 511 calls from other locations.
The concentration of Phoenix calls increased dramatically, from about 19% to about 59%. The explanation for this is unclear. It may be that the enormous spike in Phoenix call volumes observed during the statewide DMS marketing campaign (see Section 4.2.2), and the slowly diminishing but still higher than pre-campaign call volumes observed in the months after the campaign skewed the annual statistics. It is also possible that the other marketing activities (radio ads, ADOT promotions at Phoenix area freeway openings, etc.), which were oriented more to Phoenix, stimulated Phoenix 511 usage. Arguing against this explanation, however, is the user survey finding (see Table 5-1 in Section 5.0) indicating that these other marketing activities did not have a major impact.
| Location of Call Origin | Pre-enhancement Percent | Post-enhancement Percent |
|---|---|---|
| Phoenix | 19.4% | 59.2% |
| Flagstaff | 12.4% | 8.3% |
| Payson | 2.1% | 4.6% |
| Prescott | 9.1% | 4.0% |
| Yuma | 6.6% | 3.3% |
| Casa Grande | 3.9% | 1.9% |
| Sedona | 2.2% | 1.5% |
| Pine | 0.2% | 1.0% |
| Show Low | 1.6% | 1.0% |
| Mesa | 2.3% | 0.8% |
| Eloy | 1.7% | 0.7% |
| Camp Verde | 1.5% | 0.7% |
| Coolidge | 2.0% | 0.7% |
| Cottonwood | 1.4% | 0.7% |
| Heber | 0.2% | 0.7% |
| Chandler | 1.7% | 0.7% |
| Winslow | 0.5% | 0.7% |
| Williams | 0.7% | 0.5% |
| Maricopa | 0.7% | 0.5% |
| Somerton | 0.8% | 0.5% |
4.1.6 Handoffs and Transfers
Handoffs are transfers out of the 511 system to other agencies or to the 511 comment line voicemail box. Table 4-4 presents the post-enhancement percentage of all 511 calls that included various types of call transfers, and the percentage of total hand-offs by type. Overall, about 11% of calls to the 511 system included transfers out of the system. This is generally consistent with the menu selection results (which showed that about 91% of all information requests were for roadways) since the non-roadway menu items include call transfer options and relatively little in the way of imbedded content. The most common transfers, accounting for a combined 45% of all transfers and about 5% of all calls, were to either the Phoenix or Tucson transit services. Other fairly popular transfers were to the Arizona Office of Tourism (10% of all transfers; 1% of all calls) and the 511 caller comment voice mailbox (18% of all transfers; 2% of all calls).
| Handoffs | Number of Handoffs | Percentage of Handoffs | Percentage of Calls |
|---|---|---|---|
| Airports – Phoenix Sky Harbor | 12,198 | 16.2% | 1.8% |
| Airports – Tucson International | 7,240 | 9.6% | 1.1% |
| Transit – Phoenix Valley Metro | 19,826 | 26.3% | 3.0% |
| Transit – Tucson Sun Tran | 13,900 | 18.5% | 2.1% |
| Transit – Others | 501 | 0.7% | 0.1% |
| Tourism | 7,781 | 10.3% | 1.2% |
| Comment | 13,853 | 18.4% | 2.1% |
| Total | 75,299 | 100.0% | 11.2% |
4.1.7 Line Utilization
In the pre-enhancement period, the total line capacity was 48 lines, consisting of 2 servers each managing 24 phone lines. In the post-enhancement period, the total number of phone lines available within the 511 system is 92 lines, distributed as 23 lines on each of 4 servers (one line is reserved for data transmission for each server). Calls entering the system are first assigned to one of the servers and then are assigned randomly to a specific line on that server.
Figure 4-11 indicates the percentage of calls in the 511 system occurring under various line utilization levels. The horizontal axis records, to the nearest four lines, the total number of lines in use when a call comes in. About 43% of the calls enter the system when there are 4 or fewer calls in the system; about 22% of the calls enter the system when there are between 5 and 8 calls in the system; etc.
In over 95% of the cases, the number of lines in use is less than or equal to 32 lines, with about 4.7% of the calls entering the system when at least 32 lines are in use. In 99% of the cases, there are 48 or fewer lines in use (about 50% of the total capacity), and at no point did the total lines in use exceed 84 lines, with this maximum being reached during the May marketing campaign. Clearly, the system has significant available capacity.
Figure 4-11. 511 System Line Utilization (Post-Enhancement)
4.1.8 Contents Accessed
The variety of information available in the enhanced 511 system is very broad. During the one-year post-enhancement period, the 511 system interpreted over 2.1 million information requests (with multiple requests per call possible). In general, these requests break out into the categories as shown in Table 4-5. About 91% of the total requests went through the Roads menu, while slightly over 4% of the requests fell under the Quick Reports. Smaller volumes of requests were observed for transit information (2.4%), airport information (1.3%), tourism and Grand Canyon information (0.4%), and to leave a comment (0.6%). The relatively low percentage of transit requests is in part a function of the fact that very few person trips in the Phoenix region (a major source of 511 calls) are made by transit—just over 1% of all trips (per the Maricopa Association of Governments)—and that very little 511 marketing was targeted to transit users.
It is important to note that in this table the "requests" include requests for the category in the menu structure, as well as the specific piece of information under a category. For example, a call asking for "Roads" followed by "I-17" would appear twice in the "Roads" requests in this summary. A call in which the caller asked for "Roads" followed by "I-17" and "I-40" would appear as three requests in the "Roads" category. That is, the statistics reported in Table 4-3 include some top-level menu requests, as well as requests for specific information. In the analysis that follows, however, only specific information requests within each category are analyzed. As a result, the sample sizes are smaller than the totals in the Table 4-5.
| Category | Requests | Percent |
|---|---|---|
| Roads | 1,983,457 | 91.0% |
| Quick Reports | 94,556 | 4.3% |
| Transit | 52,646 | 2.4% |
| Airports | 28,205 | 1.3% |
| Tourism | 7,781 | 0.4% |
| Comment | 13,853 | 0.6% |
| Total | 2,180,498 | 100.0% |
As shown in Table 4-5, "Roads" is the most requested type of information. At only 4.3%, "Quick Reports" is utilized infrequently. This feature was added a few months after the debut of the enhanced system.
Roads
In this analysis, roadways with equivalent names and numbers are grouped together under a single heading. For example, "One Oh One" is the same as "Loop One Oh One" and, depending on the specific segment, is also known as the "Price", "Pima", and "Agua Fria" freeways.
One of the most fundamental findings is that there are a large number of roadway information requests that were not successfully interpreted by the system. For example, 38% of all requests for roadway information were rejected by the system because they did not include a valid numeric roadway reference. Some additional, perhaps substantial, percentage of requests can assumed to have been rejected because they included no valid roadway name (it was not possible to isolate these instances). Section 4.1.9 considers the issue of unrecognized call inputs in greater depth.
Table 4-6 presents the number and the percentage of requests for all roads (the 38% of all road information requests that could not be interpreted by the system are not included; Table 4-6 shows the break-down of the 61% of requests that were interpreted.) The total number of requests for information on specific roadways is 935,767. Table 4-6 identifies those numbered or named roadways receiving more than 1% of these requests. The high number of requests for I-8 are probably a function of the fact that (as discovered by ADOT) the voice recognition system tends to interpret many extraneous noises as the utterance "eight". If most of the I-8 requests shown in Table 4-6 are in fact spread out proportionately over the other roadways, the findings here are much closer to those obtained in the user survey (see Table 5-8). After I-8, the highest percentage of roadway requests was for Loop 101 (18.6%), a Phoenix area freeway. Interstates 10 and 17, both of which include considerable inter-city and urban area mileage, also had high fractions (almost 10% each) of the requests. Smaller but significant fractions were also observed for SR 51, US 60, Loop 202, and I-40 freeways (all except I-40 are located in the Phoenix area).
| Roadway | Number of Requests | Percentage |
|---|---|---|
| I-8 | 174,872 | 18.7% |
| Loop 101 | 174,455 | 18.6% |
| I-10 | 92,432 | 9.9% |
| I-17 | 90,765 | 9.7% |
| SR 51 | 60,287 | 6.4% |
| US 60 | 47,198 | 5.0% |
| Loop 202 | 53,877 | 5.8% |
| I-40 | 45,872 | 4.9% |
| SR 87 | 18,018 | 1.9% |
| SR 277 | 14,725 | 1.6% |
| SR 587 | 13,468 | 1.4% |
| US 89 | 10,463 | 1.1% |
| SR 143 | 18,103 | 1.9% |
| Others | 121,232 | 13.0% |
Quick Reports
Among the 65,246 requests for Quick Reports, the number and percentage of requests are shown in Table 4-7. Most requests (about 69%) are for the Phoenix metropolitan area, with slightly over 30% of the requests for a quick report from Tucson. Within the Phoenix area, Phoenix (includes "Phoenix" and "Central Phoenix", which are the same area) accounts for about 27% of the requests. The West Valley and the East Valley both received about the same amount of requests (about 11% each), with smaller shares for the Northeast and Northwest Valley (around 9% each). The North Phoenix area received the smallest share, at 2.3%.
| Area | Number of Requests | Percentage |
|---|---|---|
| Tucson | 20,105 | 30.8% |
| Phoenix | 12,418 | 19.0% |
| West Valley – Phoenix | 7,258 | 11.1% |
| East Valley – Phoenix | 7,047 | 10.8% |
| Northwest Valley – Phoenix | 5,854 | 9.0% |
| Northeast Valley – Phoenix | 5,658 | 8.7% |
| Central Phoenix | 5,249 | 8.0% |
| North Phoenix | 1,518 | 2.3% |
| West – Phoenix | 88 | 0.1% |
| East – Phoenix | 51 | 0.1% |
4.1.9 Unrecognized Caller Inputs
A useful surrogate measure of the overall effectiveness of the 511 user interface and specifically, the effectiveness of the voice recognition system, is the proportion of caller inputs that were not comprehended by the system. Caller comments during the first month or two after the roll out of the new user interface—coupled with the first—hand observations of some members of the evaluation team-indicate that the system's misinterpretation of inputs, including mistaking background noise for an input, can be a major source of frustration.
Table 4-8 illustrates the percentage of requests that were not understood by the 511 system during the post-enhancement period. For interpreting these data, these "errors" were situations where the caller received the following response from the system: "I'm sorry, I didn't understand your selection…" The nature of these errors include requests for information that is simply not available in the current location where the caller is in the 511 system (e.g., there is no roadway named "Five") as well as cases where the system could not correctly interpret the caller input.
| Month | Total Requests | Errors | Percentage Errors |
|---|---|---|---|
| January | 171,694 | 42,891 | 25.0% |
| February | 191,356 | 31,153 | 16.3% |
| March | 106,840 | 20,810 | 19.5% |
| April | 60,266 | 14,442 | 24.0% |
| May | 495,838 | 68,484 | 13.8% |
| June | 160,901 | 23,870 | 14.8% |
| July | 213,037 | 37,227 | 17.5% |
| August | 148,511 | 28,323 | 19.1% |
| September | 155,120 | 31,051 | 20.0% |
| October | 127,319 | 24,338 | 19.1% |
| November | 144,667 | 26,178 | 18.1% |
| December | 204,949 | 39,724 | 19.4% |
| Total | 2,180,498 | 388,491 | 17.8% |
The highest percentage of errors (25%) occurs at the beginning of the post-implementation period, in January 2004, when the voice recognition system was still being refined (as noted in item 1 in Section 7.2.1 the voice recognition consultant believed that additional time for testing would have been useful.) By May 2004, the system was at the lowest error rate in the year, at 13.8% of all requests. This number has gradually increased, to the point where it was back near 20% by the end of the year. The average error rate over the one year post-enhancement period was 17.8% of all requests. The exceptionally low rate in May might have been a result of many new callers attracted by the May DMS ad campaign exploring the system, making only one simple information request from the top menu (e.g., simply saying "roads" then hanging up without making a specific roadway request.)
Curiously, as indicated in Table 4-9, almost all (93%) of unrecognized inputs occur at the top-level (main) menu. This may be because either the potential for errors is highest at the main menu (as every call includes activity at that level), or it may be because the voice recognition system has to "listen" for a much wider range of utterances at that level and therefore the possibility of misinterpretation is greater.
| Menu Location | Total Errors | Percentage of Errors |
|---|---|---|
| Airports | 20 | 0.0% |
| Comment | 32 | 0.0% |
| Help | 12,902 | 3.3% |
| Main Menu | 361,822 | 93.1% |
| Misc Transfers | 8,760 | 2.3% |
| Quick Reports | 175 | 0.0% |
| Roads | 1,538 | 0.4% |
| Tourism | 350 | 0.1% |
| Transit | 2,892 | 0.7% |
| Total | 388,491 | 100.0% |
Table 4-10 presents the percentage of calls with unrecognized user input errors by month (as opposed to Table 4-8 which examines the percentage of requests, with most calls containing multiple requests.) Overall, about 37% of all post-enhancement (2004) calls to the 511 system included errors of this type. The fact that this is higher than the 18% figure in Table 4-8 is because it only takes one misinterpreted request to be considered an "error call" in Table 4-10 (and conversely, it indicates that not all of the multiple requests common to most calls were misinterpreted.) Expectedly, errors were much more common (about 54%) in the first month of enhanced operation, consistent with the caller comments which indicate significant difficulties and frustration with the early voice recognition system. The percentage of errors declined over the first couple of months, probably reflecting both the refinements to the user interface and users' increasing familiarity. After March, the percentage of errors fluctuated in the range of 30-37%.
| Month | Calls | Calls with Errors | Percent |
|---|---|---|---|
| January | 38,170 | 20,548 | 53.8% |
| February | 41,680 | 18,053 | 43.3% |
| March | 26,923 | 11,002 | 40.9% |
| April | 20,417 | 6,857 | 33.6% |
| May | 149,310 | 49,955 | 33.5% |
| June | 45,513 | 15,847 | 34.8% |
| July | 62,533 | 22,835 | 36.5% |
| August | 39,459 | 14,428 | 36.6% |
| September | 42,329 | 15,176 | 35.9% |
| October | 42,069 | 14,734 | 35.0% |
| November | 44,959 | 13,519 | 30.1% |
| December | 53,943 | 19,975 | 37.0% |
| Total | 607,305 | 222,929 | 36.7% |
4.2 Usage During Special Occasions
Several scenarios that could have potential impacts on 511 usage during the post-enhancement period were identified as follows:
- The transition to the enhanced 511 system during December 2003 and January 2004
- The statewide marketing of 511 via dynamic message signs
- Wildfires and snow events
- A major traffic accident
4.2.1 Usage During Transition to Enhanced 511
On December 17, 2003, the new voice recognition-based 511 system replaced the old touch tone system without formal announcement from ADOT. Other enhancements included the addition of call transfer options to airports. One major concern was whether there would be a significant adverse impact on 511 use immediately following the roll out of the new user interface. It was speculated that callers accustomed to using the old system might react negatively, including no longer using the system. This concern reflected the understanding that the performance of the voice recognition system was less than optimal at the time of the roll out.
Figure 4-12 illustrates the call volumes during the transition period between December 2003 and January 2004. During this period, call volumes averaged near 1,000 calls per day, with slightly higher volumes observed in January. No adverse effects on the number of 511 calls during the transition were seen.
Figure 4-12. Call Volumes at Transition to Enhanced 511 System
In the months of December and January, only 8.6% and 8.3% of the total calls for each month, respectively, were found to have ended before the caller made a request. While slightly higher than the average for the entire post-implementation period, these values suggest that only a small fraction of 511 users chose not to interact with the new system.
During the transition, 23,355 individual callers (unique phone numbers) were identified in the 511 system. By the end of April 2005, 3,263 of these individual callers (about 14%) had called back. This compares closely to repeat call behavior during the pre-enhancement period.
4.2.2 Marketing Campaign Using Dynamic Message Signs
During the course of 2004, ADOT posted general (not incident or location specific) references to 511 messages on DMS throughout the state. The campaign occurred during the week of May 3-9, 2004 when the message "ROAD CONDITIONS, DIAL 511" was displayed 24 hours a day for the entire week (see Figure 4-13). The effects of the marketing campaign have been profound in publicizing the Arizona 511 system.
Figure 4-13. Picture of 511 Marketing Campaign via Dynamic Message Sign
Daily call volumes during the marketing campaign are presented in Figure 4-14, showing the magnitude of the calls in each day of the marketing effort compared with the weeks over April, May, and June. Call volumes during the DMS marketing campaign were dramatically higher, up to 20 times higher, than during the preceding weeks. There was a maximum daily volume of 17,265 calls on Monday, May 3, 2004, and daily volumes well over 11,000 all week, including the weekend (May 8 and 9). At total of 94,023 calls were made over the week-long DMS marketing campaign.
During this week, the call volumes by time of day reflect much higher volumes during the morning and evening peak periods. This is shown in Figure 4-15, illustrating the strong demand from 7 a.m. (hour 8) to 8 p.m. (hour 19), with the highest call volume in the evening peak period. Such a pattern is not unexpected since the DMS advertising has the most immediate impact on en-route travelers and these time periods correspond to peak traffic hours.
During the week of the marketing campaign, approximately 84% of all calls were made from a wireless phone, which is also not unexpected given the exposure of en-route travelers to DMS. This significantly increased the percentage of wireless calls for the month of May (83.1%) compared with other months in 2004 (ranged from about 20 to 60%).
Only 7.6% of calls made during the marketing campaign ended before information was requested. This percentage is actually slightly lower than that observed throughout 2004. This suggests that many first-time callers were probably curious and were willing to explore the new system.
Figure 4-14. Call Volumes During DMS Marketing Campaign
Figure 4-15. Percentage of Calls by Time of Day During DMS Marketing Campaign
4.2.3 Wildfires
Primarily during the months of June and July, Arizona often experiences significant wildfires. In both 2003 and 2004, during the implementation of the Arizona 511 system, several major wildfires occurred. Roadways are often closed or restricted during wildfires.
There were a number of major wildfires that could be analyzed. The National Forest Service listed 34 major fires (over 100 acres) in Arizona during 2004 and 26 during 2003. However, rather than addressing all fires, one example is given to illustrate the potential effects on 511 use. As was noted previously, fairly high call volumes were observed during the first and second weeks of July 2004. During that time, there were several major fires burning in Arizona, which included most notably the Willow fire, to the northwest of Payson along State Route 87 (begun June 24 and contained July 21, 2004), and the Ponderosa fire, just east of Payson in the Tonto National Forest, off of State Route 260 (begun and contained on July 8). The combination of these fires affected traffic patterns in and around Payson (2 hours northeast of Phoenix), and as a result generated some calls to the 511 system.
Higher-than-average call volumes for several state routes in the vicinity of the fires, including 87,587, and 260 were recorded. Table 4-11 illustrates the road information requests from July 7 through July 10, 2004, during these fires. The volume of calls on these days ranged from over 3,600 to about 6,000 calls per day, compared with the annual average daily call volume of 1,832. The percentage of information requests for wildfire-impacted roadways (e.g., SR 87, SR 587 and SR 260) was higher during the wildfires than for the one-year average. Evidently, travelers concerned about travel in areas near wildfires turned to 511 for information on road closures and restrictions.
| Roadway | Percentage of All Roadway Requests During Fire |
Percentage of All Roadway Requests Annual Average |
|---|---|---|
| SR 87 | 11.5% | 1.9% |
| SR 587 | 3.8% | 1.4% |
| SR 260 | 3.3% | < 1.0% |
4.2.4 Major Snows
Table 4-12 identifies dates of major snowfall in northern Arizona, as recorded in Flagstaff by the National Oceanic and Atmospheric Administration (NOAA). Major snowfalls obviously impact travel and would presumably increase the demand for traveler information.
| 2004 Major Snow Storms in Northern Arizona (Post-Enhancement Period)10 |
|---|
| January 3-5 and 20-21 |
| February 1, 4, 19, 21-25, 27-29 |
| March 2-3, 5, 13 |
| April 3-4 |
| October 22, 28-29, 31-November 4 |
| November 21-23, 27-29 |
| December 5-9, 29-30 |
The major snowfalls identified in Table 4-12 are circled in Figure 4-16, which presents daily 511 call volumes. Snow events appear to be closely related to 511 usage. In fact, with the exception of the DMS marketing in May 2004, the snow events produced the highest volumes of calls to 511.
During those snow events, higher volumes of information requests were received for roadways in northern and eastern Arizona. As an example, during the major snow event of December 5-9, 2004, call volumes began climbing on December 4 (a Saturday), peaked on December 5, and began to drop off over December 7-9, as the snow tapered off.
The major roadway information requests received during this snow event are shown in Table 4-13. Relative to other times of the year, a heavier volume of requests was observed for roadways with significant mileage in snowfall areas: I-40 (16.8%) and I-17 (12%). In addition, State Routes 89, 87, 587, 260, and 80 also received a significant number of requests during this snow event. Such a pattern of information requests can be considered typical of major snow events in northern and eastern Arizona.
Figure 4-16. 511 Usage Spikes Correlated with Major Snows
| Roadway | Percentage of All Roadway Requests During Major Snow |
Percentage of All Roadway Requests Annual Average |
|---|---|---|
| I-40 | 16.8% | 4.9% |
| I-17 | 12.0% | 9.7% |
| SR 89 | 2.3% | 1.1% |
| SR 87 | 2.0% | 1.9% |
| SR 587 | 1.7% | 1.4% |
| SR 260 | 1.5% | < 1.0% |
| SR 80 | 1.2% | < 1.0% |
4.2.5 Major Crash
On the afternoon of August 11, 2004, a sandstorm crossed I-10 west of Buckeye, at Milepost 96 between Phoenix and the California state line. During the storm, a fiery crash in the westbound lanes of I-10 caused several fatalities. The roadway was closed for over 24 hours, until late in the day on August 12.
During this time, the 511 system carried typical total volume of daily calls, with 1,961 calls on August 11 and 1,887 calls on August 12. However, the calling patterns by hour of the day reveal a considerable peaking of calls during the evening of August 11 and again during the morning of August 12, as shown in Figure 4-17. A very large spike of 344 calls occurred during the first hour after the incident (5 p.m. on August 11, or hour 17 in the figure), followed by much heavier volumes for the next four hours (up to about 10 p.m. or hour 22). There was also a considerable peak the following morning, around 9 to 11 a.m. (hours 33 and 34 in Figure 4-17).
Figure 4-17. Call Volumes During Major Traffic Incident in Phoenix
Not surprisingly, the percentage of information requests for I-10 increased substantially during August 11-12, 2004, to about 26% of total requests (up from the 10% annual average).
4.3 Conclusions
- Considerable usage increase in the post-deployment period. The overall 511 usage increased by 74% during the one-year post-enhancement period with a total of 670,369 calls. The increased usage appears to be attributable to the week-long statewide DMS marketing campaign. Considering only the four post-enhancement months prior to the campaign (January – April 2004) call volumes were actually about 2% lower than the same period in the previous (pre-enhancement) year.
- Significantly higher call volume (36%) was observed during weekends with the peak on Saturdays (20%) in the post-enhancement period. Such a pattern was not found in the pre-enhancement period, which showed a relatively flat weekly call distribution, with the highest call volumes on Tuesdays. This suggests that the growth in call volume during the post-enhancement period was accounted for by the weekend users more than by the daily commuters.
- Call frequency increased slightly in the post-enhancement period but it is still low for most callers. Average call frequency increased slightly, from 2.0 to 2.3 calls per year per unique phone number. The data show that 68% of users made one call per month, a decrease from 71% in the pre-enhancement period, and that 24% of users made two or three calls per month, essentially the same as the 23% before the enhancement. Relatively low call frequency may suggest that the overall dependency on 511 is relatively low in the state of Arizona. A plausible explanation is the competition from other information sources (e.g., dynamic message signs, radio) and degrading but relatively manageable commute conditions.
- Increased repeat users in post-enhancement period, despite the somewhat low overall percentage of repeat users. Overall percentage of repeat users increased significantly from 10% to19% before the DMS marketing campaign to approximately 27% to 37% after the campaign. However, the relatively low percentage of repeat users suggests a small user base and also reveals that the potential of Arizona 511 is yet to be fully exploited. This is not surprising considering that it is only one year after the enhancement and the lack of sustained, intensive marketing efforts (e.g., roadside signs, periodic promotion on DMS, etc.).
- Longer call durations experienced with enhanced 511. Average call duration increased about 41%, from 66 to 93 seconds in the post-enhancement period. The percentage of very short calls, of 20 seconds or less, dropped significantly (15% post-enhancement versus 51% pre-enhancement). If very short calls are those where the caller hangs up in frustration before getting specific information, this reduction is positive. If short calls are those where skilled users quickly work through the menu system and obtain their information, then the findings are ambiguous. If it's taking callers longer to access the same amount of information, then this is obviously not a positive development. If users are collecting more information per call, that would be a positive outcome.
- The proportion of wireless calls increased. The overall percentage of wireless calls increased from 20% (pre-enhancement) to 53% in the post-enhancement period. However, much of this increase is probably due to the very high (84%) cell phone use during the week-long statewide DMS ad campaign. Wireless percentages before and after the DMS campaign were 20-30%.
- Significant growth in 511 use from greater Phoenix area. The overall share of calls from the greater Phoenix metropolitan area (including Mesa, Tempe, Chandler, North Phoenix) has grown from 36% to 62% in the post-enhancement period. This most likely reflects the impact of wider marketing in the Phoenix metropolitan area, when compared with other areas in the state.
- The enhanced system still has significant available capacity. The enhanced 511 has a total capacity of 92 lines managed by 4 servers. In 99% of the cases, 48 or fewer lines were in use (about 50% of the total capacity), and at no point did the total lines in use exceed 84 lines, with this maximum being reached during the May 2004 DMS marketing campaign.
- Road conditions are the predominant type of information requested. "Roads" accounted for 91% of all information requests during the post-enhancement period. At only 4.3%, "Quick Reports" are very little utilized, although this feature was added a few months after the debut of the enhanced system. Transit, airport, and tourism were minimally used at 2.4%, 1.3%, and 0.4% of all requests, respectively. These results are not unexpected given the absence of 511 marketing emphasizing these new information types (e.g., announcing and explaining Quick Reports) and the absence of marketing directed to the specific user groups that would have an interest in these new, non-roadway information types (e.g., transit riders). The relatively low utilization of transit information is also consistent with the very small regional transit mode share (about 1.2% of all regional trips).
- Many of the large number of misinterpreted commands related to the performance of voice recognition. Over the entire post-enhancement year, 37% of calls included one or more user inputs that could not be interpreted by the 511 system. That percentage was highest (58%) in January 2004, just after the enhanced system was rolled out. This number gradually declined over the first several months of 2004, presumably as a result of the intensive refinements that were made to the user interface, and by July had settled in at about the annual average.
- A freeway dynamic message sign is an effective marketing tool in promoting the awareness of 511 service. Call volumes increased dramatically during the month of the campaign (May 2004) and although they decreased in the following weeks and months, they remained higher than pre-campaign levels. During the month of DMS marketing, monthly call volume increased by a factor of 3.1, with 96% of users being new to the system. Approximately 84% of 511 calls during the DMS marketing campaign were made from cell phones, up from about 30% immediately prior to the campaign. This suggests that the DMS marketing campaign had, expectedly, a disproportionate impact on en-route travelers. The DMS marketing campaign, though short-lived, was extremely effective in publicizing the Arizona 511 system.
- 511 usage is strongly correlated with regional events such as wildfires and major snows. The scenario-based analyses showed that the 511 call volumes and content being accessed correlate closely with the time and area (in terms of affected roads) of the events. In the case of snows, the major snow events appear to be closely related to increased 511 usage.
- 511 usage surged during a major crash on I-10 west of Phoenix. Despite minimal changes in total daily call volumes, the analysis of hourly call volumes and peaking patterns indicated that calls to the system increased in the hours following the crash and the percentage of calls on the impacted roadway (I-10) increased.
