Final Report

Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System

3.0 Evaluation Approach

This section describes the data collection methods associated with each of the evaluation analyses: system usage, 511 user telephone survey, stakeholder interviews and workshops, and costs.

3.1 Usage Analysis

Two types of usage analyses were carried out. First, the general usage analysis focused on the assessment of changes in usage patterns before and after the enhancements. One year of usage before (2003) was compared and contrasted with one year of usage after (2004). The analysis considered call volume distribution, call duration, type of phone, call geographic location, new and repeat users, and content assessed.

Second, over the course of the post-enhancement, 511 usage during specific scenarios such as marketing campaigns, major traffic incidents, major holidays, significant regional events (e.g., forest fire) were analyzed in the scenario-based analyses. While the before-and-after comparison assessed the overall usage patterns and changes brought along by the enhanced system, the scenario-based analyses provided a more in-depth look into 511 usage in response to special events.

During the post-enhancement period, ADOT systematically noted events that could have significant impacts on 511 usage. Subsequently, a list of scenarios was identified for evaluation including:

• Usage during transition to enhanced system
• Marketing campaign using dynamic message signs
• Wildfires
• Major snows
• Major crash

The primary source of data for usage analysis is the logs generated by the 511 VRAS computers. The VRAS logs contain detailed information on time/date of call, duration, caller ID, line used (of 23 available lines on each of the 4 VRAS computers), and information assessed (detailed menu navigation). The line used and call duration information were used to derive information on line capacity that indicated the number of current calls in the 511 system at any given time. Phone bills were used to supplement the analysis of the pre-enhancement system because the old system did not preserve the caller ID information.

A number of ancillary data sources were exploited in support of the analysis. For example, system availability information was collected using ADOT's monthly system outage report during the post-deployment period. Information on VRAS update history was obtained to identify the incremental changes in the 511 menu and data contents over the course of post-deployment period. Information on the 511 marketing campaign such as the messages posted on dynamic message signs to promote 511 usage was obtained from ADOT.

A computer program was created by the University of Arizona to parse the VRAS logs and produce statistics of interest. External data such as block assignment telephone numbers was used to distinguish the wireless from wireline calls and identify registered location of the calls.

One critical data set of interest, ANI II (advanced Automatic Number Identification), was not collected because it is not available from the 511 line communication service provider Qwest. Such data are essential in accurately identifying wireless line vs. wireline, especially after FCC's rule change in November 2003 to allow a wireline number to be changed to wireless service and vice versa. Consequently, it was necessary to use block assignment methods for identifying type of phone service. Although this was suboptimal, it was the only method available to the evaluation team.

3.2 User Telephone Survey

Assessment of 511 users' experience with the service was a key component of the evaluation. A decision was made not to undertake a comparison of the enhanced service with the original service because the service had changed substantially and because of the impracticality of trying to find sufficient numbers of users of both versions of the 511 service. Thus, the objective of surveying users was to measure their satisfaction with the enhanced service only. The focus was on their attitudes toward the type of content offered on 511 and its quality, the user interface, the benefits they perceived the service provided, and how using the information impacted their travel. To achieve this objective a representative cross section of 511 callers was needed. This section describes the approach used for the user survey. The findings from the survey are presented in Section 5.0.

The study team considered alternative methods for identifying the target population of 511 users and determined that the most efficient means was to use the 511 service itself by intercepting incoming calls to the system. A live recruitment of users by an operator was used to increase the potential for a representative sample of 511 users compared to a self-selected sample of users who respond to an automated recruitment message on the 511 service. Thus, the recruitment of survey subjects relied on the intercept of calls to the 511 system by professional call takers. The intercepted calls, before reaching the 511 greeting message, were transferred to the call takers located in a local (Phoenix, AZ) survey firm.

3.2.1 Creating the Intercept Function

Conducting a live intercept survey required close coordination among ADOT, their software provider, the national evaluation team, and the local survey research firm. The functional requirements for the intercept survey included:

• Ability of the 511 system to randomly sample (select) and forward a call to an outside number based on a specifiable interval (every n^th call);
• Avoidance of additional long distance charges by forwarding the intercepted calls only to a local number;
• An off-premises transfer of an intercepted call from the 511 system to the designated number, thereby not occupying available lines of the 511 system; and
• Upon the completion of recruitment, transfer of the caller back to 511 using the off-premises transfer mechanism at the survey research firm.

Figure 3-1 shows the high-level configuration of the Arizona 511 system and the call intercept mechanism in support of survey recruitment. The intercept function underwent a variety of tests to assure that it would operate properly once going "live" with actual 511 callers, and it was deemed successful so that pre-testing of the survey with callers could begin in October 2004.

Figure 3-1. High-Level 511 System Configuration in Support of Call Intercept

3.2.2 The Sampling Plan

The design of the sampling plan included the following elements: target population and sampling frame; sample sizes and anticipated power; and sampling methodology.

Target Population and Sampling Frame

The 511 system in Arizona can be used by virtually anyone because there are no restrictions other than requiring a working telephone and the ability to hear and comprehend the instructions. Therefore, the likely users of the system include a wide variety of people, such as commuters, tourists, commercial vehicle operators, etc., and obtaining the perception of the 511 system from all of these users was of interest. This implied that the target population includes any person who is a user of the 511 system. The sampling frame, or population from which persons were selected, consisted of all users of 511 during a two-week period. However, the sampling frame needed to be further restricted because it was not cost-feasible to sample earlier than 5 a.m. or later than 9 p.m. (none of the candidate Phoenix-area research firms routinely staffed longer hours.) Thus, the results of the survey were viewed as representative of daytime and early evening users of 511 assuming that users during the two-week field period are representative of all such users.

Sample Sizes and Statistical Power

A key component of the sampling plan was establishment of the number of 511 callers who would complete the telephone survey, i.e., the sample size. Estimation of sample size rests on a combination of several factors, including the level of precision sought for the survey results in statistical terms, an understanding of how the results might be analyzed, such as comparison of subgroups of types of users, and the resources available for the survey. One key measure collected through the survey of 511 users is the percentage of respondents who answer a question about an attribute of 511 in a particular way. Based on these factors, a sample size of 400 completed interviews was estimated to provide adequate precision (95% confidence) for the estimation of the percentage of participants with a characteristic of interest (e.g., satisfaction). Further, a sample size of 400 completed interviews would also permit the identification of statistically significant differences in the responses among subpopulations (e.g., repeat callers versus new callers) of 14% to 21%, depending on the balance of the sample. Smaller differences would not be identified as statistically significant, but that trade-off was considered acceptable from a resource consideration.

Sampling Methodology

A two-stage interviewing process was employed to elicit information from users of the 511 system. First, an intercept of "live" calls was conducted to obtain contact information from users and willingness to participate during a call averaging 2 minutes in duration. Upon the completion of the intercept interview, callers were transferred back into the 511 system. Among 511 callers who agreed to the survey, a second interview initiated by the survey administrators was conducted at a time convenient to each caller to collect detailed information about their experience with the 511 service. The second interview averaged 16 minutes in length.

The general sampling approach to identify initial users for the intercept interview was a stratified, systematic sample of calls to the 511 system. Day-of-the-week and time-of-the-day (in one hour increments) formed the sampling strata. Within each stratum, a systematic random sample of 511 callers was selected. That is, every n^th call was intercepted. A proportional allocation based upon the percentage of all calls that were observed during June 2004 was used to preserve the inherent distribution of 511 calls over weekdays and times. This was important since it is likely that different types of users access the 511 system at different times of the day or on different weekdays. With a proportional allocation, it was expected that the sample distribution would be roughly similar to the actual distribution of users. To minimize the possibility of a sampling bias and to maximize the call volumes during the data collection period, calls from all four VRAS boxes were sampled simultaneously, though with the same sampling interval.

The sampling interval (i.e., the n^th caller) is a function of the expected response rates at each data collection phase, the anticipated call volumes, the degree of repeat callers, and the expected 511 usage during the data collection period. Based upon the evaluation team's experience with previous customer satisfaction surveys, it was assumed that roughly 20% of the intercepted callers, or every 5^th caller, would agree to participate in the survey, with approximately 60% of them actually participating in the second stage of data collection. VRAS log data from June 2004 indicate that there were 45,512 calls into the Arizona 511 system during the month of June. These calls were distributed throughout the day and days-of-the-week, though a higher volume of calls was observed during weekday rush-hours and on the weekends, particularly Saturday. During the first two weeks of June 2004, the majority of callers (62.5%) used the system only once; 19.2% used the system more than once, but on a single day; and the remainder or 18.3% used the system on multiple times on multiple days. Based upon these factors, it was estimated 4,315 callers would need to be intercepted to achieve the target of 400 completed detailed interviews. In summary, the plan estimated that 21.5% (4,315) of all daytime callers in a two-week period would be intercepted between 5:00 a.m. and 9:00 p.m. That would yield roughly 3,333 unique users of the 511 system, of which 667 were expected to complete the intercept interview and agree to participate with a more extensive survey (20%). Approximately 60% of the 667 initial respondents would actually complete the detailed interview (400 respondents).

Fielding of the Survey

The evaluation team selected a local survey firm that used a Computer Assisted Telephone Interview (CATI) methodology to conduct both the intercept survey and the detailed survey. Under this approach, trained interviewers administered the survey by reading the questions to the respondents as they appeared on the interviewer's computer terminal. Responses from the respondent were keyed into the computer as they were given (i.e., real-time data entry of respondents' information).

The CATI system includes automated range checks and other checks to ensure that the data collected are within pre-determined criteria. This is the major advantage of collecting data using a CATI system. It allows for instantaneous clarification of inconsistent respondent information. Errors in logic are avoided by programming the questionnaire skip patterns into the system. The "customized" questionnaires also greatly speed interviewing by reducing redundancy. This substantially reduces respondent burden, and as a result there is a corresponding increase in response rates and the accuracy and completeness of data provided.

The objective of the telephone survey was to collect data on 511 users' experience with and perceptions of the 511 system in Arizona in approximately 15 minutes. The questionnaire was based on the set of "core questions" that were developed by the 511 Evaluation Panel, consisting of 511 deployers and other interested parties. The Panel was established in January 2003 to work with U.S. DOT and the evaluation team to provide input on the evaluation of the 511 Model Deployment and to develop a set of common metrics and methods that 511 deployers can use to evaluate their systems. A set of core questions for assessing customer satisfaction was developed by the Panel in 2003 to address the following areas:

• Frequency of use
• Occasion for use of 511 when caller was intercepted
• Satisfaction with the 511 service
• Recommended improvements to 511
• Demographic information.

The core questions were adapted to the Arizona 511 service by adding or deleting items or rewording as appropriate to test the hypotheses for the national evaluation. Pre-testing of the questionnaire demonstrated that not all the core questions could be accommodated within the 15-minute targeted length of the survey, and therefore the Arizona survey would not constitute a valid test of the national core questions. As a result, once the 400 interviews were completed, a set of 30 additional interviews was conducted to test the core questions that did not fit within the Arizona questionnaire. Guidelines on the use of the standard evaluation approach and core questions are available in a separate report.⁷

Following pretests of the survey instruments in late October 2004, the telephone survey of callers to 511 was undertaken from November 3 through 17, approximately eleven months after the enhanced service was launched in December 2003. The November time frame had the advantage of providing sufficient time for regular callers to 511 to have become fully acquainted with the service, and it avoided the holiday periods when it might have been difficult to reach individuals for interviews.

The procedures for response rate calculation are based on the guidelines established by the Council of American Survey Research Organizations (CASRO) in defining a response rate. The initial response rate (i.e., those that agreed to make an appointment to be interviewed) and final response rate (i.e., those that completed the interview) for the survey were obtained using the following formulas:

Table 3-1 presents the distribution of intercept attempts by disposition category. The number of attempts in each category was then used in the above formulas to calculate initial and final response rates of 31% and 22%, respectively.

Table 3-1. Distribution of Intercept Attempts by Disposition Category

Disposition Category	Number of Cases
Total Intercepts	2,205
Total Eligible	1,604
Made appointment*	581
Hung up	797
Refused	226
Total Ineligible	283
Automated message	60
Recruited before	72
Wrong number	70
Other-ineligible**	81
Total Unknown Eligibility	318
Silence	200
Other-unknown	118

The response rates for the survey were better than the original estimates. Prior to the survey, it was estimated that 20% of intercepted callers would agree to be interviewed and 60% of those would complete the actual survey. As shown above, the actual response rate of callers agreeing to be interviewed was 31%, and 71% of them (411/581) completed the survey.

Post Survey Analysis

The first step in the analysis was to compare the actual intercept frequencies by time of day and day of week with those in the sampling plan and apply weights to adjust the sample distribution. A standard survey practice, weighting helps reduce bias and improve the precision of estimates. Appendix B describes the weighting scheme applied to the data.

Using the weighted results, the next step in the analysis was to calculate simple frequencies of responses for each question. For selected questions, the next step was to prepare cross-tabulations to examine possible relationships between dependent and independent variables. For verbatim comments made by respondents to open-ended questions, coding the responses into categories was performed when responses were sufficient to make coding a useful means for assessing the data. The findings from the analysis are reported in Section 5.0. The frequencies of responses for each of the questions are presented in Appendix C.

3.3 Stakeholder Interviews and Workshops

Interviews with 511 stakeholders—the agencies and consultants who participated in the Model Deployment—and "lessons learned workshops" were the primary mechanisms for collecting enhancement process data. A useful secondary source of information was the many meetings attended by the evaluation team throughout the planning, implementation, and operational stages of the Model Deployment. These meetings included the half dozen 511 Task Force meetings as well as regular teleconferences between the local (Arizona) members of the evaluation team and the ADOT 511 evaluation point-of-contact (the manager of the Information Technology Section). The teleconferences in particular provided an excellent opportunity to collect information reflecting ADOT's perspectives throughout the project. Much of that information was critical in providing input to the interviews with other stakeholders.

Two rounds of stakeholder interviews and follow-up lesson learned workshops were held. The first round of interviews and the first workshop were held in January and February 2004, at the conclusion of the planning and implementation stage of the project. The second round of interviews and second workshop were conducted in January and February 2005, following the conclusion of the one-year post-enhancement period.

The stakeholder interviews were conducted either one-on-one or with small groups of stakeholders. The interviews provided an opportunity to collect in-depth information from specific stakeholders. This information was then synthesized and used to guide group discussions with the full 511 Task Force at the lessons learned workshops. The purpose of the workshops was to allow all of the stakeholders to comment on the evaluation team's interview observations. For the workshops, brief bullet-item summaries of input from the interviews were used as a discussion guide without attribution to individual stakeholder agencies.

The specific questions and topics covered in the discussion guide varied somewhat by interview, depending on the role played by the interviewee in the Model Deployment. The general questions and topics addressed at the interviews included the following:

• Overall role in the 511 project
• Organization's specific Model Deployment activities
• Experience in inputting data to the 511 system (discuss each type of data)
• Major successes of the Model Deployment
• Shortcomings and challenges encountered (including solutions to problems)
• Plans and desires for "next steps" for their agency and the overall 511 program
• Major unanswered questions for the Model Deployment and for 511 as a national program/strategy
• Lessons learned (i.e., advice to other regions deploying or operating 511, or what they would do differently)
• Feedback from their customers/constituency regarding their 511 involvement.

The stakeholder interview discussions were also facilitated with a list of potential issues, reflecting the evaluation team's expectations and input from other organizations. This list provided prompts to stimulate discussion in various areas. A formal list of potential issues was developed during the evaluation planning stage and supplemented informally with information gained over the course of the implementation and operation.

Most of the stakeholder interviews were conducted by telephone and lasted between 30 and 90 minutes. Table 3-2 identifies the stakeholder agencies' participation in both rounds of interviews and workshops. Fewer stakeholder interviews were conducted in the first round, which was restricted to the relatively small group of agencies and their consultants who had played a key role in the Model Deployment through the planning and implementation stage. Stakeholder participation in both rounds of interviews and the first workshop was excellent. Although the turn-out for the final workshop was small, it included most of the stakeholders who played major roles in the Model Deployment.

3.4 Cost Data

All cost data were provided by the ADOT evaluation point-of-contact, the Information Technologies Manager, who played a key role in most of the data enhancements, including directing the work of the ADOT consultants. In developing the overall approach to the cost analysis and preparing specific ADOT cost data requests, the evaluation team coordinated with Mitretek, U.S. DOT's consultant that manages the National ITS Benefits and Costs Database. The purpose of the coordination was to help ensure that the cost data for the Model Deployment would be as comprehensive and as consistent as possible with the national cost data, especially data for other 511 and conventional (10-digit) traveler information systems.

The overall approach to the cost analysis and associated data requirements was shared with ADOT during the planning stage of the evaluation. Cost data were collected in two rounds. The first occurred in December 2003-January 2004 and focused on pre-enhanced system costs. The second round of data collection occurred in December 2004-January 2005 and focused on post-enhancement data.

The evaluation team provided ADOT with a spreadsheet template identifying the desired pre- and post-enhancement cost data. That template included cost break-downs by enhancement and by phase (planning and design, implementation, and operation). Implementation costs were broken-down into hardware and software (including engineering) elements. ADOT gathered the cost data from various sources, including monthly phone bills, consultant contracts, and internal records and provided the data to the evaluation team. Remaining issues were resolved through follow-up phone conversations and e-mails.

Table 3-2. 511 Stakeholder Participation in Evaluation Interviews and Workshops

Stakeholder Organization	Interview Participation		Lessons Learned Workshop Participation
	Round 1 January 2003	Round 2 January 2004	Round 1 January 2003	Round 2 January 2004
Agencies
ADOT Headquarters (Phoenix) – Management
ADOT Headquarters – HCRS Operators
City of Glendale
City of Phoenix (Traffic)
City of Tucson (Traffic)
Federal Highway Administration (District)
Maricopa Association of Governments (MPO)
Maricopa County Department of Transportation (MCDOT)
Phoenix Airport (Sky Harbor)
Phoenix Transit (Valley Metro)
Pima Association of Governments (MPO)
Tucson Airport
Tucson Transit (SunTran)
Consultants
Call Processing (IVR system)
Kimley-Horn (Marketing)
OZ Engineering (HCRS)
PBS&J (ADOT Program Support)
PIPS (Arterial Street Travel Time System)

---

---

ITS Home | US DOT Home | Privacy | Feedback

United States Department of Transportation - ITS