Key Informant Interviews Test Plan Model Deployment of a Regional, Multi-Modal 511 Traveler Information System Task Order BA7746

1.0 Introduction and Background

This document presents the detailed plan to conduct the Key Informants Interviews Test, one of several test activities to be conducted as part of the national evaluation of the regional, multi-modal 511 Traveler Information System Model Deployment. The model deployment is an enhancement of an existing statewide 511 telephone traveler information system operated by the Arizona Department of Transportation. As a United States Department of Transportation (U.S. DOT) Intelligent Transportation System (ITS) Model Deployment, the project is intended to generate findings that will help shape U.S. DOT approaches to 511 and that will be of use to others implementing and operating 511 systems. The national evaluation is the primary mechanism for documenting the performance of the model deployment and the lessons learned.

1.1 511 Model Deployment

The Arizona Department of Transportation (ADOT) is leading the 511 Model Deployment in partnership with other transportation agencies in the Phoenix and Tucson regions, including local traffic jurisdictions, airports and public transit operators. The 511 Model Deployment implements a number of key enhancements to the existing statewide 511 system that became operational in March 2002, when ADOT converted their ten-digit system that had been operational for several years to 511. The enhancements to the system include:

· new content on arterial streets, airports, neighboring states (Utah), transit (major service disruptions and estimated arrival times), weather, and downtown Phoenix special events and parking;

· a complete redesign of the menu system, conversion from a keypad system to voice recognition, conversion from a highway route-based reporting to segment-based reporting; and

· enhanced 511 marketing.

Planning for the Model Deployment began in earnest in August 2002 when ADOT convened the 511 Task Force, composed of representatives of the various participating agencies. Planning and design continued through early 2003. Since that time, the focus has been on implementation, with a certain degree of final design occurring as part of the implementation process. The model deployment enhancements are planned to become operational in a phased manner, beginning in late November 2003 and continuing through early-mid 2004.

The current menu system contains only three options: “roadways”, “transit” and “comments” (callers can leave voicemail comments). No transit information is available directly on the current 511 system; all transit selections are routed to the Phoenix and Tucson transit agencies’ customer service lines. Under “roadways” callers can either input the route number of the roadway or, for local street information, the first three letters of a city name. However, the current approach to gathering city street information (which relies on local traffic jurisdictions to enter information) is not effective and essentially, no city information is available.

The roadway information available on the current 511 system is oriented largely toward intercity travelers. Much of the information that is available is for so-called “planned incidents”—non-real-time information entered by ADOT in advance describing roadway construction, route closures and restrictions. Information is provided by Interstate and State Highway routes, rather than route segments, and is reported by milepost. Therefore, someone interested in a quick report on commute conditions on I-10 in central Phoenix, located in the central part of Arizona, would have to listen through (or know enough to skip past) all the information for I-10 between California and Phoenix. The statewide system does cover major metropolitan areas like Phoenix but information of interest to commuters, like travel times and information on local streets, is not available. One important goal of the model deployment is to make the 511 system more useful for commuters in the two largest urbanized areas in Arizona: the Tucson region and the Phoenix region.

Little is known about current usage of the 511 system. The only statistics that are tracked are total hourly call volumes, which are aggregated to daily and monthly totals. That data indicates that prior to converting to the 511 number in March 2002, the usage of the ADOT traveler information telephone system ranged from about 4,000 calls per month upwards to about 20,000 calls during winter storms or during unusual events like major forest fires. After the conversion to 511, usage spiked from around 7,000 calls in January and February to about 20,000 calls per month in March. Since that time, usage has remained at that level or higher, ranging up to as high as 100,000 calls per month (in December 2003). No information on call origin (geographic or cell phone/wireline) is available.

The only interaction ADOT has had with 511 users has occurred as part of the development of the model deployment, when they convened focus groups in Phoenix, Tucson and Flagstaff. A total of 67 people participated in the six focus groups. Very few of the focus group participants had used 511 prior to being recruited, at which time they were asked to familiarize themselves with the system. Most focus group participants found the existing system at best only moderately useful (average ranking of 4.7 out of 10) but indicated that with improvements, they would expect to find the system very useful (8.8 out of 10). Desired improvements included many of the planned model deployment enhancements, including voice recognition; segment-based and sub-region-based reporting; more information on incidents; and “quick reports” highlighting conditions in specific geographic areas, similar to commercial radio traffic reports.

1.2 Evaluation Plan Overview

The overall plan for the national evaluation of the 511 model deployment is presented in the Evaluation Plan report. This section highlights the major elements of the evaluation.

The Statement of Work for the national evaluation identifies the following purposes:

Document how the model deployment was implemented, including system costs and how technical and institutional (especially cross-modal and interstate) issues were resolved.
Provide an independent review of the performance of the model deployment.
Deliver lessons learned to other 511 system development and deployment efforts.

The evaluation consists of three phases. Phase I consists of the evaluation planning activities, including developing the Evaluation Strategy, Evaluation Plan and Detailed Test Plans, and preliminary analysis of baseline data. Phase II is scheduled to conclude in early 2004. Phase II consists of the analysis of the implementation phase portion of the enhancement process, including conclusion of baseline data analysis and documentation of management and deployment issues. Phase II is scheduled to be completed in February 2004. Phase III consists of the analysis of post-enhancement system performance, concluding with the Evaluation Report that will present all findings. Phase III is scheduled to be completed in April 2005.

Evaluation analyses have been organized into individual “tests”, with each test focusing on a particular type of data. The tests and their relationship to the major focal points of national Intelligent Transportation System (of which 511 systems are an example) evaluations are shown in Table 1. In addition to these formal tests, a cost analysis will be performed and various other types of supporting data will be collected and utilized either to aid in the interpretation of test results or to identify management and deployment issues and lessons learned.

Table 1 Evaluation Tests
Analysis Area	Tests
Analysis Area	Usage Logs⁽¹⁾	User Survey	Key Informant Interviews
Customer Satisfaction	X	X
Mobility	X	X
Efficiency	X	X	X
Management and Deployment Issues			X

⁽¹⁾The analysis of system usage will include consideration of system data content (inputs from the various agencies that supply data to the system) and the reliability (e.g., system downtime) and availability (phone line utilization) of the system.

1.3 Report Organization

This report is the Test Plan for the Key Informant Interviews. Following this Introduction, this report contains the following sections:

· Test Plan Objective – including the hypotheses to be tested and key supporting conditions necessary to successfully complete the test.

· Test Data – a description of data requirements, data sources, and data collection methods.

· Analysis Methods

· Test Schedule

· Results Report Format and Contents

· Estimated Resource Requirements

2.0 Test Plan Objective

As indicated in Table 1 on page 3, the Key Informants Interviews test will serve two purposes. First, the interviews will be the primary mechanism for identifying various management and deployment issues associated with the model deployment (including lessons learned). Second, the interviews will help support the evaluation of several of the project partners’ objectives related to improving the efficiency of the 511 system operation.

In regard to addressing efficiency objectives, interviews with ADOT technical staff responsible for overseeing the operation of the Highway Conditions and Reporting System (HCRS) and the Voice Response Activated System (VRAS) servers will complement the system data in the assessment of system availability. Interviews with ADOT Traffic Operations Center (Phoenix headquarters) HCRS operators and supervisors will complement the system input data in the assessment of improvements in the capture of arterial street data. Interviews with HCRS operators and supervisors at Tucson area agencies (ADOT District Offices, City of Tucson traffic, airport and transit) will help assess the extent to which the model deployment is successful in making available more information on Tucson. Interviews with HCRS operators and supervisors will also help assess the effectiveness of increased training and changes in data entry and verification procedures.

2.1 Hypotheses

The primary purpose of key informant interviews is to identify deployment issues and their resolutions, and, thus, the Evaluation Plan does not identify any specific management and deployment-related hypotheses. Rather, the interviews will be guided by questions/topics and prompts which reflect the evaluation team’s expectations regarding potential issues of interest, and where appropriate they will help illuminate results found in data collected relative to hypotheses in other test plans. As summarized in Table 2, the key informant interviews are expected to provide information that will assist in the evaluation of several hypotheses related to certain project objectives. The far right column in Table 2 identifies (in bold) potential roles for key informant interview data and identifies data from other test plans that will support the hypothesis testing of specific model deployment objectives.

Table 2

Hypotheses Related to Other (Non-Management and Deployment Issues)

Project Objectives

National ITS Goal Area

Objective

Hypothesis to be Tested

Key Informant Interviews and

Other Data⁽¹⁾

Efficiency

Maintain acceptable system availability.

Improved VRAS server and HCRS server reliability will maintain acceptable system availability.

· Interviews with ADOT technical staff

· ADOT HCRS and VRAS system data

· ADOT 511 caller comment records

Increase the effectiveness of capturing arterial street incident data.

Having ADOT Traffic Operations Center Staff monitor law enforcement scanners will increase the amount of arterial street traffic information.

· Interviews with ADOT TOC HCRS operators and supervisors

· System input data

Providing Tucson area agencies the ability to enter information to 511 will increase the amount of Tucson area information.

· Interviews with Tucson area agency HCRS operators and supervisors

· System input data

Improve the quality of data entered into the 511 system.

Increased training of HCRS operators, refinement of landmarks and terminology used in HCRS, and enhanced on going monitoring of data quality will result in more effective data entry, with fewer inaccurate and duplicative entries.

· Interviews with HCRS operators and supervisors and staff responsible for developing and implementing training and revisions to manuals

· HCRS operators and supervisors’ manuals and training materials.

⁽¹⁾ Other data (other than key information interview data) is addressed in other test plans. These other data sources are shown here to illustrate how interview data will be used in conjunction with other data.

2.2 Key Supporting Conditions

The key supporting condition for successful execution of this test is the cooperation of the interviewees. They will need to devote the necessary time to participate in the interviews and to speak freely regarding their perspective on the model deployment.

3.0 Test Data

3.1 Data Requirements

Key requirements of the key informant interview data consist of the following:

· To be useful, the input from the interviewees should be direct and forthright, which requires that they feel reasonably at ease in sharing their perspectives on the model deployment.

· The input from the interviewees should conceptually address all of the various potential types of management and deployment issues relevant to the specific interviewee. That will require that a comprehensive list of potential questions and examples be prepared that will effectively probe for input across the full range of potential areas. Any given interviewee may not have input in all areas, but it will be important to structure the interviews so as to draw out all of the potential input.

· Interview data will consist of notes taken by the interviewer. The notes must be detailed and accurate.

3.2 Data Sources

Table 3 identifies the key informants that will be interviewed. The primary focus of most interview sessions will be on identifying management and deployment issues. However, the sessions that include the ADOT HCRS Supervisor, ADOT HCRS operators, City of Tucson HCRS entry personnel, and staff of PBS&J will include deeper discussion about the various data quality and data capture activities of the model deployment. That information will contribute to investigation of hypotheses related to improving the efficiency of the 511 system (see Table 2, above). Interview subjects include all of the major active participants in the model deployment, including those leading the design, implementation and operation (ADOT and their consultants) and an organization or individual associated with each new data source, e.g., airports, transit, etc.

Two rounds of interviews will be conducted for identification of management and deployment issues, the first focusing on implementation and the second focusing on operation. Information needed to support the testing of efficiency-related hypotheses will be collected during the management and deployment interviews.

Not all management and deployment issue interviewees will participate in both rounds of interviews. The first round of interviews will include only those organizations and individuals playing a major role in planning, designing and implementing the system, consisting of ADOT, their consultants, and Valley Metro (Phoenix transit). The second round of management and deployment interviews will include all of the organizations and individuals identified in Table 3.

Table 3 Key Informant Interview Subjects
Type of Interview/Organization	Interviewee	Role in the Model Deployment
Agencies
Federal Highway Administration	Alan Hansen Transportation Engineer	FHWA Project Liaison
Arizona Department of Transportation – Headquarters (Phoenix)	Tim Wolfe Assistant State Engineer	Project Manager
	Darrell Bingham Information Technology Section Manager	Project Technical Coordinator and Evaluation Point-of-Contact
	Donna Ayers Public Information Officer	Compiles caller comments from 511 voicemail box. Coordinates marketing activities.
	Linda Anestasi HCRS Supervisor	Overseeing revisions in HCRS operator procedures to improve data quality.
	To be identified HCRS Operators	Inputting information into HCRS.
Arizona Department of Transportation – Tucson District	To be identified	Inputting Tucson area freeway information into HCRS.
Valley Metro (Phoenix Transit)	Mike Nevarez	Providing transit major service disruption and estimated bus arrival time information to the 511 system.
Sky Harbor (Phoenix Airport)	To be identified	Providing airport construction and parking information to the 511 system.
City of Phoenix Traffic	Bob Steele	Providing information from Downtown Traffic Management System and participating in Arterial Street Travel Time enhancement.
City of Tucson Traffic	Richard Nassi Ron Platt	Inputting City of Tucson roadway information.
Pima County	Albert Letzkus	Inputting Pima County roadway information.
SunTran (Tucson Transit)	Mike Beel	Providing major service disruption information.
Tucson International Airport (Tucson Airport Authority)	Jill Merrick	Providing airport construction and parking information to the 511 system.
Maricopa County (REACT)	Barbara Hauser REACT Manager	REACT (arterial street incident response team) personnel are inputting information to the 511 system via PDA’s.
Utah Department of Transportation	Brian Chamberlin	Contributing information on Utah-Arizona border roadway conditions to the 511 system.
National Park Service (Grand Canyon)	Jim Tuck	Inputting Grand Canyon information.
Consultants/Vendors
OZ Engineering	Tomas Guerra 511 Project Manager	Performing all enhancements to HCRS.
Call Processing	Larry Atkins 511 Project Manager	Performing all enhancements to the 511 phone system (VRAS), including implementation of new menu system and voice recognition.
PBS&J	Eli Sherer Rick Schuman Pete Costello	Providing overall program management support (developing individual task work orders, conducting and documenting meetings, monitoring schedule and budget, etc.) and leading several tasks, including design of new menu system, data quality enhancements (HCRS operator procedures), and segment weather reports.
Kimley-Horn Associates	Pierre Pretorius 511 Project Manager	Developed Marketing Plan.
Kimley-Horn Associates	Lisa Burgess	Developed Marketing Plan.
PIPS Technology, Inc.	Tom Owen 511 Project Manager	Providing arterial street travel time estimates to the 511 system.

If it can be coordinated, after each of the two rounds of management and deployment issue interviews, the regular 511 Task Force meetings will be used as an opportunity to review and potentially verify and expand input of the individual interviewees. It is likely that the task force will hold a meeting at approximately the time of the two rounds of interviews, which represent key milestones, and so it is expected that this activity can occur.

3.3 Data Collection Methods

3.3.1 Grouping Interviews

Given the large number of individuals and organizations participating in the model deployment who are expected to provide useful input on management and deployment issues, and given limited evaluation resources, interview subjects have been grouped into sessions, as indicated in Table 4. In grouping interviewees, the potential for different individuals to either inhibit or stimulate information exchange if grouped together both have been considered and groupings were identified accordingly. For example, grouping Phoenix and Tucson transit agencies together will hopefully stimulate discussion of transit 511 issues in general. Table 4 indicates which interview sessions, and in some cases which interviewees within a given session, will be interviewed in-person and which will be interviewed via teleconference.

The shaded interview sessions are those that will occur in both the first (oriented around planning, design and implementation) and second rounds (operation and overall project) of management and deployment interviews. The non-shaded sessions, which include organizations and individuals involved primarily in the operational phase of the deployment, will occur only in the second round of interviews.

Table 4 Interview Sessions
Interview Session	Participants	Method – Location
1.	ADOT Headquarters - Management Kimley-Horn	In person – Phoenix
2.	Valley Metro SunTran	Valley Metro in person - Phoenix SunTran via teleconference
3.	ADOT Headquarters – HCRS Operators	In person - Phoenix
4.	ADOT Tucson District City of Tucson Traffic Pima County	In person – Tucson
5.	Sky Harbor International Airport Tucson International Airport	Sky Harbor in person - Phoenix Tucson via teleconference
6.	City of Phoenix Traffic Maricopa County	In person – Phoenix
7.	FHWA	Via teleconference
8.	National Park Service	Via teleconference
9.	Utah Department of Transportation	Via teleconference
10.	OZ Engineering PBS&J	OZ in person – Phoenix PBS&J via teleconference
11.	PIPS Call Processing	Via teleconference

= two rounds of interviews to be conducted; all others are one round, near the end of the one-year operational period.

3.3.2 Interview Questionnaire

Interview questionnaires will be used for both the first and second round of management and deployment issue key informant interviews. During the scheduling of the interviews the role of the interviews and their importance to the national evaluation will be explained to the interviewees.

The questionnaire for the first round of interviews is shown in Table 5. These interviews will be conducted approximately one month after the basic enhanced 511 system has become operational and will focus on the planning and implementation phase of the deployment. The discussion of various management and deployment issues associated with individual enhancements (question #5) is supported by the check-list and specific prompts shown in Table 6. Those prompts reflect the evaluation team’s preliminary understanding of some of the potential issues that were observed during the development of the evaluation planning documents. Table 7 presents the round 2 questionnaire for key informant interviews conducted near the end of the one year of enhanced 511 operation.

Table 5 Round 1 Interview Questionnaire
Role and Expectations	What has been your organization’s role, and your personal role, in the planning, design and implementation of the 511 system? What is your organization’s objective in participating in the deployment; what benefits did you expect when you decided to participate in the project? Have your expectations of benefits changed since the project got started? In what way; why? What would constitute a “success” from your organization’s point of view? What about for the model deployment overall? Has the definition of success changed since the project has started? In what way; why?
Management and Deployment Issues	During the planning and implementation portion of the model deployment, what have been the challenges? How have they been addressed? Have they been effectively overcome? What was your approach to dealing with the risks (prompts below categorize the types of risk)—were there fall-back plans for the major enhancements like voice-recognition, transit, and arterial street travel times? Prompts: i. Mobilizing support within your organization, including senior management buy-in. ii. Mobilizing the support of the other 511 partners, including development of agreements? iii. In working with the other 511 partners, have you been satisfied with the level of cooperation, including honoring of data input commitments? What went well and what didn’t go well? What would have made things go better? iv. Organizational staff resources (labor hours, training, administrative burden). Were any changes to your organization’s organizational structure or way of doing business required? v. Delegation of 511 model deployment responsibilities within your organization and among the various partners vi. Public expectations vii. Legal/regulatory issues, including liability, procurement/acquisition, profits, etc. Any changes required in your organization’s policies? viii. Cost-sharing among 511 partners During the planning and implementation phase of the deployment, what have been the technical and other issues associated with the specific 511 enhancements (Review checklist in Table 6) and of the deployment overall?
Performance	Now that the enhanced system has become operational, how well does it measure up to the original vision, from your organization’s point of view? For each of the following aspects of the deployment thus far, please rate the success on a scale of A (excellent) through F (failure): i. Schedule ii. Budget iii. Agency adherence to commitments iv. Content added/enhanced in the 511 system v. Phone system modifications, e.g., voice recognition and menu vi. Overall deployment
Lessons Learned	What would you do differently in the planning, design and implementation phase assuming the same level of funding? What if you had twice the funding? What do you feel have been the lessons learned from the model deployment so far?

Table 6 Potential Issues List by 511 Enhancement
511 Model Deployment Enhancement	Prompts
Tucson and Phoenix Local Street Data	· Why has ADOT decided to take responsibility for entering Phoenix region local arterial street information? Why aren’t the individual agencies doing it, since they have the ability? · What’s the long-term solution? · What sort of staffing impact does this new responsibility have on ADOT? · Is the approach working? Why/why not? · What were the impediments to getting Tucson data into the system prior to the model deployment and how have they been overcome? · Is the data input from City of Tucson consistent with expectations? (interview input will later be compared to HCRS event entry data)
Standards Upgrades	· Our understanding is that the only new standard associated with the deployment is SAE J2354 (ATIS Message Set), used to import Utah data. Is that correct? How has that effort gone? · What standards have been used for interfacing with Valley Metro’s vehicle management system?
Utah Data	· Working as planned? · What have been the challenges and solutions? · Is Utah also receiving AZ data? How are they using it?
Data Quality Improvements	· Have the intended revisions to procedures been implemented? · Have HCRS operators responded well to the revised procedures? · Are the revised procedures effective? How? Why? · How is the effectiveness being monitored?
Phoenix Arterial Street Travel Times	· Working as planned? · What have been the challenges and solutions? · The contractor’s proposal suggests that the effort is under-funded (loss leader for them). Do you think the budget is sufficient? · What considerations have shaped the decision about the format/nature of the information to be provided to the public?
Transit Service Disruptions and Bus Arrival Times	· Have expectations been met? Why or why not? · How does transit information on 511 relate to transit agencies’ stated plans for their own IVR? What is the expected relationship between the two systems? · What is the strategy relative to marketing 511 to transit riders? · How has the 511 system interfaced with Valley Metro’s vehicle management system? · Have the enhancements to the 511 system made it more feasible for you to contribute data? Why or why not? What features have made the new system more conducive to data entry by your agency? How?
Phoenix and Tucson Airport Information	· Have expectations been met? Why or why not? · Is the staffing commitment required for the airports to enter their own information problematic? · Have the enhancements to the 511 system made it more feasible for you to contribute data? Why or why not? What features have made the new system more conducive to data entry by your agency? How?
Grand Canyon National Park Data	· Have expectations been met? Why or why not? · Is the resource commitment required for the park staff to enter their own information problematic? · Given that the park had the ability to input information prior to the model deployment but was not inputting much, what’s been done to improve that and has it worked?
Segment Weather Reports	· How is current approach different than the original concept (i.e., using NWS rather than a private provider)? What considerations shaped the current approach? · How useful is the NWS information? What has been done to make it useful in the Arizona 511 system?
Downtown Phoenix Special Event and Parking Information	· Have expectations been met? Why or why not? · Did the original vision include more real-time parking capacity information? · What have been the issues and how have they been addressed? · Have the enhancements to the 511 system made it more feasible for you to contribute data? Why or why not? What features have made the new system more conducive to data entry by your agency? How?
Segment-Based Highway Reports and Regional Quick Reports	· What have been the challenges relative to converting to a segment and region-based reporting scheme? Has it worked?
Voice Recognition and Menu System Redesign	· Have expectations been met? Why or why not? · What have been the issues and how have they been addressed? · How has it been working with the phone companies? · What have been the experiences with the caller comment option? · What have been the experiences and issues associated with the un-avoidable “floodgate” message at the top of the menu?
System Tracking, Reporting and Performance Assessment	· ADOT had very limited data on system performance prior to the deployment. What has changed and is it helpful?
Premium Service (Private Partner)	· What contributed to the termination of plans to partner with the private provider? · What challenges were faced?
Enhanced 511 Marketing	· Were plans carried out as intended? · Have expectations been met? · What have been the issues and challenges? · Have these been adequate, i.e., produced results?

Table 7 Round 2 Interview Questionnaire
Role and Benefits	Has your role in the operational phase of the model deployment been as expected, or has it changed? How and why? Were the benefits your organization hoped to realize through participation in the model deployment realized? Why or why not?
Management and Deployment Issues	During the operational phase of the deployment, what have been the challenges? How have they been addressed? Have they effectively been overcome? Prompts: i. Staff resources and costs ii. Capital resources (facilities, equipment) and costs iii. Working with 511 partners (adhering to commitments, etc.) iv. Public reactions v. Working with the media vi. Reactions of senior staff, including maintaining your agency’s commitment to fulfill your role in the project? 4. During the operational phase of the deployment, what have been the technical and other issues associated with the specific 511 enhancements (Review prompts list – new version of Table 6 to be developed following first round of interviews) and of the deployment overall?
Performance	5. What has been the most positive outcome of the model deployment? What’s worked well? Anything else that stands out in your mind? 6. What has been the most negative outcome of the model deployment? Anything else that stands out in your mind? 7. Now that the enhanced system has been in operation for about a year, how well does it measure up to the original vision? For each of the following aspects of the deployment thus far, please rate the success on a scale of A (excellent) through F (failure): i. Schedule ii. Budget iii. Agency adherence to commitments iv. Content added/enhanced in the 511 system v. Phone system modifications, e.g., voice recognition and menu vi. Overall deployment How often and on what occasions do you use the 511 system? Do you find it useful (rate it on a scale of 1-10)? How could it be improved for you, as a user?
Lessons Learned	9. What would you do differently? 10. What do you feel have been the major lessons learned from the model deployment? 11. What advice would you give other organizations thinking of undertaking similar 511 activities?
Looking Forward	12. What major questions do you think remain unanswered relative to 511 systems? How do you think those questions can be answered? 13. What’s the next step for the Arizona 511 system; what do you think still needs to be done?

The interviews will be conducted by a single member of the evaluation team. Interview questions/topics will be provided to the interview subjects in advance. Notes will be taken during the interviews on a special note form that will include sections for each question and the response from each organization participating in the interview session.

The interviewer will endeavor to establish a non-judgmental, casual atmosphere so as to promote the free exchange of ideas. This will be attempted through opening statements intended to loosen people up (e.g., “there are no wrong answers”; “your opinions matter”; “it’s OK if you disagree on things”, etc.) and a friendly, casual (yet structured) interview style. The effort to relax the interview subjects should be aided by the fact that the primary interviewer (Matt Burt) is well known to most of the interview candidates; they have worked together before and have a relationship that transcends that of “Federal Evaluator” on this particular project.

Questions will be asked in “round robin” format, that is, each question will be asked separately to representatives of each organization participating in the interview, responses will be recorded individually, and then any combined discussion will be captured. With the permission of the interviewees, all interviews will be tape recorded to help with documentation. Plans to tape the interviews will be made clear during the initial scheduling contact. Interviews are expected to last no more than 2 hours. The interview with the ADOT 511 team is expected to be the longest interview, with interviews with some of the more peripheral participants in the model deployment (e.g., airport) taking considerably less time.

4.0 Analysis Methods

Immediately following each round of interviews, all of the interview notes (and tapes, as necessary) will be studied and a summary of results will be prepared. The summary will be organized by interview question with a final section identifying major themes.

Assuming that the schedule for the two rounds of interviews can be coordinated with scheduled ADOT 511 Task Force meetings, draft summaries of the interviews will be discussed with the 511 Task Force at meetings to be held shortly after completion of the interviews. Notes will be taken that capture those discussions and the results will be reflected in the final version of the interview summaries.

5.0 Test Schedule

The first round of interviews will occur approximately one month after the basic enhanced 511 system, which will include voice recognition and the new menu structure, is completed. This first round of interviews will focus on identifying management and deployment issues associated with the planning, design and implementation phase of the project. The second round of interviews will occur near the end of the one year of system operation and will focus on issues encountered during the operational phase of the project, as well as overall project lessons learned.

Given the partners’ current plans to implement the basic enhanced 511 system by the end of November 2003, the first round of interviews is planned to begin in mid-January 2004, after the first of the year holiday/vacation season. The second round of interviews is planned to begin in December 2004.

It is expected that the first round of interviews, which includes six organizations comprising four interview sessions, can be completed over a two week period. The second round of interviews, which includes 17 organizations in ten interview sessions, is expected to be completed over a period of approximately one month.

Pre-Test activities, to be conducted up until the first round of interviews, consist of continued monitoring of the 511 deployment and revisions to the individual enhancement-related questionnaire prompts (Table 7) as appropriate. Test activities consist of scheduling, conducting, analyzing and documenting the interviews, and the follow-on 511 Task Force meeting debriefings.

6.0 Results Report Format and Contents

The results of this test will be documented in two ways. First, the results of the first round of interviews—those focusing on the planning and implementation portion of the model deployment—will be presented in an enhancement process briefing to the USDOT. That briefing is scheduled for February 2004. The full results, including both the round 1 and round 2 interview results as well as an overall project summary, will appear in the Model Deployment Evaluation Report scheduled for February 28, 2005.

The briefing will include the following elements:

· Summary of the model deployment

· High-level summary of the evaluation plan

· Summary of the model deployment planning and implementation process

· Summary of key informant interview process

· Summary of enhancement process management and deployment issues

The management and deployment issues chapter of the Model Deployment Evaluation Report will include an expanded (non-summary) version of the last three items, as well as a conclusions section that identifies remaining questions to be addressed through future 511 deployments and implications and guidance for other 511 deployers.

Table 8 Estimated Resource Requirements
Task	Hours by Staff
Task	Zimmerman	Burt	Jenq	Total
Round 1 Interviews and 511 Task Force Debriefing	8	36	6	50
Analysis of Round 1 results and USDOT Enhancement Process Briefing	2	6	4	12
Round 2 Interviews and 511 Task Force Debriefing		54	6	60
Analysis of Round 2 results and Model Deployment Evaluation Report	6	20	4	30
Total	16	116	20	152