|
Advanced Public Transportation Systems: Evaluation Guidelines January 1994
Advanced Public Transportation Systems: Evaluation Guidelines January 1994 Click HERE for graphic. Advanced Public Transportation Systems: Evaluation Guidelines Final Report January 1994 Prepared by Robert F. Casey and John Collura John A. Volpe National Transportation Systems Center Kendall Square Cambridge, MA 02142 Prepared for Office of Technical Assistance Federal Transit Administration 400 Seventh Street SW Washington, DC 20590 Distributed in Cooperation with Technology Sharing Program U.S. Department of Transportation Washington, D.C. 20590 DOT-T-94-1 0 PREFACE This document was prepared by the Office of Research and Analysis, Volpe National Transportation Systems Center, under the sponsorship of the Advanced Public Transportation Systems (APTS) Program, Federal Transit Administration (FTA) and with the guidance of Mr. Ronald Fisher, FTA's Director of Training, Research, and Rural Transportation. The Volpe Center operates under the auspices of DOT's Research and Special Programs Administration (RSPA). The major contributors were Mr. Robert Casey, RSPA/Volpe Center Operations Research Analyst, and Dr. John Collura, RSPA/Volpe Center Faculty Fellow and Professor of Civil Engineering at the University of Massachusetts, Amherst. Technical assistance also was provided by Ms. Judith Schwenk and Mr. Lawrence Labell of the RSPA/Volpe Center and Dr. Thomas Horan of the Institute of Public Policy at George Mason University. The summaries of the breakout sessions at the recent National Workshop on APTS Evaluations also were useful in the completion of the guidelines. The summaries were prepared by Ms. Katherine Turnbull of the Texas Transportation Institute, Mr. John Mason of Science Applications International Corporation, Mr. Joel Markowitz of the Metropolitan Transportation Commission (San Francisco), and Mr. Philip Shucet of Michael Baker Jr., Inc. The preparation of this document was also facilitated by prior evaluation work by Mr. Mark Abkowitz, Ms. Carla Heaton, Mr. Chester McCall, Mr. Howard S. Slavin, and Mr. Robert Waksman as part of the Federal Transit Administration's Service and Methods Demonstration program. The document consists of evaluation guidelines for use by contractors responsible for evaluating APTS operational tests. Although these guidelines are intended for the APTS Program, their potential applicability extends beyond the evaluation of FTA-sponsored operational tests to the evaluation of any innovative use of advanced technology in public transportation. It is anticipated that this document will be modified periodically to reflect additional experience gained in evaluating APTS operational tests. iii Click HERE for graphic. TABLE OF CONTENTS PAGE 1. OVERVIEW OF EVALUATION GUIDELINES 1 2. BACKGROUND 3 2.1 Overview of the Evaluation Process 7 2.1.1 Evaluation Frame of Reference 7 2.1.2 Evaluation Planning 12 2.1.3 Evaluation Implementation 13 2.1.4 Potential Evaluation Spin-offs. 15 2.2 Coordination of APTS Evaluations 16 3. GUIDELINES FOR PLANNING EVALUATION ACTIVITIES 19 3.1 Determination of Site Data Requirements and Sources 21 3.2 Determination of Measures and Collection/ Derivation Techniques Required to Address APTS; Program Objectives and Other Relevant Project Objectives/Issues 24 3.2.1 Basic Set of Measures 25 3.2.1.1 APTS Costs and Functional Characteristics 27 3.2.1.2 User Acceptance 28 3.2.1.3 System Efficiency and Effectiveness 28 3.2.1.4 Impacts 29 3.2.1.5 Relationship Between APTS Program Objectives and the Categories of Measures 29 3.2.1.6 Other Objectives and Measures 34 3.2.2 Data Collection/Derivation Techniques 34 3.3 Planning Considerations Relative to Data Collection and Analysis 38 3.3.1 Basic Data Collection/Analysis Design 39 3.3.2 Measure Stratification 43 3.3.2.1 Categorization of a Measure Into Additive Components 44 3.3.2.2 Categorization of a Measure According to Target Market, Operational, Geographic, or Time Categories 48 3.3.3 Grouping of Raw Data Into Class Intervals 49 3.3.4 Sampling Requirements 51 3.3.5 Timing of Data Collection 52 v 4. GUIDELINES FOR PERFORMING EVALUATION ACTIVITIES 55 4.1 Monitoring/Performance of Data Collection 55 4.2 Data Reduction, Analysis, and Presentation 59 5. CONTENT AND ORGANIZATION OF REPORTS 75 5.1 Evaluation Plan 75 5.2 Monthly Evaluation Progress Reports 77 5.3 Interim Evaluation Reports 78 5.4 Final Summary Evaluation Report 78 5.5 Quarterly Project Progress Reports 79 APPENDIX A - SURVEY EXECUTION AND DESIGN A-1 A-1 Defining the Survey Universe A-2 A-2 Sampling the Survey Universe A-3 A-3 Techniques for Surveying the Samples Selected A-7 A-4 Survey Design Principles A-10 A.4.1 Organization A-11 A.4.2 Length A-11 A.4.3 Question Sequence and Wording A-12 A.4.4 Standardized Questions A-14 A.4.4.1 Behavioral Measures A-14 A.4.4.2 Attitudinal Measures A-15 A.4.4.3 Social and Demographic Measures A-16 A.5 Non-Response Bias A-17 A.6 Interviews With Transportation Agency Personnel A-18 A.7 References A-19 vi APPENDIX B - STATISTICAL METHODOLOGY B-1 B.1 Definitions B-1 B.2 Data Analysis Determination B-2 B.3 Sample Size Determination B-4 B.4 Data Collection B-7 B.5 Analysis Methods B-7 B.6 Methodology Documentation B-10 B.7 References B-10 APPENDIX C-GLOSSARY C-1 APPENDIX D-BIBLIOGRAPHY D-1 APPENDIX GENERAL WORK TASKS OF EVALUATION CONTRACTOR E-1 E.1 Task 1: Evaluation and Task Administration Plans E-1 E.2 Task 2: Implementation and Analysis of Data E-2 E.3 Task 3: Report Preparation E-3 vii LIST OF EXHIBITS PAGE EXHIBIT 1: Selected Examples and Applications of APTS 4 EXHIBIT 2: Evaluation Relationships 6 EXHIBIT 3: Evaluation Process 8 EXHIBIT 4: APTS Operational Test Planning, Implementation, and Evaluation Sequence of and Responsibility for Activities 17 EXHIBIT 5: Basic Site Data Requirements for APTS Operational Tests 20 EXHIBIT 6: Typical Sources for Site Data 22 EXHIBIT 7: APTS Program Objectives and Examples of Corresponding Measures 31 EXHIBIT 8: Examples of Data Collection Techniques for Selected Measures 36 EXHIBIT 9: FTA/Section 15 Worksheet for Functional Distribution of Expense Object Classes/ Level B 47 EXHIBIT 10: Service Area for the Seattle Project 63 EXHIBIT 11: Distribution of Park-and-Ride Users for the Seattle Project 64 EXHIBIT 12: Passenger Volume for the Seattle Project 65 EXHIBIT 13: Bus Schedule Adherence for the Minneapolis Urban Corridor Project 66 EXHIBIT 14: Corridor Demographic Characteristics for the Shirley Highway Express Bus-On-Freeway Project 67 EXHIBIT 15: Charge-A-Ride Usage by Card Type and Time Period for the Merrimack Valley Charge-A-Ride Program 68 EXHIBIT 16: Comparison of Fare Payment Times Using Different Methods 69 EXHIBIT 17: Project Effectiveness Measures for the Seattle Project 70 EXHIBIT 18: Highway Travel Time Distributions for the Minneapolis Urban Corridor Project 71 viii EXHIBIT 19: Results of Before and After Analyses for Portland Self-Service Fare Collection 72 EXHIBIT 20: Benefit-Cost Analysis Results of Salt Lake City Rider Information System 73 EXHIBIT A-1: Summary of Survey Sampling Methods and Applicable Survey Techniques A-20 EXHIBIT A-2: On-Board Bus Survey -- Katy Transitway Transit User Survey A-21 EXHIBIT A-3: Carpool/Vanpool Survey A-22 EXHIBIT A-4: Freeway Motorist Survey A-24 EXHIBIT A-5: Bus Riders Survey A-26 EXHIBIT A-6: Bus Riders Mail-Back Survey A-28 EXHIBIT A-7: Bellevue Smart Traveler Project Surveys A-30 EXHIBIT A-8: Washington, DC, Self-Administered Post Card Bus Survey A-34 EXHIBIT A-9: 1979 Downtown Crossing Bus Passenger Survey A-35 EXHIBIT A-10: Recommendations for Questions on Boarding and Alighting Points (for user surveys only) A-36 EXHIBIT A-11: Recommendations for Questions on Trip Origin A-37 EXHIBIT A-12: Recommendations for Questions on Trip Destination A-38 EXHIBIT A-13: Recommendations for Questions on Trip Start and End Times A-39 EXHIBIT A-14: Recommendations for Questions on Access Mode to Transit Vehicle A-40 EXHIBIT A-15: Recommendations for Questions on When Present Mode was First Used A-41 EXHIBIT A-16: Recommendations for Questions on Former Transportation Mode A-42 EXHIBIT A-17: Set of Attitudinal Questions on Travel by Transit and Auto A-43 EXHIBIT A-18: Set of Questions on General Attitudes of the Population A-44 ix EXHIBIT A-19: Recommendations for Questions on Respondents' Sex A-46 EXHIBIT A-20: Recommendations for Questions on Respondents' Age A-47 EXHIBIT A-21: Recommendations for Questions on Respondents' income A-48 EXHIBIT A-22: Recommendations for Questions on Auto Availability A-49 EXHIBIT A-23: Recommendations for Questions on Auto Ownership A-50 EXHIBIT A-24: Recommendations for Questions on Whether Respondent has a Driver's License A-50 EXHIBIT A-25: Recommendations for Questions on Respondents' Occupation A-51 EXHIBIT A-26: Recommendations for Questions on Respondents' Educational Level A-52 EXHIBIT A-27: Recommendations for Questions on Length of Residence A-53 x 1. OVERVIEW OF EVALUATION GUIDELINES This document presents guidelines for planning, implementing, and reporting the findings of the evaluation of Federal Transit Administration's (FTA) Advanced Public Transportation Systems (APTS) operational tests. These evaluation guidelines are intended for use by organizations engaged by the Research and Special Programs Administration/Volpe National Transportation Systems Center (Volpe Center) to evaluate the APTS operational tests. In addition, the guidelines will be useful to state and local organizations involved in the design and evaluation of Advanced Public Transportation Systems. An objective of these guidelines is to foster consistency of evaluation philosophy and techniques, and comparability and transferability of results to improve the quality and utility of information obtained from the APTS program. The guidelines are designed to emphasize the assessment of the APTS Program's national objectives as well as the objectives of the local implementing agency. The various operational tests implemented under the APTS Program are meant to serve as learning tools and/or as models for other locales throughout the country. In order for these tests to have maximum effectiveness in their respective operational capacities, a consistent, carefully structured approach to project evaluation is desirable. This document has been prepared to provide a common framework and methodology for developing and then executing the evaluation of individual operational tests. These evaluation guidelines are by no means comprehensive--that is, they do not offer a suggested or preferred course of action for every conceivable situation that might arise. Nor are they to be rigidly or blindly followed, since each operational test and each site wilt be unique and wilt require somewhat tailor-made evaluation procedures. It is anticipated that these guidelines wilt be modified during the course of the APTS Program to reflect experience gained in implementing and monitoring the evaluations of individual tests. Although it is not the desire to update these guidelines frequently, modifications resulting from field experience wilt be made where appropriate for enhancement of performance and evaluation of the various projects. 1 In order to put these guidelines into a meaningful context, Chapter 2 provides background information on the FTA/APTS Program and the operational test evaluation process. Chapters 3 and 4 present guidelines relative to planning and executing operational test evaluations. Finally, Chapter 5 presents the recommended content and organization for each type of report to be prepared in conjunction with the evaluation process. 2 2. BACKGROUND The Federal Transit Administration has developed the Advanced Public Transportation Systems (APTS) Program which is an integral part of the overall U.S. DOT Intelligent Vehicle Highway Systems (IVHS) effort. A major aim of the APTS Program is to promote research and development of innovative applications of advanced navigation, information, and communication technologies. These technologies would be designed and tested to achieve APTS Program goals directed toward enhancing the ability of public transportation systems to satisfy customer needs and contributing to the achievement of broader community goals and local objectives. The APTS Program goals and objectives will be discussed further within the context of the evaluation frame of reference. The wide array of new technologies provides a unique opportunity to discover innovative and useful applications in public transportation. These operational tests and evaluations will be the principal activities of the APTS Program. Real world testing will be done in urban and rural areas using those technologies which appear to offer promise and represent useful applications. Major technologies include automated vehicle location systems, smart card systems, dynamic ridesharing systems, passenger information systems, high occupancy vehicle systems, and vehicle component monitoring systems. Exhibit 1 provides selected examples of these technologies and associated applications. Tests will involve joint ventures with state and local governments, and, when appropriate, universities and private vendors. Tests may range from 3 - years: 1-2 years to develop implementation plans, 1 year to implement service, and 1 year to evaluate the APTS application and associated impacts. In order for the APTS Program to encourage significant technological innovations by many urban and rural areas, the technologies tested and the results obtained must be evaluated, well documented, and widely distributed. It is important not only that the operational tests be structured and evaluated to facilitate transferability of results but also that evaluation results be disseminated so that prospective beneficiaries in other urban and rural areas are made aware of the potential of such technologies. Accordingly, a significant element of the APTS Program is the technology sharing function. 3 EXHIBIT 1. SELECTED EXAMPLES AND APPLICATIONS OF APTS APTS Examples Applications . Automated vehicle location (AVL) . controlling and monitoring system using satellite or ground- the use of vehicles based technologies and compute- rized dispatching techniques . estimating vehicle positions to assist dispatchers in improv- ing on-street schedule adherence . obtaining boarding and alighting information in conjunction with automatic passenger counters (APCs) . assisting in the development of more realistic schedules . facilitating the assignment of individuals to shared ride, demand response services . assisting in the preparation of daily driver logs . Smart card systems using a contact . facilitating the or contactless plastic card with a collection of fares, microchip and storage and processing the verification of capabilities travel, and the acquistion of information about passengers and vehicle usage . encouraging the coordination of various modes including bus, rail, auto, and parking services . aiding in the establishment of a postpayment fare system and the application of employer and human service agency-based subsidy programs . assisting in the design of a comprehensive, historical vehicle maintenance and parts inventory database 4 EXHIBIT 1 (continued) APTS Examples . Dynamic ridesharing systems using . providing quick real-time communication methods and easy access to with the aid of touch-tone telephone, up-to-date information television, radio, and videotex to aid an individual systems in arranging a carpool or vanpool the same day or evening before a trip . Passenger information system . supplying passengers using audio, visual, and/or hard with real time infor- copy methods such as digitized mation on routes, voice, interactive television, schedules, cancella- videotex, automated map displays, tions, delays, rerout- computer monitors and printers, and ing, and other aspects other devices located in terminals, of service to make stations, vehicles, places of travel easier and to employment, and at home; also facilitate intermodal could be provided in conjunction transfers with a traffic management center (TMC). . High occupancy vehicle systems . providing traffic (HOVs) including preferential control signal treatment methods and park and ride preemption capabilities facilities . monitoring vehicle occupancy remotely to enforce HOV lane restrictions . Vehicle component monitoring . assisting in the early systems detection of problems with vehicle components (e.g. engine, exhaust system) to avoid component failure while vehicle is in The exact number, general content, and location of the APTS operational tests are yet to be determined. For each fiscal year program, a series of primary objectives will be selected, and a group of proposals corresponding to each objective, and in keeping with total budgetary constraints, will be developed. Then, following an investigation, analysis, and negotiation 5 process involving FTA, the Volpe Center, and candidate sites, a final set of operational tests and respective sites will be agreed upon. Once final negotiation and transfer of funds between FTA and the APTS local sponsor are completed, the operational test can be implemented and evaluated. As part of its responsibility to evaluate the operational tests implemented under the APTS Program, the Volpe Center shall engage contractor support to participate in all phases of the evaluation process. Exhibit 2 shows the interaction among FTA, the Volpe Center, the local sponsor, the evaluation contractor, and the APTS vendors involved in the operational test. FTA/APTS staff is responsible for overseeing and guiding all aspects of the operational test including planning, site selection, negotiations with the site, implementation, and evaluation. The local sponsor is responsible for planning and implementing the actual conduct of the operational test as well as performing most of the data collection. The Volpe Center assists FTA in the activities for which FTA is responsible, and directs and monitors the efforts of the evaluation contractor. The Volpe Center, the evaluation contractor, and the vendors interface with the local sponsor (or the implementing agency, if different from the local sponsor). While being directly responsible to the Volpe Center for its activities, the evaluation contractor will 6 maintain an informal association and relationship with the local sponsor, the APTS vendors, and the cognizant FTA Project Manager. The APTS vendors, as deemed appropriate by FTA and the Volpe Center, may participate in a review of the evaluation plan, data reduction and analysis, and the interim and final reports. The APTS vendors may serve on the local evaluation review team as discussed in Section 2.2. 2.1 OVERVIEW OF THE EVALUATION PROCESS The evaluation process can be thought of conceptua11y as a link between the operational tests and technology transfer portions of the APTS Program. That is, it serves as a bridge between the conduct of an operational test at a particular site and the understanding of its actual performance at that site as well as its potential effectiveness in other locales. The quality of the evaluation process directly influences the accuracy and perceptiveness of the operational test assessment and ultimately affects the applicability and transferability of tat findings. Exhibit 3 is a flow diagram representing the evaluation process for an APTS operational test. The diagram is divided into four major sections: the evaluation frame of reference, evaluation planning, evaluation implementation, and potential evaluation spin-offs. (The specific organizational responsibilities associated untie the various aspects of each APTS test are given later in this chapter.) The first and fourth sections can be thought of, respectively, as input to and output from the active phases of the evaluation process, which are planning and implementation. A discussion of each of the four sections follows. 2.1.1 Evaluation Frame of Reference The evaluation frame of reference consists of four elements: the operational test application(s); APTS Program objectives; external influences; and local issues, objectives and site characteristics. An APTS operational test will consist of one or more technological applications introduced individually or sequentially. For example, a test might include the use of a smart card to facilitate automatic fare collection. Another example could consist of an automated vehicle location (AVL) system to determine vehicle position, followed by the installation of an automated passenger counting (APC) system and a computerized dispatching and scheduling 7 EXHIBIT 3: EVALUATION PROCESS* 8 system which work in conjunction with the AVL system. Each APTS operational test also is intended to meet the goals of the APTS Program which are: 1) to enhance the ability of public transportation to satisfy customer needs; and 2) to contribute to broader community goals by providing information on innovative applications of available IVHS technologies. These goals can be translated into the following set of objectives: Objective #1: Enhance the Quality of On-Street Service to Customers . Improve the quality, timeliness, and availability of customer information, . Increase the convenience of fare payments within and between modes, . Improve safety and security, . Reduce passenger travel times, and . Enhance opportunities for customer feedback. Objective #2: Improve System Productivity and Job Satisfaction . Reduce transit system costs, . Improve schedule adherence and incident response, . Increase the timeliness and accuracy of operating data for service planning and scheduling, . Enhance the response to vehicle and facility failures, . Provide integrated information management systems and better management practices, and . Reduce worker stress and increase job satisfaction. Objective #3: Enhance the Contribution of Public Transportation Systems to Overall Community Goals . Facilitate the ability to provide discounted fares to special user groups (e.g., disabled persons or employees eligible for tax-free employer subsidies), . Improve communication with users having disabilities (e.g., visual or hearing impairments), . Enhance the mobility of users with ambulatory disabilities, . Increase the extent, scope, and effectiveness of Transportation Demand Management programs, 9 . Increase the utilization of high occupancy vehicles, with an emphasis on reducing the use of single occupant vehicle, and . Assist in achieving regional air quality goals and mandates established in the Clean Air Act Amendments of the Intermodal Surface Transportation Efficiency Act (ISTEA). Objective #4: Expand the Knowledge Base of Professionals Concerned with APTS Innovations . Conduct thorough evaluations of operational tests, . Develop an effective information dissemination process, . Showcase successful APTS innovations in model operational tests, and . Assist system design and integration. Objective #1 relates primarily to the riders and their desire for improved transit service. Objective #2, on the other hand, deals in part with management aspects regarding system costs, service planning, scheduling, and operations. Objective #3 concerns broader impacts in terms of the degree to which an APTS application contributes to local community goals and national issue pertaining to, for example, the special needs of disabled persons, congestion management activities, user-side subsidy initiatives, energy, air quality, and accessibility. In section 3.2.1, measures are presented to examine the level to which these first three objectives are attained in each operational test. The fourth objective is directed at expanding the knowledge base of policy-makers, engineers, planners, researchers, and other individuals interested in the application of advanced technology to improve public transit. Because this objective is a broader, overarching aim of the entire evaluation program, its level of achievement will not be assessed using measures such as those discussed in section 3.2.1. Instead, an effort will be made to cull information from interim and final evaluation reports prepared as part of each operational test, and this information will be disseminated in publications such as FTA's APTS Briefs, IVHS America's Newsletter, and technical journals and conference proceedings of other organizations. In addition, selected evaluation results will be summarized on electronic bulletin boards commonly available to transportation professionals, and results will be presented at national and 10 international meetings. Finally, where appropriate, the findings and conclusions of the evaluations will be used as a basis for discussion in focus groups, meetings, and seminars. It should also be emphasized that for any given operational test, there may be objectives, over and above the APTS Program objectives, which are important evaluation considerations. These might be state or local objectives which other participants (e.g. transit operator, state transportation agency, community group, or local government) are striving to attain (e.g., to encourage ridesharing into the downtown area for the purposes of reducing parking requirements or traffic congestion in the central business district, to preserve the stability, cohesion, and authenticity of neighborhoods). The extent to which these state and local objectives relate to the APTS program objectives should be identified by the contractor. The operational test site can consist of anything from a corridor in a city to a group of cities or towns, and can be at any point along the population and density spectrum. An understanding of the unique demographic, economic, geographic, and transportation characteristics of the site, as well as prevailing attitudes toward transportation, is a useful and necessary adjunct to knowledge about the APTS application and associated objectives. To the maximum extent possible, external influences on the project should also be identified and, if necessary, appropriate strategies should be designed to reduce the likelihood that such influences will have adverse effects on the operational test. For example, if the APTS application has radio frequency (RF) spectrum requirements, such requirements should be analyzed, and political negotiation with authorized communication agencies should be initiated as early as possible. Information on the planned APTS innovations, project objectives, other issues and site characteristics, and external influences will generally be available from the application submitted to FTA by the site prior to approval of the project. Depending on the timing of the evaluation contractor's initial involvement in the project, a more detailed description of the project may be available in the form of a Project Implementation Plan. Further background on the operational test (e.g., genesis of the project concept, recent history of transit/para-transit developments at the site) can be obtained through discussions with the PTA Project Manager, the Volpe Center staff, and the local sponsor. 11 2.1.2 Evaluation Planning The evaluation planning phase of the evaluation process is the period during which the contractor interacts with FTA, the Volpe Center, and various agencies at the local level to transform the evaluation frame of reference into a detailed, structured program for conducting the evaluation. This phase sets the stage for the entire evaluation effort and, in addition, provides an opportunity to reassess and, if necessary, restructure the planned operational test. The planning phase begins with the preparation of an Evaluation Strategy for the particular project, which describes: (1) Pertinent information on the APTS application and site (in particular, an indication of what features of the operational test are unique and merit emphasis in the evaluation). (2) APTS Program objectives addressed by the operational test. (3) Relevant local, state and/or national objectives and issues addressed (and the relative emphasis to be placed on these objectives vs. APTS objectives). (4) Key issues to be resolved. (5) External influences to be addressed. (6) Recommended scope and focus of the evaluation including a discussion of the APTS costs and functional characteristics and a review of the potential efficiency, effectiveness and other impacts anticipated. The Evaluation Strategy may be prepared by the Volpe Center or the contractor. The contents of each Evaluation Strategy will vary from test to test depending on the nature and timing of the project. The Evaluation Strategy becomes the basis for the more detailed Evaluation Plan1 which is developed by the contractor. While the Volpe Center will provide a general evaluation strategy including suggestions regarding measures to be used, data to be collected, and analytical techniques to be employed, it is generally the contractor's responsibility to refine and elaborate on the Volpe Center's suggested strategy by developing specific procedures for collecting and analyzing data relative to project objectives, issues, and the site. ____________________ [1] Chapter 3 presents guidelines relative to the evaluation planning phase. The recommended content and organization of the Evaluation Plan presented in Chapter 5. 12 In developing the Evaluation Plan, the contractor is encouraged to propose changes to the approach recommended by the Volpe Center, particularly if the proposed modifications have significant potential to improve the objectivity, accuracy, completeness, and/or efficiency of the project evaluation effort or to enhance the transferability of project findings. In addition, total evaluation costs relative to potential findings must be borne in mind at all times. Throughout the process of developing the Evaluation Plan, the contractor is urged to keep in close contact with the local sponsor or project team responsible for implementing and operating the test and performing data collection. This continuing liaison with the local sponsor will ensure that the proposed methods of data collection are consistent with the resources available at the local level, with the operational implementation plan developed by the site, with important local objectives, and with reasonable costs for the evaluation contractor efforts. As is apparent from the preceding discussion, the evaluation planning phase entails substantial and continued interaction among all parties involved in the operational test. Ideally, planning of the evaluation effort should be coordinated, and take place concurrently with the planning of the project itself. This coordination between the implementation/operation and evaluation planning cycles permits optimum flexibility in the conduct of the overall test. Where possible, operational aspects of the test will be planned to conform to requirements of the evaluation, rather than the evaluation having to be integrated into a pre-existing, rigid operational structure. The concurrence of the two planning cycles ensures that the Evaluation Plan is completed prior to the implementation of the project. Early development of the Plan, in turn, allows the necessary lead time for "before" data collection -- that is, observations of phenomena such as transit system performance prior to the introduction of the APTS application(s) as well as possible information on community awareness and attitudes prior to project implementation. Throughout this phase of the project, it is critical to recognize that the FTA Project Manager is the final authority in negotiating any operational test modifications with the local sponsor. 2.1.3 Evaluation Implementation The evaluation implementation phase is the period during which the approved Evaluation Plan is executed. Activities during this phase include collection/analysis of data relative to 13 project objectives and issues, collection/analysis of data on site characteristics, compilation of a chronology describing the implementation and operation of the test, and recording of external factors which might influence operational test findings and results. Contractor functions during this phase include monitoring and in selected instances, supervising the data collection process (generally to be performed by the local sponsor), any data collection not performed by the local sponsor, data reduction and analysis, subjective analysis of information relative to project issues, and synthesis of project findings into one or more Interim Evaluation Reports and a Final Summary Evaluation Report.2 This phase not only generates information on which the final assessment of the operational test is based but also provides feedback information relative to ongoing transit operations. The ongoing evaluation activities, while adding to the cumulative body of quantitative and qualitative information regarding the project impacts, provide interim indications of costs and functions of APTS applications and the preliminary effects of these applications on transit system efficiency and effectiveness. These interim findings serve as useful input to the local agency responsible for implementing and operating the test by suggesting the need for operational modifications. During this phase, modifications may be made to the evaluation procedures originally specified in the Evaluation Plan. For instance, examination of interim findings may reveal certain gaps or redundancies in the originally planned data collection program. Still other reasons for modifying the evaluation procedure might be changes in the operational test, unanticipated developments or institutional factors at the site, or discovery of an improved evaluation procedure. Procedural steps to accomplish this necessary update for the Evaluation Plan appear in Chapter 5. The culmination of the evaluation implementation phase is the Final Summary Evaluation Report, which presents the following types of findings: _________________ [2] Chapter 4 presents guidelines relative to the evaluation implementation phase. Chapter 5 gives the recommended content and organization of the various contractor reports prepared during this phase, including the Monthly Evaluation Progress Report, the Annual Project Status Summary, the Interim Evaluation Report, and Final Summary Evaluation Report. In addition, Chapter 5 describes the content of local sponsor’s quarterly Project Progress Report to FTA, which can serve as useful input to the contractor’s work. 14 (1) Evaluation of the project in terms of its attainment of relevant APTS Program objectives and other (local and/or national) project objectives. (2) Insight into project issues associated with operational feasibility and characteristics of the applications. (3) Assessment of the influence of site-specific characteristics and external factors on the outcome of the operational test. (4) Lessons learned, based on practical experience, relative to the implementation and operation of the APTS applications (possibly to include recommendations for project modifications in the test site or for future applications in other locales). (5) Appraisal of the evaluation procedures employed in terms of effectiveness, cost, accuracy, etc. In essence, this report presents an assessment of the impact of the APTS applications at the site and provides guidance for the transferability of results to other locales. The body of the Final Summary Evaluation Report includes both narrative and graphic exposition, while detailed quantitative data and documentation of procedures are provided in technical appendices. Since the report is intended for a variety of audiences -- including transportation planners; transit operators; federal, state, and local officials; and private industry - - it contains an Executive Summary which highlights the salient project findings. 2.1.4 Potential Evaluation Spin-Offs It is anticipated that each operational test will give rise to potential implementation and analytical spin-offs. The Final Summary Evaluation Report, while essentially documenting the history and effects of a single project, also serves the broader function of increasing the understanding of and stimulating the application of the demonstrated APTS technologies in other localities. Information presented in the report provides a versatile basis for comparing the effects of a particular APTS application with those of other similar projects, suggesting modifications to the applications for future use, and predicting the effectiveness and utility of the APTS applications in other cities. Moreover, the report's assessment of project evaluation procedures can serve as a stimulus for improving the state-of-art of evaluation techniques. Since these broader functions of the Final Summary Evaluation Report generally materialize after 15 the test period and are not within the purview of the evaluation contractor assigned to a particular project, they are shown in Exhibit 3 as potential evaluation spin-offs. 2.2 COORDINATION OF APTS EVALUATIONS Exhibit 4 summarizes the various activities involved in planning, implementing, and evaluating an APTS operational test and indicates the allocation of responsibility for these activities. The sequence of activities ranges from overall APTS Program definition, to the operation and evaluation of an individual test, to the spin-off uses of the project. It can be seen that the entire stream of activities, especially those comprising the evaluation process, involves extensive interaction among FTA, the local sponsor, the Volpe Center, the evaluation contractor, and the APTS vendors. Moreover, it should be noted that the activities shown do not always occur in a fixed sequence. Time constraints may require that some of the steps be performed in parallel, and there will ideally be considerable interaction and feedback between the project planning and evaluation planning phases. The review functions of the Volpe Center, the local sponsor, and the APTS vendors associated with the data analysis provide a mechanism to identify, on a continuing basis, major problems (if any) so that APTS operational changes can be made (if necessary) during the course of the test. Evaluation spin-offs, while arising out of individual tests, will result in activities which extend beyond the FTA, Volpe Center, local sponsor, and evaluation contractor. The diversity of activities and generally long (three to four years) time frame for an individual test necessitate close and continual coordination among the groups involved. To facilitate communication among local test participants and the contractor concerning the evaluations, FTA will encourage the establishment of a local evaluation review team consisting of representatives of transit providers, metropolitan planning agencies, human service organizations, environmental groups, APTS vendors, and the general public. It may also be appropriate to include faculty from local colleges and universities on the evaluation review team. The contractor will meet with- the local evaluation review team to discuss the project objectives and the emphasis to be placed on each objective in the evaluation; to determine the roles and responsibilities of all parties involved in the anticipated data collection activities; to review problems encountered (if any) during the conduct of major data collection activities and overall 16 EXHIBIT 4: APTS OPERATIONAL TEST PLANNING, IMPLEMENTATION, AND EVALUATION: SEQUENCE OF AND RESPONSIBILITY FOR ACTIVITIES Click HERE for graphic. KEY: P = Primary role M = Monitoring role R = Review function a Includes local evaluation review team. b Local evaluation review team will be established as part of negotiations. c Primary role may also be assigned to the contractor. It may be necessary to have the contractor on-site to monitor the conduct of some data collection efforts such as an on-board survey to ensure that such efforts are carried out properly and that appropriate personnel are available to address unanticipated problems and questions. d FTA will disseminate information from these reports, where appropriate. Such information will appear in FTA’s APTS Briefs, IVHS America’s Newsletter, professional conference papers, and electronic bulletin boards. The final evaluation reports themselves will also be published. 17 operational test implementation; to present preliminary findings and results of the data analyses; and to seek the team's input. However, equally as important as coordination within a particular project is coordination across test sites, so as to maximize the effectiveness of the APTS Program in encouraging the application of new innovations. This coordination across sites is essentially important with respect to the evaluation process. Given the multiplicity of sites, operational tests, and participating organizations within the APTS Program, there is a strong need for coordination of the evaluation process so as to achieve consistency in the planning, implementation, and output of individual project evaluations. With respect to the conduct of the evaluations, such coordination will ensure that: (1) the scope of each evaluation effort is consistent with the importance of that particular APTS test relative to other APTS tests; (2) the technical approaches used to evaluate tests are consistent with the current state-of-the-art of evaluation techniques; (3) common data and definitions are employed; and (4) statistical reliability is maintained. With respect to evaluation output, such coordination will ensure that the Final Summary Evaluation Reports associated with individual projects are consistent in terms of content, format, perspective, and level of detail. This consistency in output will, in addition, enhance the spin-off potential of the evaluations. The achievement of a basic data set of uniform quality across operational tests will make possible inter-project comparisons in terms of rider characteristics, site characteristics, user acceptance, and system efficiency and effectiveness and associated criteria. These types of comparisons will be especially significant in the case of multiple applications of a particular APTS technology in several locations, or in the case of operational tests involving alternative APTS technologies directed towards a particular APTS Program objective. The coordination of the individual evaluation efforts will be achieved through the Volpe Center's active and continual participation in the program, with functions ranging from initial planning of each project evaluation effort, to monitoring of the contractor team, and finally to the synthesis of individual operational tests, evaluation reports and results. This document constitutes the first stage of the Volpe Center's evaluation coordination function, in that it describes general procedures to be followed by each contractor in performing the various evaluation tasks speeded in the contract. 18 3. GUIDELINES FOR PLANNING EVALUATION ACTIVITIES This chapter presents guidelines for planning the evaluation activities associated with a particular APTS operational test. As was mentioned in Chapter 2, the evaluation planning phase of the evaluation process is that period during which the contractor prepares a detailed Evaluation Plan based on the Volpe Center's Evaluation Strategy. The Evaluation Plan contains, among other things, a listing of relevant quantitative and qualitative measures related to various APTS, local, and national objectives and relevant issues, associated data collection and analysis procedures, and site specific data requirements and sources (both one-time and recurring). As such, the Evaluation Plan constitutes a structured, time-phased program for subsequently conducting the evaluation. The chapter is organized into three sections, corresponding to the basic decision-making elements shown in Exhibit 3: . determination of site data requirements and sources, . determination of measures and collection/derivation techniques required to address APTS Program objectives and other relevant objectives/issues, and . planning considerations relative to data collection and analysis. The organization of the chapter is not meant to imply a highly ordered time-sequencing of activities, since the evaluation planning phase is in fact highly iterative and dynamic. Moreover, it is important to realize that the guidelines comprise a basic set of ground rules for planning evaluations. The evaluation contractor will, in all probability, need to depart from these guidelines during the actual planning phase, so as to conform to the unique conditions surrounding a given operational test. The contractor should recognize his responsibility in working with the local sponsor and the Volpe Center to assure that an objective assessment of the project is achieved. One or more site visits during the evaluation planning phase is desirable to establish working relationships and channels of communication among the involved organizations and to uncover any constraints which may have a significant bearing on the development of the Evaluation Plan. During this planning effort, clarification must be made regarding responsibilities for performing and/or 19 EXHIBIT 5: BASIC SITE DATA REQUIREMENTS FOR APTS OPERATIONAL TESTS 1. Population 2. Square miles 3. Population density, persons per square mile 4. Number of persons in the labor force 5. Number of households, by type 6. Age, sex, education, occupation, income distributions 7. Household auto ownership 8. Number of persons with no drivers license 9. Modal split, by trip purpose or time of day if available 10. Existing (Pre-operational test) transit service characteristics . Organizational arrangements . Route miles (fixed route systems) . Tour area (non-faced route systems) . In-service vehicles per square mile of service area (non-faced route system) . In-service vehicles per hour within service area . Time of service operation throughout day . Days of service operation throughout year . Service frequency (fixed route systems) . Fare schedule 11. Description of para-ransit service characteristics . Data on taxi operations . Information on carpool promotion/matching programs 12. Map of the site showing: . The APTS project service area - note that this might be a contiguous area served throughout by the APTS transit system, or it might be two or more non-contiguous areas linked by the APTS service through a travel corridor . The existing transportation network - major highways, transit lines, commuter rail lines . Air quality attainment and non-attainment areas . Major topographical features such as rivers . The central business district . Any other important activity centers 13. Description of relevant site features such as: . Weather conditions . Seasonal population variations . Institutional/political climate . Economic conditions . Cost indices (e.g., cost of living index, prevailing transit wage rates) . Population/employment growth rate, land use development patterns . Residential mobility . Air quality conditions concerning ozone, lead, carbon monoxide, PM10, and other environmental concerns 20 overseeing various activities. The Evaluation Plan should indicate the finally agreed upon allocation of responsibility between the contractor and local evaluation review teams. 3.1 DETERMINATION OF SITE DATA REQUIREMENTS AND SOURCES The purpose of the site data is to provide an in-depth understanding of those characteristics of the site which might in some way influence the outcome of the project or the interpretation of project results. Obviously, the APTS operational test will not be implemented in a static environment, but rather it will affect the surrounding area. Thus, an examination of certain site characteristics is necessary in order to assess fully and accurately the impacts of the APTS application. An additional function of site data is to enhance the comparability and transferability of APTS project findings. Specifically, if conclusions drawn from one project are to be compared with findings of other similar projects or "transferred" to other potential sites, there must exist an objective approach for such a comparison or transfer. This requires the identification of a set of site-specific measures which permit one to classify sites in terms of meaningful similarities or to identify significant areas in which sites differ. Such measures might employ data pertaining to demographic and land use attributes, transportation facilities, and vehicle travel characteristics, both intra and inter-urban. In addition, information on the political/institutional climate of the area and prevailing attitudes toward transportation-related issues might be helpful in anticipating or understanding any problems regarding implementation and evaluation of the project. A review of past transit project evaluations indicates an inconsistency in both the amounts of and details concerning reported site-specific data. To some extent this inconsistency reflects a lack of standardized site data requirements, but more significantly it reflects deficiencies in knowledge regarding the interplay between site characteristics and test results. In an attempt to shed further light on the subject, a basic set of data requirements has been developed for use in APTS operational test projects (see Exhibit 5). Contractors are encouraged to propose additions or deletions to this list, in the context of particular projects, if it is felt that the nature and scope of the project call for a wider or narrower set of site descriptors. Contractors are also encouraged to propose permanent 21 EXHIBIT 6: TYPICAL SOURCES FOR SITE DATA DATA NEEDED TYPICAL SOURCES Demographic U.S. Bureau of the Census City of County Clerk State Department of Labor State Department of Internal Revenue City or County Planning Board Air Quality Environmental Protection Agency Land Use Characteristics City Directories Local, Regional and State Planning Agencies Tax Assessor's Records Planning Studies Motor Vehicle Travel State Highway department (or State DOT) U.S. Census (Journey-to-work) Local Traffic Department Earlier Travel Surveys State Registration Records Gasoline Tax Collection Records Public Transportation Travel Private Transit-Paratransit Companies Transit Authorities State Highway Department (or State DOT) Local Planning Agency U.S. Census (Journey-to-work) Earlier Travel Surveys Travel by Intercity Modes Federal Agencies such as: (air, rail, bus) Federal Aviation Administration Interstate Railroad Administration Federal Railroad Administration Department of Commerce State Regulatory Agencies Earlier Travel Surveys Private Carriers 22 additions, deletions, or changes to this minimum list based on their cumulative experience in conducting APTS evaluations. Aside from the site data requirements in Exhibit 5, it may be desirable in certain instances to collect a standardized set of attitudinal measures to obtain a profile of the community. Examples would be general opinions regarding the role of government, environmental issues, adequacy of transportation facilities, and desirability of travel by alternative modes. Since the value of this type of data for evaluation and transferability purposes has not yet been fully explored, community profile data will be collected only in selected operational tests (to be identified by the Volpe Center). Appendix A contains sample questionnaires which might be used to obtain such data. As experience is gained in this area, a standardized approach to developing an attitudinal profile of the test site may be formally incorporated into these guidelines. It is anticipated that the data set and descriptive information shown in Exhibit 5 will be available from secondary sources or from the local sponsor and will not involve specialized data collection activities (an exception being attitudinal profile data, which will entail surveys). Exhibit 6 indicates typical sources for various categories of site-specific data.3 Once the contractor has determined the type of site data required and the appropriate sources, two decisions remain: (1) the geographic scope of the area, and (2) the time period (s). Regarding the geographic scope, it was indicated above that a basic data set should be assembled for the APTS service area.4 In some cases, data conforming exactly to the service area boundary may be unavailable or may be obtained only by aggregation of fine-grained data (e.g., Census tract). If data is available for an area approximating the service area, the contractor may choose to use this pre-existing data base rather than deriving a special data base, provided that such a substitution will not be misleading and bias the evaluation. O n the other _______________________________ [3] Adapted from Heaton, Carla; McCall, Chester; and Waksman, Robert; "Evaluation Guidelines for Service and Methods Demonstration Projects", USDOT/UMTA-SMD; Washington, DC, 1976. [4] A definition of the APTS service area may not be available at the outset of the project, but rather will need to be developed during the evaluation implementation phase on the basis of user surveys. 23 hand, the use of fine-grained data may be appropriate if the service area is large and heterogeneous and thus should be divided into zones. The time period(s) for which data is to be assembled depends on the time period of the operational test and the rate at which conditions at the site are changing. If the project spans a fairly long period it may be desirable to gather site data for periods before, during, and after the project. In the case of a rapidly changing area or a staged project, data for even more points in time may be necessary. Moreover, if an historical perspective on the site is deemed relevant to the evaluation, it may be desirable to obtain 1980 as well as 1990 Census figures or recent trend data for key variables such as population, employment, and modal split. Since original data collection by the contractor is not anticipated, the number and exact timing of site data periods will be constrained by the collection cycles of existing sources. 3.2 DETERMINATION OF MEASURES AND COLLECTION/DERIVATION TECHNIQUES REQUIRED TO ADDRESS APTS PROGRAM OBJECTIVES AND OTHER RELEVANT PROJECT OBJECTIVES/ISSUES It was pointed out in Chapter 2 that the Evaluation Strategy will set forth a recommended set of APTS Program objectives, relevant project objectives (of local and national significance), and project issues to be examined. The contractor, in developing the Evaluation Plan, is responsible for reviewing this recommended set in the context of the local sponsor's Project implementation Plan and the various national and local perspectives, and then proposing appropriate modifications to the list of objectives and issues. Once the set of project objectives and issues has been finalized (which involves obtaining concurrence from the Volpe Center), the contractor must associate with these items a set of germane measures and identify suitable techniques to derive each measure and to collect necessary data. It is important to note that certain issues may not lend themselves to the use of quantitative measures but may rather involve qualitative analysis of pertinent information. The material presented below is intended to guide the contractor in developing appropriate measures and associated collection/derivation techniques. It is important to recognize that this material will undoubtedly be modified as information is gained through the consistent application and analysis of evaluation techniques on the operational tests. Therefore, because revisions to data program requirements in terms of basic data sets, collection and 24 analysis procedures, and presentation techniques can be expected, the fundamental value of this section of the guidelines lies in the manner in which it structures the approach to the selection of measures and the selection of techniques for collecting/deriving them. In preparing this material, considerable documentation was reviewed (see Bibliography). In addition, direct observance and participation in many previous and ongoing Federally-funded projects has permitted those preparing this document to identify not only a logical structure for project evaluation but also to highlight problem areas of which all potential project evaluators should be aware. The specific projects which contributed the greatest amount of insight were the evaluation plan development for the APTS/AVL operational tests and the Service and Methods demonstration projects. 3.2.1 Basic Set of Measures To assist the evaluation contractor and the local evaluation team in the selection of measures to assess operational test objectives, six categories of measures are suggested: . APTS costs, . APTS functional characteristics, . user acceptance, . transit system efficiency, . transit system effectiveness, and . impacts. The first three categories of measures relate directly to the costs, functional aspects, and utility of the APTS application and associated equipment. The next two categories pertain to transit system performance in terms of actual delivery and usage of the transit services provided. The final category of measures addresses project impacts related to critical transportation issues and societal goals and concerns. While many operational tests will be designed to achieve the same (or similar) objectives, some tests might be particularly unique in their ability to address certain objectives. Consequently, "priority objectives" should be identified in these unique tests, and a corresponding set of measures should be formulated so that these "priority objectives" are given 25 proper attention, emphasis and evaluation resources. Furthermore, the type of measure and the method of measurement should be considered as discussed below. . Type of measure Quantitative -- a measure which is expressed in terms of counts, measurements, dollars, or other physical units Qualitative -- a measure which is expressed in terms of people's attitudes, perceptions, or observations . Method of obtaining measure Collected -- obtained by measurement (vehicle travel time), counting number of passengers), surveying (perceived reliability), or from records (daily revenue) Derived -- calculated from collected measures either by simple arithmetic procedures (passenger miles per seat mile) or through use of analytic models (reduction in air pollution or fuel consumption) In reviewing the basic set of measures, it is important to note that some of these measures would be more meaningful if stratified by time of day (beak versus off-peak), location (corridor versus arterial), person time segments (waiting, access, transfer, in-vehicle), route type (fixed route versus demand responsive), and vehicle tour segments (in-service, non-service). Because such a classification of measures would have needlessly extended the list, the subject of stratification, or categorization, with respect to specific data collection plans is discussed separately in Section 3.3.2 of this chapter. The above categories of measures are not to be construed as a minimum requirement for every APTS project, since an evaluation need only encompass measures corresponding to the APTS Program objectives and other project objectives/issues addressed by the particular 26 operational test. Rather, the categories of measures should be used by the contractor as a checklist from which the most germane measures can be selected and to which other relevant measures can be added as appropriate. It will be noted that for each of the APTS Program objectives, it is possible to measure attainment of some objectives from two vantage points: the actual and the perceived attributes of the transit system (as represented by quantitative and qualitative measures, respectively). In the case of transit travel time, it might be appropriate to measure actual changes in travel time and then to compare with perceived travel time. Similarly, in the case of APTS equipment reliability and user acceptance measures, comparisons with user perceptions and attitudes might also be appropriate. Until more is learned about the interrelationship between actual measurements and attitudinal data, it is not possible to set forth hard and fast rules for when to supplement quantitative measures with qualitative measures. Clearly, it may be prohibitively expensive to employ this two-pronged procedure for each area of interest; on the other hand, mere reliance on quantitative measures may result in overlooking what is in fact the major behavioral determinant -- people's perceptions of the system. For the time being, the contractor must exercise sound judgment in deciding which situations are unique and instructive enough to warrant a two-pronged data collection effort. In no case should an attitudinal measure ever be used in place of a quantitative measure, where both are available. The rationale underlying each category of measures and their association with operational test objectives is discussed in Sections 3.2.1.1 to 3.2.1.6. Further discussion of data collection/derivation techniques appears in Section 3.2.2. 3.2.1.1 APTS Costs and Functional Characteristics Central to an operational test evaluation is the performance of the APTS system and its individual components. Questions surrounding the costs and functional characteristics (including reliability, usefulness, maintainability, adherence to specifications) should be addressed, and the relationship between these APTS characteristics and overall operational test objectives should be examined. Examples of such questions are: 27 . What are the life cycle costs (including fixed and recurring expenses) of the APTS system and its individual components? Which are "start-up" costs associated with the newness of the system and might be avoided in future applications? . Is the automated vehicle location system easy to use and are vehicle positions determined quickly and accurately so that on-time scheduling can be carried out and that passengers are provided with timely information? . Is the smart card system reliable, and does the system meet the required design specifications? To the extent possible, the objective (or objectives) related to a particular APTS component should be clearly articulated and the specific component costs and associated functionality should be determined. This will facilitate the comparison of APTS costs and associated benefits. It is recognized, however, that individual component costs may be difficult to determine if the procurement process allows lump sum bids. 3.2.1.2 User Acceptance The extent to which various APTS applications are actually utilized will be an extremely important dimension of performance in each operational test. The percentages and numbers of riders using a smart card for fare payments are just examples of quantitative measurements in this category. In addition, qualitative measures of user acceptance (or utility) would be employed, examples of which include the attitudes of riders regarding the usefulness of AVL-based pre-trip information and the perceptions of dispatchers concerning the benefits of component monitoring equipment. 3.2.1.3 System Efficiency and Effectiveness Transit system performance is typically viewed in terms of efficiency and effectiveness, both of which may be influenced by the use of the APTS application and other technology. Efficiency is related to the extent to which system inputs such as vehicles, personnel, fuel, and funds are employed to produce outputs; examples of outputs include the actual number of vehicle miles or vehicle hours of service. For example, reductions in unit operating costs would be 28 examined in part with the use of efficiency measures such as the operating cost per vehicle mile or operating cost per vehicle hour. Effectiveness concerns the users and actual demand for service and relates to financial aspects such as revenue and cost effectiveness service utilization, quality, convenience, safety, security, and reliability. In addition, non-financial aspects of effectiveness include service utilization, safety, security, and service reliability. 3.2.1.4 Impacts To examine the extent to which the operational test responds to critical transportation issues and national mandates such as the Americans with Disabilities Act, the Clean Air Act, and other Federal legislative efforts, both quantitative and qualitative impact measures are required. Such impacts may be anticipated or unanticipated and positive or negative. These impacts relate to, for example, the transit agency and its internal activities and administrative procedures; aspects of human factors; privacy; and matters dealing with equity, social, energy, traffic congestion, air quality, special mobility needs, institutional and political concerns. For example, the use of a smart card might facilitate the implementation of a more equitable and efficient fare policy as may have been anticipated, but it unexpectedly required a reorganization of the transit system's finance department and the existing fare collection and accounting activities and procedures. Another example concerns the use of an automated vehicle location (AVL) system which, as intended, may improve on-time scheduling; however, such scheduling improvements will only be realized after the transit dispatching staff has been properly trained and has learned to use the AVL system for the purpose of communicating with the bus operators. 3.2.1.5 Relationship Between APTS Program Objectives and the Categories of Measures While the six categories of measures discussed above are not meant to be exhaustive, they do provide structure and guidance in the selection of measures to evaluate the APTS program objectives #1$ #2$ and #3 to the extent that they are associated with each operational test. 29 The first APTS program objective, as stated in Section 2.1.1, focuses on enhancing the quality of on-street service to riders in terms of safety, security, convenience, ease of travel, and travel time. These concerns fall largely under the categories of APTS functional characteristics, transit service efficiency and effectiveness, user acceptance, and impacts as discussed above. Examples of corresponding measures appear in Exhibit 7. The second APTS program objective is to improve system productivity and job satisfaction. Anticipated system productivity improvements might result from reductions in system costs; better schedule adherence; quick and effective responses to incidents and vehicle and facility failures; and information management systems to provide reliable and accurate operating data in a timely manner. Job satisfaction pertains directly to another group of potential APTS beneficiaries; that is, the employees, such as drivers, dispatchers, and data analysts. An APTS application may lead to a change in the day-to-day activities of such employees and may, in turn, lead to reductions in worker stress and increases in job satisfaction. Examples of measures to evaluate the association of each test with this objective are given in Exhibit 7. The third APTS program objective centers around the contribution of public transportation to larger societal issues and community goals. These issues and goals relate to such elements as special mobility needs, traffic congestion, air quality, energy, privacy, equity, and other concerns. Appropriate measures to assess this APTS objective are mainly included in Exhibit 7 under the categories of user acceptance, effectiveness, and impacts. As discussed in Section 2.1.1., the fourth APTS program objective is a somewhat broader objective than the other three and consequently, the above measures will not be used to measure its level of achievement in each test. However, as mentioned in Section 2.1.1., to expand the knowledge base, results of tests will be disseminated in journals, conference proceedings, electronic bulletin boards, technical meetings, and seminars. Each category of measures includes criteria associated with various aspects of APTS applications ranging from their costs and functional characteristics to their association with overall transit system efficiency and effectiveness and other broader societal issues, such as air quality, energy, and special mobility needs. The results of each evaluation will be widely disseminated as discussed in Chapter 2, so that professionals have access to the knowledge they need regarding the actual performance of APTS technologies and the use of the analytical 30 Click HERE for graphic. 31 Click HERE for graphic. 32 Click HERE for graphic. 33 techniques employed in the analyses. The availability of such knowledge will lead to the design of improved APTS applications, in the conduct of more thorough evaluations, and the utilization of enhanced evaluation analysis tools. 3.2.1.6 Other Objectives and Measures The six categories of measures in Exhibit 7 are also useful in the selection of measures for other operational test objectives. As pointed out in Section 2.1.2, there will likely be state or local objectives in addition to the APTS program objectives. For example, a state objective might be to reduce the amount of financial operating assistance needed. This would imply that either operating costs must decrease or operating revenues (e.g., fares) must increase. Measures associated with this objective relate to system efficiency and effectiveness. Another example might be a desire to revitalize the central business district. Measures for this objective would fall under the area of economic concerns in the impacts category. 3.2.2 Data Collection/Derivation Techniques Once the relevant measures for project evaluation have been determined, it is necessary to identify appropriate collection or derivation techniques. Collected measures can be obtained through the following four basic methods: (1) By measurements, using various instruments, such as stopwatches, odometers, speedometers, and lap-top computers. The accuracy of the recorded data is a function of the accuracy of the measuring instrument itself. Typical measurements include travel times and vehicle velocities. (2) By counts or observations involving tallies either from discrete digitized recording equipment, lap-top computers, or manual counts. Typical counts would be numbers of passengers in vehicles. (3) By surveys or interviews which provide information relative to the individual being questioned, said information to include such items as origin, destination, income level, previous travel modes, observations of how the service is functioning, and attitudes towards transit amenities. (4) By searching records such as those available through the transit system, local sponsor, and other local planning agencies and Census records. 34 Derived measures can be calculated either through the use of simple arithmetic processes or special analytic models. This form of measures builds upon basic data collected through some of the above means. An illustration of a simple derived measure might be dividing passengers per day by vehicle miles per day to obtain passengers per vehicle mile. Examples of the latter type of derived measures resulting from analytic models might be the use of a time-delay curve to estimate vehicle speeds or the calculation of reductions in fuel consumption and air pollution based on a model using changes in traffic volumes as input. In view of the large number and variety of measures in Exhibit 7 and the even larger number which are likely to arise during the course of the APTS Program, it would be very difficult to specify in these guidelines a preferred method of data collection for each measure. Moreover, it would be inappropriate to attempt to choose a set of "best" methods from among the techniques already tried; rather, it is desirable to encourage the continual development and implementation of novel techniques with potential for increasing the efficiency or accuracy of evaluations. Finally, there is really no requirement for uniformity among data collection techniques, but rather there is a need for consistency and comparability of the data obtained by these collection techniques. The techniques can differ from project to project, as long as they are comparable in terms of accuracy and yield data in a form suitable for analysis both within the project and among projects. For the above reasons, it is not the intent here to prescribe a standardized approach to data collection. However, it is appropriate to discuss the potential applicability of some of the specific techniques, drawing where possible from previous experience. Exhibit 8 illustrates the range of techniques employed for selected measures in past transportation projects.5 Specific comments on these techniques and general recommendations applicable to collecting the measures follow: (1) Travel time, speed, and vehicle volume data collection techniques can range from manual to automatic. In general, automatic techniques are effective only where the magnitude of data requirements or some -other special circumstances warrant their use. Some of the more sophisticated automatic procedures are subject to reliability problems. Failure of ____________________ [5] For further details on collecting transit date, see "Review of Data Collection Techniques," prepared by Booz-Allen & Hamilton, Inc. for FTA, March 1985 35 EXHIBIT 8. EXAMPLES OF DATA COLLECTION TECHNIQUES FOR SELECTED MEASURES Travel times for transit vehicles: . On-board checkers or on-street checker with stop watches or lap-top computers . Time referenced equipment connected to bus Speeds for transit vehicles and autos: . On-street checkers with radar units or other equipment . Test vehicle with use of odometer, clock, and other equipment . Real-time surveillance system with image processing capabilities Counting auto occupants: . On-street counts recorded on paper, counters, or lap-top computers Counting transit vehicle passengers: . On-board checkers or on-street counts recorded on paper, counters, or lap-top computers . Bus drivers recording passenger load . Automatic Passenger Counters Travel times for autos: . On-street checkers at selected locations recording license plates and times; calculation of elapsed time by matching plates; possibly in conjunction with video camera and image processing technology . Time lapse aerial photographs or video . Floating car with observers to record travel time and stopped time delay using stop watches or other equipment Counting of transit vehicles and autos: . Permanent or temporary tube counters or loop detector in lanes or zones of interest . Visual counts recorded by persons . Time lapse aerial photographs or video . Real-time surveillance system with image processing capabilities . Electronic detectors Demographic/behavioral/attitudinal data on users/non-users/ operators: . Post cards distributed to auto drivers at exit ramps, to boarding and on-board passengers, and at park-n-ride facilities . Forms, usually no longer than one page, distributed and returned by mail or collected on buses . Sampling of autos by noting license plates and subsequent identification through Department of Motor Vehicles files; possibly with video camera and image processing technology . Interview conducted either at home, work, or within the transit system itself (on board, at stations, etc.) or with transit or local officials 36 these devices can cause loss of vital data, which will in turn delay the evaluation, and considerably increase costs. In addition, the measurement accuracy of automatic or semiautomatic devices may be questionable, particularly if they have not been used extensively before. In cases where definitive information on devise accuracy is not available, it is essential to confirm the accuracy of automatically collected data by periodic use of manual devices. Simple manual devices can be deployed so as to maximize utilization of roadside personnel. For example, in one project, the use of special counters by each observer permitted keeping track of the auto occupancy of each vehicle counted, with the result that two measures were obtained at once. In other projects, special manual devices were used to obtain vehicle counts and occupancy data simultaneously. (2) Past experience has shown that there is a lack of consistency between passenger counts recorded by transit personnel and counts by on-board or roadside observers. For instance, in one project, it was found that bus drivers tend to overestimate the passenger load and that on-board and on-street counters tend, on the average, to be consistent with the other. If transit personnel are to record such data, it is essential that verifications be made during the project to detect any potential bias or unusual variability in this data. (3) In utilizing transit system records and service area records, such as census data, it is critical to ascertain accuracy of these data. Usually, discussions with personnel who initially record these data will provide an assessment of accuracy. Further, where special data are collected for the project by a local organization, monitoring procedures will be established to assure that no modifications in procedures or notations have occurred which might have an impact on the evaluation process. (4) Demographic, behavioral, and attitudinal data on users and non-users of the services provided as part of the operational test, as well as attitudinal information from transit operators, can be collected through a wide variety of survey and interview techniques, with varying degrees of respondent cooperation, accuracy, and cost. In view of the large amount of documented survey experience relating to both transportation and general market research contexts, and in view of the large anticipated role of surveys in APTS evaluations, Appendix A has been devoted to a discussion of survey design and execution. In evaluating the array of existing and potentially innovative collection techniques relative to a particular measure, some of which are included in Exhibit 8 as examples, the contractor should consider factors such as the cost and accuracy of each method, the availability of local resources to implement each method, the ease of implementation, and the ultimate data analysis requirements. 37 With respect to cost, the contractor should apply sound judgment in determining whether the anticipated cost of using a particular technique is justifiable in terms of the contribution to the overall project evaluation of the specific measure being collected. Clearly, the total project expenditure for data collection should be allocated among individual measures, taking into account each measure's contribution to the project evaluation. The contractor should make special note of any data item which is relevant to the evaluation but whose collection cost appears to be disproportionately high in relation to other items. The contractor should determine whether the accuracy of a particular technique is consistent with the accuracy requirement for the measure, which in turn is dependent on the relative importance of the measure. A very accurate technique is probably not warranted for a relatively insignificant measure, especially if that technique would be expensive to implement. In addition, a high degree of accuracy for some measures may be inconsistent with a lesser degree of accuracy for others. The contractor should also evaluate alternative techniques in light of the available local resources-- labor resources as well as equipment. An attempt should be made to utilize existing equipment or rental equipment arrangements wherever feasible, rather than opting for techniques which require the purchase of new equipment (which might not be needed by the locality after the APTS evaluation). The contractor's Evaluation Plan should contain justification for selecting the particular technique applicable to each measure in terms of these considerations. In the case of a novel technique, it is required that the contractor demonstrate acceptable accuracy before it can be used as the sole source for data collection. It is further required that the evaluation contractor document his experience with those data collection methods employed in an evaluation, as explained below in Chapter IV. As this further experience develops, the Volpe Center will make this information available via updates to this Guidelines document. 3.3 PLANNING CONSIDERATIONS RELATIVE TO DATA COLLECTION AND ANALYSIS The preceding section contained-guidelines relative to specifying appropriate measures and collection/derivation techniques for addressing APTS Program objectives and other project objectives and issues. This section completes the discussion of evaluation planning activities with general guidelines for data collection and analysis procedures. The material in this section, 38 while intended to be applied to individual measures selected for inclusion in the evaluation, is presented in a general context. The following topics are included: basic data collection/analysis design, measure stratification, sampling requirements, and the timing of data collection. 3.3.1 Basic Data Collection/Analysis Design A significant aspect of the evaluation process for APTS operational tests is determining the basic data collection and analysis design to be employed relative to specific project objectives. There are a great variety of potential design approaches, ranging from an "after-only" design (a one-shot case study approach involving a single set of measurements taken after the project is operational) to a "before-after with control group" design (involving a comparison of multiple measurements). A General Accounting Office (1991) Report entitled, "Design Evaluations," presents guidelines with the use of a "decision tree" to assist in the selection of an evaluation design including case studies, cross-section or panel surveys, comparative group analyses, or a before and after study. A comprehensive discussion of the specific utility and the relative pros and cons of the various design approaches can be found in Donald T. Campbell and Julian C. Stanley, Experimental and Quasi-Experimental Designs for Research, 1968, and L. Mohr, Impact Analysis for Program Evaluation, 1988. The information which follows is intended to discuss the relative advantages of various approaches in the context of the APTS program and to highlight the major considerations involved in selecting the appropriate design for each APTS evaluation, or for individual measures included in the evaluations. In general, a single set of measurements (for example, taken while the test is in operation) will be insufficient for assessing the impact of the test, since it will not provide any yardstick with which to interpret the measurements. It is recommended, therefore, that every data collection/analysis program be structured around some form of comparison. If such an approach is for some reason infeasible, the contractor must indicate the reason(s) in the Evaluation Plan. Given that the basic data collection/analysis design will generally be in the form of a comparison of multiple measurements, the next question to be considered is what types of comparison are appropriate. The two main forms of comparison are before vs. after and test 39 vs. control. In a before-after comparison, a given measure is collected on a given system element before the experimental or exemplary operational test technique is instituted and then again while the technique is operational.6 In a test-control comparison, a given measure is collected on a system element which has been affected by the introduction of a technique (test unit) and also on an equivalent system element which has not been similarly treated (control unit). Each type of comparison is somewhat limited: the before-after comparison fails to show what portion of the change in the measure is due to external factors; the test-control comparison shows the difference between "after" measures and hence accounts for external factors, but fails to indicate the degree of change from the before state to the after state. Accordingly, it is desirable, where feasible, to conduct a before-after comparison in conjunction with a test-control comparison. In other words, the data design should, if possible, involve collection/analysis observation of both a control and test unit before and after the institution of the APTS application. To make the foregoing discussion more concrete, consider a large area with many bus routes and suppose that a certain fraction of them are treated in some manner (i.e., an APTS application is implemented which can be expected to reduce bus travel time). If pre-application and post-application measures of travel time are made only on the treated routes and a reduction in time is indicated, there is no way of knowing the extent to which the improvement is attributable to external factors (for instance, a decrease in auto traffic on the streets where the buses operate). In order to account for, in a quantitative fashion, these known or unknown factors which have arisen during the interval between the before and after measurements, it is necessary to make before and after measurements of bus travel time on routes which are comparable to the test routes and therefore susceptible to the same set of external factors. The difference between the travel time reduction on the test vs. control routes can then be taken as the true change due to the application. To make these statements, it is necessary to be fairly ___________________ [6] As is discussed below, a before-after comparison does not necessarily imply a single measurement before the operational test is implemented and another measurement while it is operation. Rather, this type of comparison can take the form of a series of measurements prior to, during, and after the operational phase of the operational test. If the project is implemented is stages, there will be a series of measurements corresponding to each stage. 40 confident that conditions affecting both control and experimental units are reasonably similar a requirement which is sometimes difficult, if not impossible, to assure. To reiterate, the proper use of the combined before-after/test-control approach guarantees to the greatest extent that any observed improvement is indeed due to an operational test application. Thus, the contractor should employ both types of comparisons wherever appropriate and feasible. The determination of appropriateness of the combined approach involves a consideration of the time span of the operational test. Regarding the scope of the project, the larger the geographic area encompassed by or affected by the project, the greater the possibility that no control units can be identified (i.e., the entire population is composed of test units). Regarding the time span of the project, no generalizations can be made since tests will vary in length depending on a variety of factors. As a general rule-of-thumb, the desirability of the combined before-after/test-control approach increases with the time span of the project, since this approach reveals internal as well as external changes occurring over the project's duration. The determination of feasibility of the combined approach involves questions of data availability and project timing. If there is a known deficiency in either type of comparison, then only the valid comparison should be employed; it is generally better to do without a before observation or a control observation than to settle for unsuitable before or control data. In the event that only one type of comparison is feasible, there are alternative techniques and precautionary measures available to the contractor to compensate for the absence of the other type of comparison. If no control group exists (e.g., if the operational test affects the entire population of observation units, making each one a test-unit) or if no suitable group can be found (each test unit is unique), then the contractor should be especially observant throughout the evaluation period of possible external factors which might influence the interpretation of project results. Any statistics regarding the before vs. after change due to the applied technique should be examined very carefully in the context of these observed external factors, and any conclusions based on such statistics should be qualified accordingly. If, due to project timing, there is no opportunity to perform before measurements, or if it is known beforehand that the units to be observed will undergo considerable change between 4l the before and after periods, the contractor should attempt to obtain surrogate data for the before period. Possible sources of surrogate data would include: (1) surveys conducted after the test is operational which question people about conditions or their behavior prior to the implementation of the technique; and (2) demographic and travel data collected by the local highway department, planning agency, or transit operator some time prior to the operational test. The surrogate data can be used to provide some indication of the magnitude of the before-after change experienced by the test and control groups. In using the before-after and/or test-control approach, one of the key steps is identifying comparable units. To as great an extent as possible, the units observed for the before case must be equivalent to the units observed for the after units. Returning to the previous example of bus routes, before-after comparability is not a difficult problem, since the same routes can be observed for both time periods. The only note of caution is that the routes should be unchanged (with respect to length, number and location of stops, etc.) from one measurement period to the next. Test-control comparability, on the other hand, raises some interesting problems. Theoretically, the test and control units should be as nearly alike as possible to rule out any chance of the observed change being a result of something other than the operational test application. Test and control units should be chosen which are similar in terms of variables assumed to be related to the particular measure. Again, using the example of bus routes and the measure travel time, matching of test and control routes could be done on the basis of such descriptors as route length, total trips along the route, peak headway, and average speed. The Volpe Center's Evaluation Strategy will generally suggest the basic data collection/analysis design to be employed for each project as a whole or for particular measures (e.g., before-after comparison, test-control comparison, both types of comparison, or a single set of measurements. The contractor should determine the feasibility of such suggestions in terms of the data availability and time frame of the particular project and site. The contractor's Evaluation Plan should then elaborate on the approach finally selected for each measure, indicating information such as the specific units chosen for the control and test groups. 42 3.3.2 Measure Stratification Measure stratification refers to the categorization of individual measures for collection/derivation and/or analysis purposes. Examples of measure stratification are: (1) peak versus off-peak time periods, (2) day of the week, (3) revenue (in-service) versus non-revenue service, (4) waiting, access, transfer and in-vehicle travel times, and (5) fixed route versus demand responsive. Measure stratification improves the quality of the evaluation by allowing an assessment of how changes in measures relate to the stratification categories, hence facilitating the formulation of more specific findings and conclusions. Whereas collection of an unstratified measure provides only a single, average reference point, the use of a stratified measure provides a series of reference points, each of which may be significant to the analysis and interpretation of results. Knowledge of inter-category differences in results enhances transferability; for instance, if a particular operational test proves to be especially beneficial in congested areas but of limited value in sparsely traveled areas, then other sites considering implementation of the service will know to focus their efforts in congested areas. Stratification can take the following forms: (1) categorization of a measure into additive components (e.g., measuring person trip time in terms of trip components such as access time, line-haul time); (2) categorization of a measure, and possibly its components, according to target market, operational, geographic, or time categories (e.g., measuring trip time for peak and off-peak periods); and (3) grouping of raw values of a measure into class intervals, with class intervals determined either before or after data collection (e.g., determining the distribution of early, late, and on-time arrivals). It is not possible apriori to present a standardized approach to be used for each measure. Clearly, the appropriate type and level of stratification depend on the particular measure and on the characteristics of the site and project. However, in order to provide the contractor with 43 some guidance in this area, examples of possible types and levels of stratification are presented below. 3.3.2.1 Categorization of a Measure Into Additive Components This form of stratification involves collecting and reporting data separately for specific components, or sub-breakdowns, of a measure. The purpose of categorizing in this manner is to single out the effect of an APTS application on these specific components. Examples of this form of stratification are available for measures relating to travel time, reliability, and productivity. Person transit trip time for fixed route systems can be broken into segments as depicted in the following diagram: Origin Destination where: Segment A = Access time Segment W = Waiting time for first vehicle or for subsequent transfer Segment T = In-vehicle transit time Segment E = Egress time ti = Time for ith trip segment If further amplification is desired, access time and egress time can be subdivided into walking, riding, and other portions; or in-vehicle transit time can be subdivided into collection, line-haul, and distribution phases. In the case of demand-responsive systems, some of the trip time components might take on a different definition: for example, access time would be zero, and waiting time would refer to the difference between the caller's requested time of pick up and the arrival time of the vehicle at the origin. In cases where the caller is told that pick up can only be made later than 44 the requested time,7 wait time can be further divided into the time between the requested pick-up time and the promised pick-up time, and the time between the promised pick-up time and the arrival time of the vehicle at the origin. This latter travel time component, is, in itself, a basic transit system reliability measure in the category of effectiveness measures summarized in Exhibit 7. In-vehicle transit time, if desired, can be divided into the direct routing travel time (the time between the person's origin and destination if no other pick-ups or drop-offs are made) and the detour travel time (the time spent detouring to make other pick-ups and drop-offs). Transit vehicle time is always to be broken into in-service time and non-service time. However, if desired, these two prime categories can be further divided as indicated below. For fixed route systems: In-service In motion Loading Non-productive -- waiting for lights, metering, or other obstacles to motion Non-service Garage to first service point Last service point to garage Dead turnaround time Deadhead time Other For demand responsive systems: In-service In motion with one or more passengers onboard In motion with no passengers onboard and in the act of picking up one or more passengers Loading Non-service Garage to first pick-up point Last drop-off point to garage Between first pick-up point and last drop-off point with no passengers onboard and not in the act of picking up one or more passengers ________________________ [7] Due to the potential ambiguity associated with requests for immediate service, the contractor should note how the particular transit operator maintains data on requested and promised pick-up times. 45 These time segments are depicted in the following diagram: where: Point A = Garage B = First pick-up point C = Drop off point -- no passengers on vehicle but driver is instructed to proceed immediately to pick up a passenger D = Pick-up point E = Drop-off point -- no passengers on vehicle and there are no requests for immediate pick-up; driver is instructed to proceed to central waiting point F = Point enroute to central waiting point-- driver is instructed to proceed immediately to pick up a passenger G = Pick-up point H = Last drop-off point of day I = Garage Note that in segments BC and GH pick-ups and drop-offs are being made and at least one passenger is always onboard. Also, all pick-up and drop-off points include time spent waiting for riders to board and deboard vehicles. For operating costs of APTS operational tests, it has been decided that the aggregation of cost items should be consistent with FTA Section 15 expense categories. Exhibit 9 is a matrix showing the distribution of expense object classes into functional areas under Section 15. 46 Click HERE for graphic. 47 Because of possible differences in current internal accounting practices, it is essential that any techniques for disaggregation and allocation of costs be described in the Evaluation Plan. In addition, because of different funding mechanisms, it is important to review in depth individual transit authority practices. It is also recognized that the reporting of operating costs should be carried out using a consistent time frame for reporting periods. 3.3.2.2 Categorization of a Measure According to Target Market, Operational, Geographic, or Time Categories The primary purpose of this form of stratification is to evaluate the effect of APTS applications in different contexts. As in the case of categorization into additive components, this form of stratification involves collecting and reporting measures separately for each category. Examples are as follows: Target Market: Trip purpose -- work/non-work User group -- commuters/non-commuters Mode -- auto/transit/other Operational: Type of transit service -- express/local; fixed route/demand responsive Direction of traffic flow - inbound/outbound Type of thoroughfare -- freeway/arterial Geographic: Within/outside central business district Zones with different demographic characteristics Time: peak/off-peak weekday/week-end Finer stratification in the above examples is also possible. For instance, within the target market category, the trip purpose "non-work" can be divided into medical, social, recreational, etc.; non-commuter can be stratified into elderly, disabled (ambulatory and non-ambulatory) unemployed, etc.; and mode can be divided into solo driver auto, carpool auto, chauffeured auto, and specific local transit service options. Types of bus service can be divided into local feeder, local line-haul, and express line-haul, and further divided into individual routes, and 48 beyond that into route segments. Time of day can be refined into the four Section 15 categories (A.M. peak, midday, P.M. peak, night) or even further into hour, half-hour, or 15-minute segments within certain categories. In general, in some instances it will be desirable to partition collected data into various target market categories, since most operational tests will probably consist of specific innovations aimed at particular user groups. The decision as to whether to stratify collected data by operational and geographic categories depends on the nature of the project and thus will have to be made on a case-by-case basis. However, it is recommended that serious consideration be given to using a minimum time of day stratification (peak/off-peak) for every measure, since many transit system operating characteristics as well as general traffic conditions vary widely between peak and off-peak periods. The decision as to stratification of data collection within the peak period (i.e., morning vs. evening peak) and within the off-peak period (i.e., midday vs. nighttime) should be made in accordance with the time of APTS service operation throughout the day and the variability of travel conditions and other relevant factors between the different categories. It is important to note that the peak period may be a changing period depending upon distance from the CBD and type of transit system. Other issues regarding data stratification and analysis are discussed in Section 4.2. 3.3.3 Grouping of Raw Data Into Class Intervals Measure stratification can also refer to the grouping of raw data into intervals, with intervals determined before or after data collection. Whereas the first two forms of stratification involve collecting and reporting a measure separately for each category (e.g., change in travel time during peak periods, off-peak periods), this type of grouping produces a frequency distribution for the particular measure. Survey data on traveler behavior, characteristics and attitudes is a good example of pre-collection determination of intervals. For instance, comparisons of users and non-users of an APTS test can be made using distributions of-such measures as age, income, auto availability, and attitudes toward transit, with the particular response categories of each measure having been determined beforehand. Appendix A contains recommended response categories for selected 49 demographic and travel behavior measures, as well as sample questions and response categories for selected attitudinal data. Reliability measures provide examples of intervals that can be determined after data collection. The difference between scheduled and actual arrival time at an access point would be collected in its raw form (i.e., each vehicle's time difference in minutes), but would be reported as a frequency distribution. A suggested minimum stratification of this measure is: % early % on time (vehicles arriving within +x or -y minutes of scheduled time) % late The contractor should be aware of differences in transit company standards with respect to schedule adherence, and the potential impact on data collection and analysis procedures. Vehicle delays due to breakdowns can be grouped according to the following minimum stratification: % No delay (delay of 2 minutes or less) % Delayed % 96 Total disruption of service If further detail is desired, the late category under schedule adherence and the delayed category under vehicle reliability can be divided into categories such as: 1-5 minutes delay, 6-10 minutes delay, over 10 minutes delay. The basic intent of grouping is to summarize the raw data without masking the real form of the distribution for a given measure. In addition, the extent of grouping may also depend upon the specific analyses which are planned. Interval grouping can be used in conjunction with either of the two forms of stratification previously discussed. For instance, person trip time can be stratified into components (access time, etc.), and time period (peak vs. off-peak) can be grouped into 5 or 10 minute intervals to obtain a frequency distribution. As was stated above, it is not possible in these guidelines to present a standardized approach to stratification for each measure. The contractor will therefore have to rely on judgment and past experience to determine which types of variable stratification are most likely to enhance understanding of specific areas of project effectiveness and potential application. 50 The contractor should plan data collection activities with the finest stratification which can be justified as appropriate for the APTS objectives. Since the ultimate sample size will be directly related to the number of categories employed, the contractor should make sure that the available sample units are sufficient to support the level of stratification deemed desirable. The Evaluation Plan developed by the contractor should contain justification for the type(s) and level of stratification selected, as well as evidence that such stratifications are feasible from the standpoint of data and sample size availability. 3.3.4 Sampling Requirements Once the contractor has determined the basic data collection/analysis design for the project evaluation and the type(s) and level of stratification for each measure, the final question to be addressed is sampling requirements. In general, data required from records maintained by the transit operator or other organizations should be available on a continual basis over the entire lifetime of the experimental test and such data should not require sampling. On the other hand, data obtained from measurements, counts, and surveys will generally not be available on a continual basis but will have to be collected in the form of samples. There may also be situations where measurements or counts yield continual data, but sampling is desired in order to reduce data processing expenses. When collection of a particular measure involves sampling, an estimate of the minimum sample size must be made prior to the initiation of the data collection effort. In estimating sample size requirements, the objective is to have a large enough sample to be able to draw valid inferences about the population from which the sample is drawn. As might be expected, the determination of appropriate sample sizes involves trade-offs between the desired level of precision and the cost of data collection. These trade-off decisions in turn require a determination, during the evaluation planning phase, of the appropriate types of analyses to be performed (e.g.,-estimates of population parameters, comparisons-between two or more groups of sampled data). Appendix B presents specific guidelines relevant to estimating required sample sizes. Included in the discussion are: (1) references to statistics books containing sample size 51 equations, (2) recommendations regarding values for the three input factors in the sample size equation, and (3) suggestions regarding implementation of the field data collection effort based on the calculated sample size values. Appendix B also contains a section on the basic types of possible statistical analyses, appropriate confidence levels, and desirable reporting formats. The contractor should follow the guidelines in Appendix B to develop appropriate sample sizes for each measure. The Evaluation Plan should contain the sample size values, along with an explanation of any assumptions or special procedures underlying these values (e.g., equations, input factor values used). 3.3.5 Timing of Data Collection For measures based on sampling, another issue to be addressed by the contractor is the timing of data collection. The exact periods during which measures are collected have a significant effect on the validity and representativeness of evaluation results, since the operation and effectiveness of a transportation system are sensitive to various factors associated with time. Four basic questions arise concerning the timing of data collection: (1) The appropriate season(s) of the year and day(s) of the week to include in the sample, (2) The appropriate duration of each data collection period, (3) The proper time to initiate data collection, and (4) The appropriateness of "one- shot" vs. periodic monitoring The particular season(s) and day(s) depend largely on the assumed sensitivity of the APTS application to each time unit. If it is deemed appropriate to assess the impact of the APTS application under reasonably normal conditions, data collection should be performed during the fall and spring, when weather conditions are not severe, schools are in session, and few people are on vacation. To the extent that the experimental test evaluation involves measures related to travel patterns and transit usage, the contractor should attempt to schedule data collection activities during those two seasons which are most representative of normal conditions. On the other hand, if severe weather conditions or other atypical conditions are an inherent feature of the site and it is desirable to examine the experimental test under a full range 52 of possible conditions, the contractor should schedule data collection throughout the year so the sample observations include extreme as well as normal conditions. If a particular transit service operates seven days a week, then the sample of days should include both weekdays and week-end days (in fact, the data should be stratified by weekday vs. week-end day to highlight the differences during these two periods). Regarding which day(s) to include in the weekday sample, similar logic applies as in the case of seasons. If the aim is to observe the project under typical weekday conditions, then any day(s) with abnormal traffic patterns should be avoided. In some cities, there is a difference between Monday and/or Friday conditions vs. Tuesday/Wednesday/Thursday conditions; if this is known to be the case for a particular test site, then data collection should be scheduled for the three "typical" days rather than either of the typical days. The contractor should consider the special characteristics of the operational test and the site in deciding which days are appropriate. If a large number of days is going to be involved, and there is no particularly significant distinction among days of the week, then a randomly selected sample of days would be preferable. The duration of each data collection period should be determined based on the degree of day-to-day variability and on the required sample size. If the particular item being measured is suspected to vary in behavior from one day to the next, then the data collection period should include several days; if it has been determined that only Tuesdays, Wednesdays, and Thursdays can be used, then several weeks may be necessary to achieve the required sample of days. Moreover, if the sample size required for a particular variable is large, then several days of data collection may be appropriate to obtain the minimum sample of observations. The choice of initiation time for each data collection period is dependent on a number of considerations, the chief one being that the "after" data collection not begin until the use of the APTS application is fully operational and its performance has stabilized. In general, it will probably take at least a few months for an APTS application to become fully operational, with all the "bugs" worked out, and possible behavioral influences associated with the application are eliminated. The desire is to achieve a "steady state" for the system after the application has been implemented. The time to achieve this "steady state" undoubtedly will vary from project to project. Thus, data collection related to the test should not commence until these adjustments and modifications are completed. Other factors determining the initiation date for data collection 53 are the desire to avoid summer and winter months and the overall schedule of the operational test. In most instances, data collection will be performed for discrete phases of the operational test (i.e., before the project is implemented, while the project is operational, and possibly after the project is terminated). Post-operational test data collection would only be performed if there was a desire to see whether operation of the APTS experiment for a limited period had led to permanent changes in people's travel patterns or attitudes. However, if operational test elements are by nature changing continually or if it is expected that the APTS application will cause gradual but continual changes in transit performance measures, then a periodic process of data collection would be more appropriate than merely "before," "during," and "after" data collection. The multitude of data points obtained from a periodic monitoring process will make possible the examination of functional relationships either among measures of interest or in a time series. Moreover, monitoring of certain measures during the early months following introduction of the application(s) may be useful in determining when the effects have stabilized enough to initiate full-scale data collection. It should be noted that if periodic data collection is appropriate, then a sequential analysis procedure (similar to control charts) may be useful to permit reductions in sampling requirements. The contractor's Evaluation Plan should indicate the exact timing of data collection for each measure involving sampling. This information should be presented in a schedule which also shows the projected implementation dates for the various elements of the project. 54 4. GUIDELINES FOR PERFORMING EVALUATION ACTIVITIES This chapter presents suggestions relative to implementing the evaluation of an APTS operational test. During the evaluation implementation phase of the evaluation process, data collection/analysis relating to site characteristics, quantitative measures, and qualitative measures is undertaken according to the plans and procedures laid out in the Evaluation Plan. In addition, information is gathered relative to the project's operational history and external events which may have some bearing on the project outcome. This information is eventually incorporated into the analysis and interpretation of project results. Contractor functions during the evaluation implementation phase include monitoring and/or performance of data collection activities, data reduction and analysis, subjective analysis of information relative to project issues, and synthesis of project findings into a Final Summary Evaluation Report. In accordance with these contractor functions, this chapter of the guidelines is organized into two sections: (1) monitoring/performance of data collection and (2) data reduction, analysis, and presentation. The recommended content and organization of the various contractor reports prepared during this phase are presented in Chapter 5. During this phase, the contractor must maintain a sensitivity to the relationships among the organizations involved in the project -- in particular the local sponsor or project team, FTA, and the Volpe Center (see Chapter 2). The contractor must work closely with these groups at the appropriate times, while maintaining the role and perspective of an external, objective organization assessing the impact of the operational test. 4.1 MONITORING/PERFORMANCE OF DATA COLLECTION Since much of the data required for evaluations will be unavailable from pre-existing data bases and secondary sources, each operational test will undoubtedly involve significant data collection efforts. Given the considerable amount of time and money which will be spent on data collection; careful management and oversight of the data collection process are essential. Where possible and appropriate, data collection may involve the use of students from local colleges and universities. 55 The contractor is responsible for ensuring that data collection is performed according to the Volpe Center/FTA-approved Evaluation Plan. There are three potential alternatives associated with data collection. One of these occurs when the local sponsor or operator collects all data (under FTA/APTS and/or local funding), and the contractor acts in a monitoring role to assure the quality and timeliness of data collected, as well as adherence to procedures laid out in the Evaluation Plan. A second alternative occurs when the contractor collects the data, and coordinates the timing and performance of these activities through the local sponsor or operator. The third possibility is one in which both collect various elements of the data. In order to monitor and/or perform the data collection activities called for in a given evaluation, the contractor will need to maintain open channels of communication with the site, in the form of visits, telephone and written correspondence with the appropriate local agencies as well as subscriptions to local newspapers. In the rare instance where day-to-day contact with the site is necessary, the contractor should arrange to base a member of the firm at the site. Whether data collection is being performed by the contractor or by the local sponsor, the contractor must stay closely involved in all phases to make sure the procedures specified in the Evaluation Plan are followed. In cases where the local sponsor or other local agency is collecting data, the contractor should meet frequently with the agency to discuss progress and problems, work out solutions to the problems, and observe key phases of field data collection. In addition, the contractor should occasionally perform independent spot checks, especially in the case of measures for which the local agency has limited experience in data collection. The contractor is expected to inform the Volpe Center of the status of data collection in its Monthly Evaluation Progress Reports (see Chapter 5 for the recommended content and organization of this type of report). Should there be an unacceptable degradation of quality or timeliness of data collected by the local sponsor, the contractor should notify the Volpe Center in writing. The Volpe Center will in turn take steps through the FTA Project Manager to rectify the situation. Over and above monitoring data collection activities, the contractor should keep abreast of the status of the operational test. This awareness of project operational status is important so that: (1) data collection activities can be smoothly coordinated with ongoing project activities (causing minimum disruption of day-to-day operations), and (2) evaluation results can be 56 interpreted in the context of project history. The local sponsor's quarterly project progress reports to FTA/Volpe Center (see Chapter 5 for recommended content and organization) will be a useful source of information on the project's operational evaluation. However, the contractor is encouraged to obtain a more detailed account of progress/problems relative to implementing and operating the APTS test by talking with the local sponsor at the site. In addition to keeping abreast of project operations, the contractor should be continually watching at the site for unexpected (external) events which might affect the validity of project results. In any implemented operational test, no matter how well controlled or planned, the possibility remains for unexpected events to occur that may have an impact on measures of the project's performance. These unexpected occurrences are classified as threats to the validity of the operational test. Unanticipated developments at the site can take the form of temporary events such as a driver strike or longer-term phenomena such as the closing of a major thoroughfare. The following are examples of unexpected factors that have been experienced in earlier FTA projects, along with an indication of the compensatory action taken to counteract the external event: (1) Changes in employment. There were thousands of unemployed in Seattle due to the high number of layoffs in the aerospace industry. (No compensatory action was taken.) (2) Changes in freeway traffic volumes. Shirley Highway experienced a shift from arterials to the freeway upon completion of new lanes and sections. Minneapolis on the other hand, noted a shift to the freeway due to arterial street construction and land developments within the project. Seattle noted volume shifts on the freeway entrance and exit ramps where new lanes had been added or preferential treatment was given to buses. Seattle also experienced a queuing problem onto the freeway from autos that were diverted from converted ramps. (An adjustment in queuing sequence was made where necessary.) (3) The national energy crisis. Minneapolis experienced a drastic change in traffic volumes from auto to transit during the energy crisis. Although it cannot be determined whether the shift in volumes was directly attributable to this factor, the timing of the initiation of the project during this period may have had some impact on data interpretation. (Extended routes and an increase in the frequency of service were the immediate modifications made to facilitate transporting such large number of people. Also, their marketing campaign was modified -- slowed down in view of the large numbers.) 57 As previously noted, the use of a test-control evaluation design will, in certain cases, mitigate the impact of these unplanned events on the validity of the project results. For further information regarding the phenomena that can jeopardize internal and external validity, see Campbell and Stanley. The contractor is responsible for informing the Volpe Center of any unplanned phenomena which arise during the course of the evaluation. The contractor's Monthly Evaluation Progress Report should describe the potential effects on validity of any phenomena noted, as well as propose changes in the project and/or evaluation to compensate for the unplanned occurrences. Although data collection should generally proceed according to the Evaluation Plan, there may be instances where modification to the originally planned procedures is warranted. The previous paragraph indicated that external events at the site might be cause for modifying the evaluation. Two additional reasons for deviating from the planned approach are discussed below, namely, operational changes in the project, and availability of improved evaluation techniques. Operational changes in the project can come about as a result of contractor recommendations (transmitted in the Monthly Evaluation Progress Reports) or decisions by FTA and the local sponsor. Whatever the impetus for these changes in the scope or operation of the operational test, the evaluation will have to be modified accordingly. The contractor is responsible for assessing the impact on the evaluation of any forthcoming or proposed operational changes, and recommending appropriate modifications of the Evaluation Plan to the Volpe Center. As new data collection techniques are developed in the course of the APTS program, it may be appropriate to modify certain aspects of a project's Evaluation Plan. The contractor will have to assess, on a case-by case basis, whether the potential benefits of the new techniques are sufficient to justify modification to the planned evaluation activities, and then recommend the appropriate course of action to the Volpe Center. In order to further the state of the art of transit evaluation, the contractor is responsible for performing an ongoing assessment of data collection procedures used. The evaluation contractor should maintain close control over data collection procedures used and summarize 58 findings with respect to certain techniques reflected by the Volpe Center for further examination. These findings will include, as a minimum: (1) a narrative description of how the collection procedure was and implemented, (2) an indication of areas in which the technique outperformed expectation, (3) an indication of areas in which the technique was deficient, (4) some summary of the inherent variability in collecting project measures due to the technique itself, as opposed to variability due to other operational test factors, (5) an estimate of the cost of implementing the technique, and (6) where two techniques have been employed to collect the same basic measures, cross-comparisons and a recommendation as to which technique should be used in similar future operational tests. This information will ultimately be incorporated into an appendix of the Final Summary Evaluation Report. 4.2 DATA REDUCTION, ANALYSIS AND PRESENTATION The contractor is responsible for performing all data reduction and analysis, regardless of which agency has collected the data. Data reduction involves the processing of raw data, either manually or using a computer, to yield statistics such as means, standard deviations, ratios, ranges, frequency distributions, coefficients of determination, correlation coefficients, ratios, "t" statistics, and elasticities. The specific statistic to be calculated and the need to control for other variables will depend in pan on the type of measure and type of comparison involved. Quantitative measures such as travel time and vehicle passenger counts might be processed into average values for each level of stratification used. If a comparison of two time periods is involved, the percentage change from the earlier to the later period might be calculated, or two multiple regression equations might be calibrated and their coefficients compared. Quantitative measures relating to schedule dependability might be summarized into average values as well as standard deviations, with comparisons calculated as ratios of standard deviations. Some qualitative measures for example, might be obtained through surveys and 59 might be presented to yield frequency distributions for the response categories. It should be stressed that the level of analytical sophistication and choice of quantitative and qualitative measures will vary from test to test depending largely on the objectives being evaluated. Data reduction may involve the use of statistical inference techniques. If the data are based on a 100% data collection effort (i.e., no sampling), then exact values of the statistics listed above can be calculated. However, if the data has been obtained by sampling (as will usually be the case), results cannot be presented as precise values, since there is a certain probability that the calculated values are different from the true population values. It is recommended that data based on samples be processed into two-sided confidence intervals using two confidence levels: µ =.01 and µ =.05. Appendix B presents further guidelines relative to calculating confidence intervals. The contractor should arrange for smooth transfer of collected data from the collection site (e.g., buses, transit company, roadside stations) to the processing site. Special attention should be paid to details such as labeling and dating of forms, tapes, etc. to make sure that valuable data is not lost or altered. The basic data which are collected during an operational test should be maintained either on appropriate storage devices (e.g., hard discs, floppy discs). While the raw data may not be immediately utilized, it should remain with the contractor (or eventually the Volpe Center) for potential future uses. Data analysis involves the interpretation and synthesis of the processed data and other information to draw conclusions relative to the attainment of project objectives and issues and relative to project transferability. Statistics such as those cited above, which range from the simple to the complex, are carefully examined and pulled together to obtain a comprehensive, in-depth understanding of the effects of the operational test, and the underlying reasons for observed changes. The contractor must apply sound judgment as well as knowledge and experience relative to transit system operations, traffic operations and travel behavior in order to interpret the collected data and place it in proper perspective. To the extent possible, the results of the APTS applications at the site should be supplemented by an assessment of the influence of site-specific and external factors on project outcome, so that conclusions can be made regarding the potential applicability and effects of implementing the operational test in 60 other sites across the country. In order to further enhance project transferability, the analysis/synthesis phase should provide a compilation of lessons learned regarding the operation of the test. The contractor should understand and be aware of the importance that the use of appropriate statistical techniques can attach to the analysis and interpretation of project results. In view of the fact that most aspects of an urban transportation system tend to be dynamic and variable from hour-to-hour, day-to-day, and month-to-month, observed differences could be attributable only to this inherent variability and not to the APTS applications. Furthermore, factors other than the planned and controlled innovations could also be directly related to the observed changes in those measures being collected. It is important to note that, while no single technique exists for removing the potential influence of these external factors, it is possible by careful analysis, to at least point out the occurrence of such events and create an awareness for those who review the project's conclusions and/or recommendations. Hence, it is important to be able to specify whether the observed differences in, for example, travel time are within reasonable bounds of expected variability inherent in the given transportation system, or whether the observed differences cannot be accounted for just by system random variability. If the latter case were true, taking into consideration the potential external influencing factors, one could conclude that the application has in fact provided a real change in the measures being considered. It is to this capability for making valid inferences that the specific statistical techniques apply. Presentation of project results in Annual Project Status Summaries, Interim Evaluation Reports, and Final Summary Evaluation Reports should be in the form of quantitative and qualitative exposition, with exhibits such as tables, graphs, and bar charts serving as the focus for narrative discussion. In no instance should there be an excessive narrative describing all the elements of an exhibit. This tends to be redundant and masks the really important findings. Chapter 5 provides some guidance relative to overall content and organization for the aforementioned-reports. With respect to the format for exhibits, creative techniques for displaying information are encouraged, so long as the information is presented in a clear and accurate manner. In order to provide the contractor with some indication of the types of exhibits that are acceptable, some examples are presented on the following pages. 61 Exhibits 10-13 are clear and informative. While they do not present detailed information, they are useful in highlighting the project findings appearing in an executive summary, which is designed to convey rapidly to the decision-maker the significant conclusions of the project. Back-up exhibits which contain significantly more detail of simple statistical results, multiple regression analysis, and benefit-cost analysis should be contained within the body of the evaluation report or in technical appendices. Exhibits 14-20 fall into this category. The contractor should perform data reduction and analysis as data are collected, so that interim results are available throughout the project evaluation. These interim findings will not only satisfy general curiosity regarding the project's effects, but will also provide feedback information relative to ongoing project operations and evaluation. Examination of preliminary evaluation results may suggest opportunities for modifying the project and/or evaluation procedures so as to increase the utility of the operational test. To facilitate inter-project analyses, contractors will be required to deliver all survey data in an ASCII file format in machine-readable form. 62 Click HERE for graphic. 63 Click HERE for graphic. 64 Click HERE for graphic. 65 Click HERE for graphic. 66 Click HERE for graphic. 67 EXHIBIT 15. CHARGE-A-RIDE USAGE BY CARD TYPE AND TIME PERIOD FOR THE MERRIMACK VALLEY CHARGE-A-RIDE PROGRAM CARD TYPE AM PEAK MIDDAY PM PEAK TOTAL ADULT 1,761 650 1,430 3,841 COLUMN% 69 65 76 70 ROW% 46 17 37 - HANDICAPPED 472 118 331 921 COLUMN% 18 12 17 17 ROW% 51 13 36 - ELDERLY 235 199 105 539 COLUMN% 9 20 06 10 ROW% 44 37 19 - STUDENT 96 29 26 151 COLUMN% 4 03 1 03 ROW% 64 19 17 - TOTAL 2,564 996 1,892 5,452 ROW% 47 18 35 - 68 EXHIBIT 16. COMPARISON OF FARE PAYMENT TIMES USING DIFFERENT METHODS PAYMENT TYPE MEAN STANDARD NO.OF COMPARATIVE TIME(Secs) DEVIATION OBSERVS. DIFFERENCE1 CHARGE-A-RIDE AUTOMATED 6.80 3.30 131 - MANUAL 10.34 5.58 72 SLOWER EXACT FARE CASH READY TO PAY 6.18 4.02 23 NONE NOT READY 7.09 5.46 37 NONE BOTH 6.74 4.98 60 NONE CASH WITH CHANGE PASS PRESENTED 12.57 5.61 21 SLOWER DOLLAR BILL 11.70 2.24 20 SLOWER INCORRECTIONS 13.64 5.34 30 SLOWER ANY COMB. OF 12.86 4.47 50 SLOWER BILLS OF COINS TEN-RIDE TICKETS 5.77 5.80 46 FASTER ____________________ 1 REFERS TO DIFFERENCES OF MEAN OF METHOD COMPARED TO AUTOMATED CHARGE-A-RIDE. THE FARE PAYMENT METHODS ARE COMPARED TO AUTOMATED CHARGE-A-RIDE USING A SIMPLE T-TEST OF DIFFERENCE BETWEEN THE MEAN TIMES 69 EXHIBIT 17. PROJECT EFFECTIVENESS MEASURES FOR THE SEATTLE PROJECT 70 Click HERE for graphic. 71 EXHIBIT 19 RESULTS OF BEFORE AND AFTER ANALYSES FOR PORTLAND SELF-SERVICE FARE COLLECTION SPRING 1982 (PRE-SSFC): DWELL = 5.95 + 1.18 (ALIGHTINGS) + 2.46 (BOARDINGS); R2 = 0.82 (TIME/BUS STOP) (.064) (.052) SPRING 1983 (SSFC): DWELL = 8.26 + 1.58 (ALIGHTINGS) + 1.93 (BOARDINGS) R2 = 0.66 (TIME/BUS STOP) (.064) (.052) 72 EXHIBIT 20 BENEFIT-COST ANALYSIS RESULTS OF SALT LAKE CITY RIDER INFORMATION SYSTEM YEAR INCURRED BY UTA1 COST/BENEFIT CATEGORY 1983 1984 1985 TOTAL SYSTEM COSTS TELERIDE CONTRACT $ 485,918 $36,263 $ 203,070 $ 725,251 UTA LOAN FROM TELERIDE -165,050 - - - UTA SUPPORT COSTS STAFF COSTS 60,671 27,578 - 88,249 INSTALLATION/START UP 29,083 - 22,917 -52,000 TELEPHONE 34,286 15,584 - 49,870 HARDWARE & SOFTWARE 33,000 15,000 - 48,000 MAINTENANCE TOTAL INITIAL COSTS $ 477,908 $94,425 $ 225,987 $ 798,320 SYSTEM BENEFITS REVENUE INCREASES DUE TO 5,753 2,876 6,967 15,596 TRANSIT MARKETING COSTS AND BENEFITS IN 1983 DOLLARS2 INITIAL COSTS $ 77,908 $ 90,570 $ 207,913 $ 776,391 INITIAL BENEFITS 5,753 2,759 6,410 14,922 __________________ 1. EXCEPT THE LAST TWO ROWS, ALL COSTS AND BENEFITS ARE IN 1985 DOLLARS. COSTS ARE AS SPECIFIED IN THE UTA-TELERIDE AGREEMENTS 2. BASED ON ACTUAL 1983-1984 AND ASSUMED 1984-1985 INFLATION RATE OF 4.3 PERCENT 73 5. CONTENT AND ORGANIZATION OF REPORTS At various stages in the evaluation process the contractor for each APTS project is responsible for submitting specific reports to the Volpe Center. These reports include: an Evaluation Plan submitted prior to conducting the evaluation, Evaluation Progress Reports submitted monthly throughout the project, Interim Evaluation Reports submitted periodically throughout the project, and a Final Summary Evaluation Report submitted at the conclusion of the project. The local sponsor for each operational test is also responsible for submitting quarterly progress reports on project status. Appropriate information in these reports will be included in the contractor's monthly Evaluation Progress Reports. This chapter presents recommendations on content and organization which will guide the contractor in the preparation of these reports. The suggested content and organization for the local sponsor's quarterly progress reports are also presented. 5.1 EVALUATION PLAN The Evaluation Plan is written by the contractor to explain, in detail, how the evaluation of the particular project will be performed. The following is a summary of the suggested content and organization format for the Evaluation Plan: (1) Overview of the operational test . Operational test including description of APTS application . APTS program objectives addressed . Other relevant project objectives/issues addressed . Project history (events or studies leading up to test) . Project schedule . Project funding (total operational test costs by source of funding, capital costs by application) . Project local sponsor/operating agency 75 (2) Description of the operational test evaluation . Overview of basic evaluation design, constraints affecting development of the Evaluation Plan . Timing of evaluation stages as related to project implementation schedule . Site data collection plans and sources . Quantitative measures, qualitative measures, and/or information to be collected in connection with each project objective and issue . Proposed data collection/derivation and analysis techniques for each measure . Schedule of data collection activities associated with the evaluation, and identification of which organization (contractor, local sponsor, other local organization) is to perform each activity. Schedule should indicate submittal dates for any Interim Evaluation Reports and the Final Summary Evaluation Report (3) Technical management and cost information . Estimate of contractor person-hours by labor category (e.g., senior, middle, regular, administrative) and task (i.e., management and coordination of evaluation plan preparation and updating, data collection/monitoring of data collection, data reduction/data analysis, and report preparation) for the project . Estimate of contractor direct costs by category of cost (travel, computer, etc.), task and evaluation stage . Estimate of total contractor evaluation costs by task . Estimate of person-hours and costs for data collection to be performed by other organizations (by activity, if possible) To facilitate the incorporation of modifications, the Evaluation Plan should be submitted in looseleaf form and on a WordPerfect file. As modifications are made, each page will have the date of modification indicated. Modifications may result from the initial review of the Plan by the Volpe Center, FTA, and the local sponsor or they may occur during the evaluation implementation phase. As an example of the latter situation, examination of interim findings may reveal certain gaps or redundancies in the originally planned data collection program. 76 Other reasons for modifying the Evaluation Plan during the implementation phase might be operational changes in the project, unanticipated developments at the site, or identification of a refined evaluation procedure. The mechanism for obtaining Volpe Center approval to modify the Evaluation Plan procedures is described below under Monthly Evaluation Progress Reports. 5.2 MONTHLY EVALUATION PROGRESS REPORTS The Monthly Evaluation Progress Reports are written by the contractor to keep the Volpe Center and FTA abreast of the status of the project evaluation the contractor is performing. These reports are intended to be as concise as possible. The following is a summary of the suggested minimum content and organization for the Monthly Evaluation Progress Report. (1) Review of evaluation activities during the past month. Evaluation-related problems encountered and actions taken to rectify them. Narrative highlights of project and/or evaluation related external factors and other events which appear to be significant and might influence the evaluation of the project. A review of the implementation process, including a discussion of problems and issues encountered, steps taken to resolve such issues, and associated operational delays, difficulties, and other consequences, if any. (2) Status of data collection and analysis activities that have taken place in the past month (performed by both local sponsor and evaluator). Any contractor documentation on preliminary results which have been generated in this area should be appended to the Progress Reports or could be submitted separately as special technical memoranda. (3) An indication of whether the evaluation is proceeding according to schedule, and, if not, reasons for the departure. (4) A brief discussion of anticipated activities to be covered during the succeeding report period. Forthcoming Interim Reports, if any. (5) Comparisons of cumulative budget to actual expenditures. Estimate of costs to complete evaluation tasks. (6) Recommendations for changes, if any, to the Evaluation Plan, and the reasons such changes are recommended. (Volpe Center concurrence is needed before any changes to the Plan can be made.) 77 5.3 INTERIM EVALUATION REPORTS Interim Evaluation Reports are written periodically by the contractor to present interim findings relative to all or some of the operational test objectives and to evaluate those aspects of the project where it is applicable to do so. Although submitted to the Volpe Center, they will also be further disseminated to other interested parties. If the evaluation process is divided into distinct stages whose durations fall roughly within a month time frame, then interim reports should be written at the end of each stage. Otherwise, interim reports should be written annually, except that no interim report is needed at the end of the operational test. The suggested content and format for Interim Evaluation Reports is similar to that presented for the Final Evaluation Report described next. 5.4 FINAL SUMMARY EVALUATION REPORT The Final Evaluation Report is structured by the contractor early in the operational test and completed at the conclusion of the project. Its purpose is to synthesize the findings relative to each of the APTS Program objectives and other relevant objectives/issues into an evaluation of the overall project. Although submitted to the Volpe Center, it is also meant for dissemination to a technical audience. The suggested content and organization for the Final Summary Evaluation Report are given below. (1) Executive Summary . Should be capable of standing on its own and being published separately. (2) Project Overview . Description of project innovations, the APTS Program objectives addressed, and other relevant project objectives/issues. A brief overview of the operation of the project over its life, and highlights of project related external factors and other events that have been significant enough to influence the project. A review of the implementation process, including a discussion of problems and issues encountered, steps taken to resolve such issues and problems, and associated operational delays, difficulties, and other consequences, if any. (3) Site Overview . Description of the site, presentation of pertinent site data, and highlights of site related external factors that may have been significant enough to influence the project. 78 (4) Evaluation Overview . Description of the basic evaluation procedure and the timing of evaluation stages. (5) Project Results . Assessment of the project in terms of its attainment of relevant APTS Program objectives and other (local and/or national) project objectives; and insight into project issues associated with operational feasibility and characteristics of the APTS application being tested. Relevant data are analyzed and presented in the forms of charts, graphs, and/or narrative. (6) Implications Regarding Transferability . Assessment of the influence of site-specific characteristics and external factors on the outcome of the operational test. . Lessons learned, based on practical experience, relative to the implementation and operations of the APTS application. Can include suggestions for project modifications at the test site or for future APTS applications in other locales. (7) Appendices . Project costs . Data Collection: Site data, quantitative measures, and qualitative measures collected. . Assessment of evaluation procedures employed (e.g., effectiveness of particular survey approaches used, cost/ accuracy of innovative data collection techniques). 5.5 QUARTERLY PROJECT PROGRESS REPORTS The Quarterly Project Progress Reports are written by the local sponsor to keep FTA/the Volpe Center abreast of the status of the operational test project implementation for which the local sponsor is responsible. The following is a summary of the suggested minimum content and organization for the Quarterly Project Progress Reports to be prepared by the local sponsor. (1) Review of operational test activities during the past quarter. Project-related problems encountered and actions taken to rectify them. Narrative highlights of external factors and other events which appear to be significant and might influence the project. A review of the implementation process and an indication of whether the project is proceeding according to schedule and, if not, the reasons. (2) Status of planned data collection activities. 79 (3) Comparisons of budgeted to actual expenditures. Estimate of costs to complete project. (4) Recommendations for changes, if any, to the conduct of the project, and the reasons such changes are recommended to the Volpe Center. 80 APPENDIX A SURVEY EXECUTION AND DESIGN It is anticipated that the evaluation of every APTS operational test will require data that can be obtained only from surveys, and will therefore require some form of survey data collection. Among the possible survey respondents are APTS service users, auto users, service area residents who do not use transit, and transit company personnel. Typical survey objectives might include: determining user and non-user characteristics, attitudes toward transit service, and past and present travel behavior; measuring modal shift; and assessing the experience of transit officials with regard to implementing a new APTS technique. Although the specific contexts in which the surveys are conducted may differ, there is still a need for consistency of procedure in survey design and data collection to insure comparability of results. In surveys, the researcher is collecting data from real life situations, which means that many unanticipated, spontaneous, and unusual situations will arise. Some of the unanticipated situations are briefly mentioned in Section 4.1, and more detailed discussion regarding these situations and their association with the validity of the evaluation results is presented in Campbell and Stanley. To compensate for the survey researcher's lack of control of the experimental situation, the need for consistency and the establishment of general policies or guidelines to handle a great variety of possible developments is most important. This Appendix contains guidelines for use in formulating and carrying out surveys. It discusses how to define the populations to be sampled (i.e., the survey universes), describes how to select samples that will be representative of that universe, examines techniques for surveying the samples selected, presents suggestions as to survey content and format (including a list of standardized questions and, in some instances, standardized responses to serve as a basic set for most surveys), and discusses the problem of non-response bias. A separate section at the end of this Appendix contains guidelines for conducting interviews with transit company personnel (e.g., drivers, management, mechanics). It should be stressed that this Appendix presents no hard and fast rules which must be followed by each contractor. It merely guides the contractor in designing and executing surveys. In determining survey methodology, the contractor should consider potential alternatives and A-1 give the rationale for decisions made in terms of the survey objectives, site characteristics, and any other relevant factors which have influenced the decision. A.1 DEFINING THE SURVEY UNIVERSE The first step in executing surveys is to define the survey universe (i.e., the groups about which the surveys are seeking knowledge). It is apparent that knowledge about project service users' travel behavior, characteristics, and attitudes toward transit is needed in an evaluation of project service. Moreover, an evaluation of project service will usually not be complete without some data on non-users, particularly to identify who they are and why they do not use the project service. Accordingly, there are two survey universes which will be relevant for APTS projects: users of the transit service employing the APTS application, and non-users of this service. Users are defined as those who ride this service at least occasionally but still on a regular basis, e. g., regularly twice a month. Non-users are defined as those using alternate modes (i.e., other than the APTS project service) who make trips that could be made on the project service. Occasionally, there will be a third survey universe of interest, the general population of the region in which an APTS project is being implemented. Attitudinal surveys of this universe will be used to obtain a profile of the community in which the transit service is being provided. It should be apparent that many of the questions asked users, non-users, and the general population will be different. Definition of the term APTS project service area allows a more precise definition of non-users and the general population. The project service area is defined as the area that comprises on the order of 90 to 95 percent of the origins and destinations of the users of the service. Since non-users are potential users, the origins and destinations of non-users should be comparable to those of users. Non-users can now be defined as persons not using the APTS project service who make trips that begin in the origin portion of the service area and end in the destination portion of the service area at the same times as users make these trips. The general population in the region of the operational test can now be defined as the population residing within the service area. A-2 The operational test service area is usually not well defined at the outset of the project and must initially be estimated. In some projects, specifically demand-responsive projects, the origin and destination portions of the service area are given. At the other extreme, in projects in which park-and-ride is a significant access mode, it may be impossible initially to estimate the service area accurately. A conservatively estimated area that includes all possible park-and-riders would have to be initially defined as the origin portion of the project service area. Once survey data on the origins of park-and-riders is obtained, a more accurate estimate of the service area can be made, and non-users can then be identified. A.2 SAMPLING THE SURVEY UNIVERSE The next step in executing surveys is selecting an appropriate sample for surveying users, and, where applicable, selecting appropriate samples for surveying non-users and the general population. The purpose of sampling is to reduce the amount of data collection required. Rather than obtaining information from every member of the universe, the principles of sampling provide ways to obtain information from a very small portion of the universe. Sampling procedures also indicate the accuracy with which the characteristics of the universe have been represented. A key assumption in sampling is that, prior to drawing a sample, the complete universe has been identified. Therefore, every members of that universe has a known probability of being selected for inclusion in the sample. The quality, or representativeness, of any sample is directly derived from the completeness of the identification of all members of the designated universe. For these reasons, careful definition of the universe and selection of a source from which to draw a sample is very important. If the listing of the universe, or the sampling source, is biased through failure to include affected members, whether deliberate or random, the sample may magnify the bias and may not represent the universe. A sample of users can be selected from among those onboard the transit vehicles or among those at transit collection points (i.e., stations), park-and-ride lots, or transfer points. For APTS projects in which all users are registered (e.g., demand responsive or subscription A-3 service), a sample can be selected from among the registration lists. For projects which serve specific employment or activity centers, a sample can be selected at these centers. Selecting a sample of non-users is considerably more involved than it is for users. While the user group is identifiable (and can be directly sampled), the non-user group cannot explicitly be identified before it is sampled.A-1 A larger group must first be sampled, and then the trip origins and destinations of the survey respondentsA-2 examined in order to identify non-users (i.e., those whose trip origins and destinations are within the project service area). A definition of the project service area (as previously discussed) is a prerequisite for identifying non-users. In a project in which travel by users and non-users is in a specific direction through a corridor, non-users, specifically auto users, can be sampled from license plate matches. A screenline is selected which intercepts the main arterials carrying autos between the origin and destination portions of the project service area. A sample of the license plate numbers of the autos crossing the screenline is recorded and a list of names and addresses of the owners of these autos is obtained from Department of Motor Vehicle records. This list (or a subset of this list) constitutes a sample in which a large percentage are project service non-users. Some of those crossing the screenline do not make trips that begin and end in the project service area, and are, therefore, not non-users. However, the entire sample must be surveyed because it is not known who the non-users are until the trip origins and destinations of all those in the sample who completed their surveys are examined. In certain very specific cases, samples can be selected directly from the traffic stream (e.g., at toll booths, at off-ramps, or from among carpoolers assembling at parking lots). In operational tests where travel by users and non-users is not in a specific direction nor through a corridor, the non-user universe cannot be sampled using the above methods. In such cases, a sample may be drawn from households in the origin portion of the particular project's service area. Lists of households from which to select a sample could be obtained from utility ____________________________ [A-1] There may be APTS projects directed at carpooling. In this, carpools would be "users" as defined in this appendix. However, the population of carpoolers is not explicitly identifiable; therefore, it must be sampled by the same methods used for non-users. [A-2] This information is requested in the survey. A-4 records, insurance company records, census block statistics, telephone books,A-3 property tax records, etc. Many of the people in these households do not make trips ending in the destination portion of the project service area, and are, therefore, not considered non-users. As previously discussed, the entire sample must still be surveyed because the non-users cannot be identified until after the entire sample is surveyed. If the preceding method is used for obtaining a sample of non-users, it should be noted that the households selected constitute a sample in which a moderate percentage of the people are users. It may be desirable to identify users before they are surveyed (by asking all those sampled if they are users) in order to ask them questions pertaining to their use of the project service. In all samples of households, an attempt is made in each household to survey only that individual in each household who makes a trip ending in or near the destination portion of the project service area.A-4 More than one household member is surveyed only when more than one makes this type of trip. For operational tests which serve specific employment or activity centers (e.g., handicapped and elderly service or subscription service), a sample of non-users does not have to be drawn from among households. A sample can be selected from among people at these centers which would include non-users (and users also). If users are surveyed, they should be identified before they answer any questions in order that the questions asked pertain to their use of the project service. Where a sample of the general population of a region is needed, the sample will always be selected from among the households in the project service area. Again, lists of households can be obtained from utility records, insurance company records, census block statistics, etc. _____________________________ [A-3] Where the telephone book is used as the sampling source, there is considerable danger of obtaining a biased sample. Many households choose to have unlisted telephones. Also, lower income people are less likely to have telephones, as are residents of boarding houses. Random digit dialing not only poses potential bias problems but also will be costly because business ant non-residential phones will be selected. [A4] This comment is also applicable to surveys that are sent to registered automobile owners whose names were obtained from license plate matches. A-5 Regardless of the methods chosen for selecting samples of both users and non-users (and possibly of the general population), every effort should be made to assure that samples selected are unbiased and large enough for the desired statistical confidence. Such an approach involves estimating the percent of persons surveyed who are in the universe (i.e., who make applicable trips in the service area), estimating the response rate, and developing a random selection process that aims at the desired number of samples.A-5 In developing a random selection process to sample users onboard vehicles, examination of vehicle operating schedules and recent passenger counts, if available, will be necessary to design where and when to select the vehicles on which to sample users. However, the following sources of bias in vehicle operating schedules must be considered when deciding on the utility of a particular schedule for developing a sampling source: (1) unscheduled vehicle runs, most likely to occur during peak hours, and therefore with high passenger loads; (2) schedule delays, breakdowns, and accidents, also most likely to occur during peak hours when there are high load factors; and (3) the occurrence of external influences on ridership in the interim, such as a strike among people who might have formerly used this mode of transportation, the opening of a new shopping center or school along the route, or unique events such as a concert. These sampling hazards should be kept in mind and some attempt should be made to build corrections into the research design to compensate, such as oversampling on certain routes. In many situations, developing a random selection process that obtains the desired sample size simply involves selecting every Jth person going past a given point, or every Jth person on a list of users of a given system, or every Kth person on a list of employees at a given location, or recording the license plate number of every Lth auto going past a given point. To obtain a random sample of the households in the origin portion of a project service area, every Mth household on a list of all of the households in the area could be selected; or the random clustered household sampling method could be used. This method divides the origin portion of the service area into smaller areas (usually blocks) of approximately equal population and randomly chooses a sample of the resulting clusters in which every household in each cluster is a part of the sample. ___________________________ [A-5] See Appendix B for a discussion of sample size determination. A-6 The possibility of sampling bias occurring through use of a particular sampling method should not rule out its use. That sampling method may be very appropriate in certain project evaluations. However, where little can be done to minimize the effect of bias, other sampling methods should be considered. For each survey required for a particular evaluation, the contractor must carefully describe the universe to which survey research findings will be generalized and identify the most complete enumeration or sampling source available for that universe. Actual selection of a sampling source must be justified in terms of its complete coverage of the affected universe and also in light of the survey objectives. A.3 TECHNIQUES FOR SURVEYING THE SAMPLES SELECTED The final step in executing a survey is determining what techniques are applicable for surveying the samples that have been selected. There are five basic techniques for surveying these samples: (1) Self-administered questionnaires handed out by individuals (e.g., survey takers, bus operators, personnel at employment or activity centers), and collected by individuals (not necessarily the same individuals who handed out the questionnaires); (2) Self-administered questionnaires handed out by individuals and returned by mail; (3) Self-administered questionnaires given out by mail and resumed by mail; (4) Face-to-face interviews; and (5) Telephone interviews. A summary of the applicable techniques to be used with each possible sampling method is shown in Exhibit A-1, which appears at the end of this Appendix with all other exhibits. With all of these techniques, the greater the amount of personal contact between user and survey takers, the higher the response rate and the quality and detail of the responses. However, the greater the amount of personal contact, the higher the cost.A-6 In fact, the face-to-face interview initiated at homes, while eliciting the highest response rate, is generally too costly to _________________ [A-6] choosing a survey technique, careful attention should be paid to costs associated with the data processing and analysis of survey findings. A-7 be considered in the evaluation process. It should only be used in conjunction with the random clustered household sampling method, where the number of personal home interviews to be conducted is small and covers a small area. By significantly decreasing the area in which a given size sample lies, the cost of using personal home interviews is reduced. Where a self-administered questionnaire is used to survey a sample, the response rate will inevitably be lower than where a face-to-face or telephone interview is used. To improve the response rate it may be desirable to allow for a wave of follow-up procedures, such as phone calls and postcard follow-up. Generally, the self-administered questionnaire is the most easily conducted and most cost effective survey technique. Self-administered questionnaires initiated onboard or at collection points are most widely applicable. If the questionnaires are short enough to be completed by all users while they are onboard and there are few standees, the users should be instructed to complete the questionnaires while onboard and return them as they leave the vehicle. If the questionnaires are initiated onboard and the number of vehicles on which users are surveyed is not large, consideration should be given to stationing survey takers onboard each vehicle to hand out and collect the questionnaires, give instructions, and answer any questions. If the questionnaires are initiated at collection points and the number of points at which users exit their vehicles is small, consideration should be given to stationing survey takers at the exit points to collect the questionnaires. The additional expense incurred with this degree of personal contact generally pays off (i.e., the response rate is high and the cost per completed survey is low). Where self-administered questionnaires are too long to be completed by all users while they are onboard or where there are many standees, questionnaires that are to be mailed back should be used. The response rate for a mail back questionnaire will be considerably lower than for a questionnaire completed onboard. This should be kept in mind when developing the sampling techniques. When questionnaires are sent by mail, a cover letter giving instructions and explaining the purpose of the survey should accompany-each questionnaire as should a self addressed, stamped envelope for mailing back the completed questionnaire. It would also be advisable to send out "follow-up" letters a few days after the questionnaires are sent out as a reminder to complete the questionnaires. A-8 There are situations where it is advantageous to conduct personal interviews of users onboard vehicles or at employment or activity centers rather than to have these users complete self-administered questionnaires.A-7 Where the total user population to be surveyed is small, a high response rate may be needed to obtain the desired statistical confidence. In such a situation, a self-administered questionnaire may not obtain a high enough response rate, while personal interviews of users onboard vehicles would. Where there may be considerable misgivings about answering a self-administered questionnaire, as on a crowded bus or train in some parts of some large cities, personal interviews conducted onboard vehicles may be the only means of obtaining an acceptable response rate. Where the users being surveyed are asked about concepts or behavior that are somewhat complex, a personal interview will be much more effective than a self-administered questionnaire in eliciting usable responses. Handicapped and elderly users may have difficult writing and it may be difficult for them to respond to a lengthy self-administered questionnaire. It should be noted, however, that personal interviews are relatively expensive and labor intensive. Where samples are selected from service registration lists, users can be sent self-administered questionnaires by mail. Where it seems that a very low response rate would be obtained with the mail back questionnaire, or where a high response rate is necessary, the telephone interview would be superior. Moreover, sampling bias would be minimized because all of the users' telephone numbers would be known from the registration lists.A-8 For surveying non-users, no single technique is widely applicable. Where a sample of auto users crossing a screenline is surveyed, questionnaires could be sent to the auto drivers by mail (from license plate matches) or these same auto drivers could be interviewed by telephone; or auto users selected directly from the traffic stream could be given questionnaires to be returned by mail. For example, where autos are selected by license plate matches, auto occupancy would be recorded along with license plate number, and mail-back surveys mailed ______________________ [A-7] When surveying users at collection points, there generally is not enough time to question them by personal interview. [A-8] It should be noted, however, that it will not be possible to contact all the persons in the telephone survey sample within the survey time frame. Those not contacted may be a non-random group, with the result that those who are actually interviewed by telephone may no longer be representative of the universe. Therefore, great care must be exercised when sampling by telephone interview. A-9 out according to auto occupancy. Those who drove alone would be mailed one form; carpool drivers would be mailed a set of different forms -- a carpool driver form for themselves, and carpool passenger forms to be given to those who rode with them. In some projects, where autos are also selected by license plate matches, the owners of the observed autos are surveyed by telephone interview. No carpool passengers are surveyed in this fashion. Carpool passengers can be surveyed directly from the traffic stream. In one situation, many carpoolers assembled at a parking lot designated partly for that function. Before each carpool left the lot, each member of the carpool was given a self-administered questionnaire to be mailed back.A-9 Where a sample of non-users (and users also) is surveyed at specific employment or activity centers, those techniques which are applicable for user surveys initiated onboard or at collection points should be considered. This, in general, means that self-administered questionnaires should be used. Where a sample of households in the origin portion of the project service area, which includes non-users (and users), is surveyed, no single survey technique is widely applicable. Questionnaires could be sent to those households by mail to be returned by mail, telephone home interviews could be conducted, or personal home interviews could be conducted where the sample is selected using the random clustered sampling method. It is anticipated that the Volpe Center will set up a "Survey Notebook" in which will be kept a record of the survey experience of the contractors during their performance of APTS evaluations. In order for the Volpe Center to maintain this notebook, the contractor will supply the Volpe Center with a copy of the survey form, information on universe size, sample size, cost, and response rate, and reasons associated with non-response. A.4 SURVEY DESIGN PRINCIPLES It is apparent that, because different surveys are directed at different survey universes using different sampling sources and different techniques, surveys will vary in content and length. Nonetheless, all surveys should have the same basic organization, sequence, and ____________________________ [A-9] Some carpool drivers might have been surveyed twice if their license plates had been recorded. A-10 wording of standardized questions. This section presents basic principles on survey organization, length, question sequence and wording, and standardized questions that should be followed in designing the survey instrument. A.4.1 Organization There should be four elements in all surveys, whether user or non-user. They are in order of their appearance in a survey: (1) Introduction - This is a brief statement of the survey's purpose and potential utility and guarantees the respondent's anonymity. It will be verbally delivered if an interview technique is selected, or will be printed at the beginning of a self-administered questionnaire. (2) Behavioral and Attitudinal Measures - These refer to the set of questions specifically measuring the survey's objectives, such as modal shift, satisfaction with level of service, etc. (3) Social and Demographic Measures - These are measures of the respondent's characteristics which are important in interpreting responses to behavioral and attitudinal measures. Transition to this section of a survey needs to be prefaced by either a verbal or written explanation, as appropriate, such as "Now we need to know a little about you..." (4) Closing Statement - This is a brief expression of thanks to the respondent for participating, with some indication of the importance of the eventual utilization of his responses, and a request for any additional comments or observations from the respondent. A.4.2 Length The overall length of the survey depends on the particular objectives of the survey and the survey techniques used. In general, surveys which are to be completed onboard transit vehicles and at employment and activity centers should be shorter than those surveys completed at home, since they are being administered to respondents in a less comfortable and relaxed environment. Self-administered questionnaires which are handed out should be limited in length to one side of a sheet of paper or a large postcard. Surveys which are to be completed onboard transit vehicles and at employment and activity centers (whether in interview or self-administered A-11 format) should be shorter than surveys which can be filled out at the respondent's convenience and returned by mail. Moreover, they should be short enough so as not to delay the respondent in his trip or current activity. The length of surveys which are completed in the home varies depending on the method of administration. Telephone surveys should be fairly short since it is difficult to retain the respondent's attention for any longer period given the impersonal nature of the contact. Self-administered mail-back questionnaires sent by mail can be longer than self-administered mail-back questionnaires handed out because there is more opportunity to enlist the respondent's cooperation. However, mail-back questionnaires given out by mail should not be as extensive as personal interviews conducted in the home, since the personal contact is lacking which might encourage a longer attention/cooperation span on the pan of the respondent. A.4.3 Question Sequence and Wording There are several general principles describing question sequence and wording that apply to all questions. First, questions should be arranged logically to lead the respondent into the frame of reference of the issue under study.A-10 It is recommended, following the introductory material, to begin the questionnaire or interview schedule with behavioral or attitudinal measures of responses to transportation alternatives because these relate most closely to the announced purpose of the data collection effort. Social and demographic data should be collected near the end of the survey instrument, reserving any questions about income as near to the end of the survey as possible.A-11 ________________________________ [A-10] See pages 26 ff in Federal Highway Administration with Urban Mass Transportation Administration, Urban Mass Transportation Travel Surveys, for an extended discussion of the basic considerations in designing surveys. Two very practical descriptions of interviewing and coding guidelines helpful in developing format are contained in: Survey Research Center, Interviewer's Manual, Institute for Social Research, University of Michigan, May 1969 and Survey Research Center, A Manual for Coders, Institute for Social Research, University of Michigan. [A-11] Measures of-income are the most difficult to obtain accurately and arouse the greatest resistance in the respondent. Sometimes a respondent is asked to point to an amount on a card or circle an approximate amount to lessen the resistance. However, these items arouse such resistance that they must be at the end of the data collection instrument so the hostility produced will not destroy the rest of the data collection. A-12 Questions should be as short as possible and in clear, concrete language. Visual format is also important. In self-administered questionnaires, it enhances the respondent's likelihood of completing the form, and in interview format surveys, it makes the interviewer's task faster and easier. Questions should be laid out in a fashion that ensures ease of coding and processing responses and appears attractive at the same time. Fill-in questions should be avoided where possible, because they often are difficult to code. Where they are used, responses should be anticipated and precoded to reduce costs and enhance consistency. Coding blocks can be left at one side of the survey form and the field editor can check to insure that the information is transferred. This procedure makes the survey also function as a code sheet. The survey should be checked to ensure that it is as parsimonious and logical as possible. There are several ways to do this. First, every question ought to be evaluated to ensure that it contains a measure related to one of the specific project objectives.A-12 Second, advance planning of the data analysis, through the construction of dummy tables, will ensure that every variable measured contributes to the eventual data analysis. Finally, pretesting of the survey instrument will identify any questions which, because they are confusing to the respondent or of limited use in the evaluation, should be changed or omitted. Pretesting has even more far-reaching benefits. It will uncover any procedural problems which may arise during the survey process and reveal any problems which are particularly characteristic of urban areas, such as a sizable number of functional illiterates or foreign speaking respondents who cannot complete a self-administered questionnaire or a systematic refusal to participate by some sectors of the population. The pretest of the survey form must be conducted with respondents as identical to the proposed survey respondents as possible without contaminating the sampling source. Finally, all survey questions should be checked against the provisions of the Privacy Act of 1914 to verify that none of the questions violates any person's right to privacy as spelled out _______________________ [A-11 There are several exceptions to dust guideline. One is the deliberate use of one or two meaningless questions in order to lead the respondent into a particular frame of reference. This is frequently necessary when seeking information on embarrassing, unusual, highly specific or complicated issues. This technique will increase the validity of the data subsequently collected. A second exception is measuring respondent’s opinions of service features that have not changed as part of a set of questions about respondents’ reactions to improved service features. This combination of questions will measures if a "halo effect" exists in terms of respondents’ overall positive evaluation of the mode when only several aspects have been changed. A-13 in the Act. It is recommended that the contractors familiarize themselves with the provisions of the Act. A.4.4 Standardized Questions It will be useful to ensure that the data collected in different evaluation projects is consistent in format. Fostering consistency means that an economical amount of data will yield a maximum amount of information. Secondly, consistency facilitates comparisons between projects, generating a more universally applicable understanding of the responses to transit innovations. Finally, and most importantly, developing consistent data collection categories based on the U.S. Census will mean that results of any survey can be corrected for sampling error and potentially extrapolated to any other area. This section discusses standardized formats for measuring behavioral, attitudinal, and social/demographic characteristics. A.4.4.1 Behavioral Measures Selecting questions to measure travel behavior is very much influenced by the objectives of a particular survey. Some general suggestions regarding ways to collect and code such information to increase consistency among surveys will be described. The following measures of travel behavior are most likely to be asked in almost every survey: transit vehicle boarding and alighting points (user surveys only), trip origin and destination (all described in terms of addresses), trip purpose, and trip start and end times. Additional frequently collected data for surveys includes access mode to transit vehicle, when present mode was first used for this particular trip, former mode used for this particular trip (with some attempt to control for external influences, such as a residential move), reason for switching mode, fare (user surveys only), tolls and parking cost (non-user surveys only), frequency of use, access time at origin and destination (user surveys only), availability of mass transit alternatives, back-up mode, and number of transfers required (user surveys only). Exhibits A-2 through A-9 are examples of bus, automobile driver, and automobile passenger surveys. These exhibits, together with the preceding discussion, indicate the possible range of information which can be collected on travel behavior. Clearly, the determination of which particular items to include in a survey depends on the survey objective, desired survey A-14 length, and circumstances under which the survey is conducted. Furthermore, the specific wording of the questions relating to travel behavior depends on the method of administering the survey and the overall tone of the survey and sequence of questions. Exhibits A-10 through A-16 present recommended question formats and response categories for the measures of travel behavior which are likely to be included in most user and non-user surveys. These recommendations are based on a review and evaluation of questions asked in past surveys (including Census Journey-to-Work) and are directed to the five basic types of surveys (See Exhibit A-1). In designing a survey for a particular operational test, the contractor should follow these guidelines to the extent consistent with the scope and objectives of the survey. Any significant deviations from the recommendations, particularly modifications of suggested response categories, should be explained to the Volpe Center in a memorandum accompanying the draft survey instrument. A.4.4.2 Attitudinal Measures Attitudinal items will be used in many surveys to measure the respondent's evaluation of the APTS application and the transit service provided, specifically in terms of such characteristics as reliability, convenience, attractiveness, and safety of alternative modes. Attitudinal questions may also be used, if applicable, to determine what factors have influenced a modal change. Construction of such items requires careful design and will lengthen the survey's administration time. Occasionally, attitudinal questions may be used to obtain a profile of the community in which the transit service is being provided. An entire survey would then be designed explicitly for the purpose of determining the opinions of the general population in the project service area to such things as the role of government, environmental issues, adequacy of transportation facilities, and desirability of travel by alternate mode. Examples of attitudinal questions appear throughout the aforementioned Exhibits A-3 through A-9, and also in Exhibits A-17 and A-18. The set of questions in Exhibit A-17 can be used both to measure users' and non-users' evaluations of the transit service provided and the factors that have influenced their modal choices. This set of questions can also be used to learn about the opinions of the general population regarding travel by alternate modes. Note that respondents are asked not only for opinions about different travel characteristics but also for a A-15 ranking of the relative importance of these characteristics. The latter set of questions is needed to put the respondents' opinions about the different travel characteristics into proper perspective. For example, if several respondents indicated that "car" had a very high status and "bus" had a very low status, it might at first appear that the status of the automobile might deter the use of bus transit. However, the responses would be considerably less significant if these same respondents indicated that the "status" travel characteristics was rather unimportant to them. The set of attitudinal questions in Exhibit A-18 can be used to obtain a profile of the community in which transit service is being provided. There are no specific recommendations for the format of attitudinal questions, since the design of such questions is entirely dependent on the particular attitudes being measured (e.g., opinions of a very subjective item or perceptions about items which are independently measurable) and on the overall survey context. However, the following discussion presents some general informative guidelines regarding the treatment of responses to attitudinal questions. There are three types of response categories which can be used for attitudinal questions: nominal, ordinal, and interval scales. Nominal data consists of mutually exclusive categories with no implied rating of the responses (e.g., questions with "yes," "no" answers). Responses such as "like very much," "dislike," "dislike very much" represent ordinal level data, with an implied rank ordering. Interval data involves the use of numerical scales (e.g., asking people to indicate their opinions on a scale of 1 to 5). Since interval scales require prior validation and careful application, it is recommended that attitudinal questions be limited to nominal or ordinal response categories. Moreover, it is recommended that the survey data be represented in the form of frequency distributions, rather than statistics such as means which have an implied ranking. A.4.4.3 Social and Demographic Measures The inclusion of certain social/demographic questions in surveys serves the dual purpose of (1) providing data on respondent characteristics which might show a correlation (perhaps even a causal relationship) with measured behavioral attributes, and (2) providing data about respondents which can be used in conjunction with Census data to check survey accuracy, determine non-response bias, and extrapolate survey findings to other areas. A-16 The amount and nature of social/demographic information collected depends on a number of factors, in particular, the desired length of the survey and the extent to which the data will be correlated with behavioral data and used for extrapolation purposes. It is recommended that the following items be included in every survey: respondent's sex, age, household income, the number of autos in the respondent's household, and availability of an auto for the particular trip(s) made on project service (user surveys only). Depending on the survey objectives, scope, and administration format, the following are some of the additional items which might be included: whether the respondent has a driver's license, the general (regular) availability of an auto for a particular trip type (e.g., work, educational level completed, occupation, and length of residence and employment at present location). Examples of questions on social/demographic variables appear throughout Exhibits A-3 through A-9. Exhibits A-l9 through A-27 present the recommended question format and response categories for most of the social/demographic measures listed above. It is considered important to collect and code this type of data in categories which are equivalent to, or collapsible into, Census categories, so as to facilitate comparisons with the same type of Census data for the survey area (for accuracy check purposes),A-13 or to permit the use of other types of Census data to amplify survey findings (with the collected data serving as a bridge between the survey population and the Census population). Special purpose surveys may require a greater amount of detail about a particular social/demographic measure, but the stratification should be compatible with commonly used Census breakdowns.A-14 A.5 NON-RESPONSE BIAS Use of the guidelines presented in this Appendix to design and execute a survey does not insure that the responses obtained will accurately reflect the characteristics, travel behavior, and/or attitudes towards the operational field test of the entire sample selected even though the sample itself is unbiased and totally representative of the population from which the sample was _________________________________ [A-13] Census tract or block data on family income will be a good check on reporting accuracy. [A-14] See U.S. Census, Volume I: Characteristics of the Population. Part II, Appendix B for a detailed discussion on the format of questions. See also "1980 Census User Guide," U.S. Department of Commerce, Bureau of the Census, Washington, DC, June, 1983. A-17 selected. It is possible that the characteristics, behavior, and attitudes of the part of the sample that did not respond to the survey are different from those of the part that did respond, hence producing non-response bias. Pretesting of the survey instrument may or may not reveal this problem when it exists. Even if pretesting does reveal the problem, there may be no effective means of eliminating it. This is especially true if there is a systematic refusal to participate in a survey by certain segments or personality types in the population. It is recommended here that an attempt be made in every survey to determine whether or not non-response bias exists and how it might affect the validity of results. There are no specific guidelines for ascertaining the existence of non-response bias. In general, non-respondents can be reached with a very short survey containing but a few key questions that is administered with considerable personal contact. Where non-respondents cannot be identified, the special survey would be given with the regular survey to a part of the sample. Many of those who do not respond to the regular survey will respond to the special survey. There, non-respondents can be identified after the regular survey has been completed, only they would be given the special survey. The responses of respondents and non-respondents to the few key questions can then be compared to determine whether the responses of respondents and non-respondents are significantly different, and therefore, whether non-response bias exists. The contractor should attempt to devise a specific methodology for determining whether non-response bias exists in the survey responses obtained from the surveys being conducted. A.6 INTERVIEWS WITH TRANSPORTATION AGENCY PERSONNEL There are situations where it may be useful to conduct interviews with transit company personnel (e.g., drivers, dispatchers, mechanics, management personnel from the agency operating the project service). In some cases, such interviews could be used to develop ideas for questions and sets of responses for surveys of users and non-users. This is useful in situations where changes are being hypothesized, and agency personnel could give their opinions and insight on measuring these potential changes. In other cases, such interviews could be used to check the validity of collected data and survey responses. In some very specific cases, such interviews could provide first-hand data on A-18 certain APTS applications. For example, drivers and mechanics could provide information on the operating and maintenance characteristics of smart vehicle and smart card systems. Management could provide insight into the enforcement problems associated with high occupancy vehicle lanes. Dispatchers could provide insight into the operating characteristics of an automated vehicle location system. The situations discussed are not meant to be all-inclusive. No specific guidelines have been put forth. It is up to the contractor to decide whether interviews with transit company personnel would provide information needed to perform the particular evaluation, and to design the appropriate survey technique. Individual interviews and focus groups are practical methods of obtaining information from agency personnel. A.7 REFERENCES The following are considered to be excellent references on the subject of survey execution, experimental design, and associated issues, concepts, and techniques: (1) Campbell, D.T., and J.C. Stanley, Experimental and Quasi-Experimental Design for Research, Rand McNally and Company, Chicago, 1963. (2) Mohr, L., Impact Analysis for Program Evaluations, Brooks-Cole, Monterey, 1988. (3) General Accounting Office, Designing Evaluations, Washington, DC, May 1991. A-19 A-20 EXHIBIT A-2 On-Board Bus Survey KATY TRANSITWAY TRANSIT USER SURVEY This survey is being undertaken by the Texas transportation Institute, the Texas State Department of Highways and Public Transportation and METRO in order to obtain information about your use of the Katy Transitway. Please take a few minutes to answer the question below and return this form to the survey taker before leaving the bus. 1. What is the purpose of your bus trip this morning? _____Work ______School ____Other 2. What is the Zip Code of the area where this trip began? (For example, if this trip began from your home this morning, you would list your home Zip Code._____________ 3. What is your final destination on this trip _____Downtown ___Galleria/City Post Oak/Uptown ___Texas Medical Center ___Greenway Plaza ___Other (specify Zip Code____________) 4. Have you ever carpooled or vanpooled on the transitway? _____Yes, carpooled ____Yes, vanpooled ___No 5. How important was the opening of the Katy Transitway in your decision to ride the bus? ____Very important ____Somewhat important ____Not important 6. If the Katy Transitway had not opened, would you be riding a bus now? ____Yes _____No ____Not sure 7. How many minutes, if any, do you believe this bus presently saves by using the Katy Transitway instead of the regular traffic lanes? _____Minutes in the morning _____Minutes in the evening 8. How long have you been a regular bus rider on the Katy Transitway?______ 9. Does your employer pay for any part of your bus pass? ____Yes, may employer pays $_____toward the cost of my bus pass and I pay $___ ____No I pay the entire amount 10. Was a car (or other vehicle available to you for this trip? (check one) ___No, bus was only practical means ___Yes, but with considerable inconvenience to others ___Yes, but I prefer to take the bus 11. Before you began riding a bus on the Katy Transitway, how did you normally make this trip? (check one) ___Drove alone ___Rode a park-and-ride bus on the regular freeway lanes ___Carpooled ___Rode a regular route or express bus ___Vanpooled ___Did not make this trip prior to using the Katy Transitway ___Other (specify_________________________) 12. Do you feel that the Katy Transitway is, at present being sufficiently utilized to justify the project? ____Yes ____No ___Not sure 13. What is your... Age?____ Sex?___ Occupation?____ 14. What is the last level of school you have completed?_______ Comments:______________________________________________________ _______________________________________________________________ _______________________________________________________________ _______________________________________________________________ THANK YOU FOR YOUR COOPERATION A-21 EXHIBIT A-3 Carpool/Vanpool Survey Click HERE for graphic. Metropolitan Transit Authority 500 Jefferson Street P.O. Box 61429 Houston, Texas 77203-1429 713-739-4000 Dear Carpooler/Vanpooler Your vehicle was observed traveling eastbound of the Katy Transitway the week of September 11. Since you have first -hand knowledge of the transitway, we need your help in a special study being conducted by the Texas Transportation Institute, a transportation research agency of the Texas A & M University System. Because the Katy Transitway is one of the first transitways to operate in Texas, it is extremely important that we determine what effect it has had on your travel. Please take a few minutes to answer the enclosed questionnaire. Your answers will provide valuable information concerning carpooling/vanpooling on the Katy Transityway. Because of the small number of poolers contacted, your specific reply is essential to ensure the success of the project. All information you provide will remain strictly confidential. Your cooperation and timely return of the completed questionnaire in the enclosed postage-paid envelope will be greatly appreciated. Thank you for your time and assistance in this important undertaking. METRO Enclosures A-22 EXHIBIT A-3 (cont.) KATY TRANSITWAY CARPOOL/VANPOOL SURVEY Undertaken by the Texas Transportation Institute, The Texas A & M University System in cooperation with the Texas Department of Highways and Public 1. Is your vehicle a carpool or a vanpool? ___Carpool ___Vanpool 2. What is the primary purpose of your a.m. carpool/vanpool trip? ____Work ____School ____Other 3. How many members are regularly in your carpool/vanpool (including yourself)_____________ 4. Who makes up your carpool/vanpool group? ___Family Members ___Neighborhood friends ____Co-Workers 5. Does your carpool/vanpool use a park-and-ride or park-and-pool lot as a staging area? ___Yes (please specify which lot you typically use_______) ___No 6. How long have you been a regular user of the Katy Transitway?___________ 7. Which transityway entrance do you normally use to access the Katy Transityway in the morning? ___I-1-West of SH6 ____Addicks Park-and-Ride Flyover Ramp 8. What time do you normally enter the transitway in the morning?______a.m. 9. What is your a.m. carpool/ destination? ____Downtown ___Galleria/City Post Oak/Uptown ____Greenway Plaza ___Texas Medical Center ___Other (specify Zip Code______) 10. When did you join your present carpool/vanpool? Month____ Year_____ 11. How important was the Katy Transitway in your decision to carpool/vanpool? ___Very important ___Somewhat important ___Not important 12. If the Katy Transitway had not opened to carpools/vanpools, would you be carpooling/vanpooling new? ___Yes ____No ___Not sure 13. Prior to carpooling/carpooling on the Katy Transitway, how did you normally make this trip? _____On the transitway _____Bus _____Vanpool _____Carpool _____On the Katy Freeway general purpose lanes _____Bus _____Vanpool _____Carpool _____On a parallel street or highway (Street Name ________) _____Bus _____Vanpool _____Carpool _____Did not make this trip 14. How many minutes, if any, do you believe your carpool/vanpool savers by using the Katy Transitway instead of the regular traffic lanes? _____Minutes in the morning _____Minutes in the evening 15. Do you feel that the Katy Transitway is, at present, sufficiently utilized to justify the project? ____Yes ____No ____Not sure 16. What is your..... Age?____ Sex?_____ Occupation?_____ 17. What is the last level of school you have completed?______ 18. What is your home Zip Code?_____________ We would appreciate your additional comments:______________ ___________________________________________________________ ___________________________________________________________ THANK YOU FOR YOUR COOPERATION Please return this form at your earliest convenience in the postage-paid envelope provided A-23 EXHIBIT A-4 Freeway Motorist Survey Click HERE for graphic. Metropolitan Transit Authority 500 Jefferson Street P.O. Box 61429 Houston, Texas 77206-1429 713-739-4000 Dear Motorist; Your vehicle was observed traveling eastbound on the Katy Freeway between 6:00 and 9:00 a.m. the week of October 9. Since you have first-hand knowledge of traffic conditions on the Katy Freeway, we need your help in a special study being conducted by the Texas Transportation Institute, a research agency of the Texas A & M University System. To help serve the travel demand, the State Department of Highways and Public Transportation and the Metropolitan Transit Authority have constructed the Katy Transitway for use by buses, carpools and vanpools. Vehicles using the transitway travel inbound toward downtown in the morning and outbound in the afternoon. The Katy Transitway has been constructed within the median of the freeway and is protected from other traffic by concrete barriers. The location of the transitway in the median has not reduced the number of general traffic lanes available to motorists. Because of the Katy Transitway is one of the first transitways to operate in Texas, we need your help to determine how it is working. Please take a few minutes to answer the enclosed questionnaire. The questions on this survey concern your routing trips made on the Katy Freeway in the morning from 6.00 a.m. to 9.00 a.m. Because of the small number of motorists contacted your specific reply is essential to ensure the success of the project. Your answers will remain strictly confidential. Your cooperation and timely return of the completed questionnaire in the enclosed postage-paid envelope will be greatly appreciated. Thank you for your time and assistance in this important undertaking. METRO Enclosures A-24 EXHIBIT A-4 (cont.) KATY FREEWAY MOTORIST SURVEY 1. What was the purpose of your trip? ___Work ____School ____Other 2. What are your reasons for driving your car on the freeway maintains rather then traveling in a high-occupancy vehicle on the transitway? ___Need car for job ___Car is more convenient and flexible ___No convenient bus, vanpool or carpool available ___Work irregular hours ___Other (specify_________________________________________ 3. How many days per week do you normally make this trip?_____ 4. How do you usually make this trip? ____Drive alone ____Vanpool ___METRO regular route or express bus ____Carpool ____METRO park-and-ride bus ____Other (specify______) 5. How many people (including yourself) were in your vehicle for this trip?_________________________ 6. Which on-ramp did you use to enter the Katy Freeway for this trip?______ 7. What was the destination of you trip? ___Downtown ___Texas Medical Center ___Other (specify Zip Code below) _____________ ___Greenway Plaza ___Galleria/City Post Oak/Uptown 8. Based on your observation of the number of vehicles currently using the Katy Transitway, do you feel that it is being sufficiently utilized? _____Yes ____No _____Not sure 9. Based on you perception of the number of persons currently being moved on the Katy Transitway, do you feel that it is being sufficiently utilized? _____Yes _____No _____Not sure 10. Do you feel that the Katy Tansitway is a good transportation improvement? 11. What is your.... Age?____ Sex?_____ Occupation?______ 12. What is the last level of school you have completed?_______ 13. What is your home Zip Code?________________ We would appreciate your additional comments: _____________ ___________________________________________________________ ___________________________________________________________ ___________________________________________________________ THANK YOU FOR YOUR COOPERATION Please return this form at your earliest convenience in the postage-paid envelope provided. A-25 EXHIBIT A-5 BUS RIDERS SURVEY IF YOU HAVE ALREADY COMPLETED THIS SURVEY, PLEASE RETURN THIS QUESTIONNAIRE TO THE SURVEYOR WITHOUT FILLING IT OUT. The purpose of the following questions is to evaluate Tri-Met’s new fare collection system. Your answers will help Tri-met understand how well the new fare collection system is working for widers like you. Since you are part of a relatively small number of riders being surveyed, your answers are very important to the accuracy of this study. Tri-Met has hired an outside research firm to gather this information. You can be assured that the information you give is confidential, and will only be used in combination with the answers from others riders. We would like you to complete the white part of the survey while on the bus and return it to the surveyor or place it in the box near the rear door. The yellow portion is to be completed as soon as possible and mailed postage free to Tri-Met. THANK YOU FOR YOUR TIME AND HELP. 1. How many bus trips on the average do you usually take each week for each of the following trip purposes? (PLEASE COUNT EACH DIRECTION AS SEPARATE TRIP.) (Write your answer on the line. Put "0" in none NUMBER OF NUMBER OF _______WORK TRIPS _____SCHOOL TRIPS NUMBER OF NUMBER OF _______SHOPPING TRIPS _____SOCIAL/RECREATION TRIPS 2. At what time do you most often ride the bus? (Circle the one number next to your answer.) 1. WEEKDAYS: RUSH HOUR 3. WEEKDAYS: EVENING/NIGHT (7-9 a.m. & 4-8 a.m.) (6 p.m. - 7 a.m.) 2. WEEKDAYS: MID-DAY 4. SATURDAY OR SUNDAY (9 a.m. - 4 p.m.) (ALL DAY) 3. What three bus lines do you ride most often? NUMBER LINE NAME ____ ______________ ____ ______________ ____ ______________ 4. How do you usually pay your fare? (Circle the number under the proper column) CASH BUS TICKET 1 $ .75 (1- or 2- zone) 1 $ 5.00 (1-zone) 2 $ 1.00 (3-zone) 2 $ 6.50 (2.zone) 3 $ 1.25 (All zone) 3 $ 9.00 (3-zone) 4 $ .50 ( Youth) 4 $ 11.00 (All zone) 5 $ .25 (Honored Citizen) 5 24-Hour (All zone) 6 Other 6 Other PASS 1 $23 (1- or 2- zone) 2 $32 (3-zone) 3 $40 (All zone) 4 $15 (Youth) 5 $ 6 (Honored Citizen) 6 Other IF YOU PAY CASH FARES, PLEASE GO TO QUESTION #7 5. Where do you usually buy your pass or bus tickets? (Circle the one number next to your answer.) 1 DRUG STORE 5 PLACE OF WORK 2 7-ELEVEN STORE 6 BY MAIL FROM TRI-MET 3 BANK OR SAVINGS & LOAN OFFICE 7 SCHOOL 4 TRI-MET CUSTOMER ASSISTANCE OFFICE 8 OTHER______________ (PLEASE SPECIFY) 6. Are ticket and pass outlets more or less convenient for you than before self-service fare collection? 1 MORE CONVENIENT 2 SAME 3 LESS CONVENIENT 4 DON’T KNOW 7. How much discount, if any, do you think people should get for purchasing ten-ride tickets in advance? 1 NO DISCOUNT 4 20% (or $ 1.50 on ten 2-zone rides) 2 5% (or 37 on ten 2-zone rides) 5 DON’T KNOW 3 10% (or 75 on ten 2-zone rides) A-26 EXHIBIT A-5 (cont.) 8. Please circle the rating number below which best describes your opinion of the following statements regarding fare collection. STRONGLY UNDECIDED STRONGLY DISAGREE AGREE a. it is a bother to have the 1 2 3 4 5 correct change. b. i don’t like waiting while other people search for their fare. 1 2 3 4 5 c. i am uncertain about time limits and when I should pay extra fare.1 2 3 4 5 d. i’m uncertain about where zone boundaries are and when i should pay extra fare. 1 2 3 4 5 e. I have trouble understanding the information printed by the machine on my ticket. 1 2 3 4 5 8a. What problems, if any, do you have with the method of collecting fares? (Write "none" if you have no problems.) 9. How many times in the last 30 days has your fare been checked by a TRI-MET Fare inspector?________________ 10. Do you think fares should be checked more or less often? 1 MORE OFTEN 2 THE SAME 3 LESS OFTEN 4 DON’T KNOW 11. Do you think more people or fewer people pay the correct fare with self-service fare than with the old method of collecting fares. 1 MORE PAY CORRECT FARES 2 THE SAME 3 FEWER PAY CORRECT FARES 4 DON’T KNOW 12. With the new equipment and rear-door boarding, is getting an and off the bus faster or slower for you than with the old fare collection system? 1 FASTER 2 THE SAME 3 SLOWER 4 DON’T KNOW 13. In general, do you find self-service fare collection more or less confusing than the old method of collecting fares? 1 MORE CONFUSING 2 THE SAME 3 LESS CONFUSING 4 DON’T KNOW 15. Are you: 1 MALE 2 FEMALE 16. What is your age? 1 15 OR UNDER 4 45 TO 64 2 16 TO 24 5 65 OR OVER 3 25 TO 44 17. What was your approximate family income in 1982? 1 UNDER $ 5,000 4 $ 15,000 TO $ 24,999 2 $ 5,000 TO $ 9,999 5 $ 25,000 OR OVER 3 $ 10,000 TO $ 14,999 AGAIN THANK YOU! PLEASE TEAR OFF THE WHITE FORM AND RETURN IT TO THE PERSON WHO GAVE IT TO YOU OR PUT IT IN THE BOX NEAR THE REAR DOOR. PLEASE FILL OUT THE YELLOW FORM AS SOON AS POSSIBLE AND MAIL (POSTAGE-FREE) TO TRI- MET WE APPRECIATE YOUR HELP! A-27 A-28 A-29 A-30 A-31 A-32 A-33 A-34 A-35 EXHIBIT A-10 RECOMMENDATIONS FOR QUESTIONS ON BOARDING AND ALIGHTING POINTS (For User Surveys Only) QUESTION FORMAT 1. "Where did you board this (vehicle)?" __________________________________ Nearest Street Intersection 2. "Where will you (did you) get off this (vehicle)?" ________________________________________________ Nearest Street Intersection RESPONSE CATEGORIES Respondent should specify nearest street intersection. Coders can then translate street address to codes representing bus stops or, if a less fine-grained analysis is required, zonal codes. COMMENTS Question format contains parentheses to indicate where site-specific codes might be substituted. _________________________ *The use of "will you" or "did you" depends on whether the survey is filled out while the respondent is on board the vehicle or completed later and returned by mail. A-36 EXHIBIT A-ll RECOMMENDATIONS FOR QUESTIONS ON TRIP ORIGIN QUESTION FORMAT la. "Where did this trip begin?" ___________________________ Street Address, City, Zip Code 2. "Is this place --- (check one)" . Home . Place of employment . School . Retail/commercial establishment . Social-recreational facility . Medical facility . Personal business site . Other (specify) RESPONSE CATEGORIES Use categories given under "Question Format" or, if the main purpose of the question is to distinguish work vs. nonwork trips, use the following categories: . Home . Place of employment . Other Respondent should specify street address. Coders can then translate street address to zonal codes, or addresses can be decoded using the Census Bureau's TIGER files ant address program. A-37 EXHIBIT A-12 RECOMMENDATIONS FOR QUESTIONS ON TRIP DESTINATION QUESTION FORMAT 1. "What is (was) the final destination of this trip?" _______________________________________________ Street Address, City, Zip Code 2. "Is this place --- (check one)" . Home . Place of employment . School . Retail/commercial establishment . Social-recreational facility . Medical facility . Personal business site . Other (specify) RESPONSE CATEGORIES Use categories given under "Question Format" or, if the main purpose of the question is to distinguish work vs. nonwork trips, use the following categories: . Home . Place of employment . Other Respondents should specify street address. Coders can then translate street addresses to zonal codes, or addresses can be decoded using the Census Bureau's TIGER files ant address progress. Another option, for interview surveys, is to have the interviewer show the respondent a map with numbered zones super-imposed, and ask the respondent to identify the destination zone. COMMENTS The question classifying nature of trip destination, in combination with a question classifying nature of trip origin, is a better indication of trip purpose than a question explicitly asking trip purpose, which can be confusing to persons making multiple-purpose trips. A-38 EXHIBIT A-13 RECOMMENDATIONS FOR QUESTIONS ON TRIP START AND END TIMES QUESTION FORMAT 1. "What time did you begin this trip?" A.M. _________P.M. 2. "What time did you arrive at your destination?" A.M. _________P.M. RESPONSE CATEGORIES Depending on the survey objectives, beginning/ending times can be used as given to compute total trip times, or they can be coded using categories such as A.M. peak, midday, P.M. peak, nighttime. _____________________ * With personal interviews onboard vehicles, it is not possible to ask time of arrival at destination. A-39 EXHIBIT A-14 RECOMMENDATIONS FOR QUESTIONS ON ACCESS MODE TO TRANSIT VEHICLE QUESTION FORMAT 1. "How did you get from the place where this trip began to the (place) where you boarded this (vehicle)?" 2. "How will you (did you) get to your destination after leaving this (vehicle)?"* RESPONSE CATEGORIES U.S. Census Recommended Private auto, driver Park 'n' ride Private auto, passenger Carpool Kiss ‘n’ ride Subway, elevated train, railroad Same walked Same Worked at home Omit Taxi Same Bicycle or motorcycle} Other Other COMMENTS Question format contains several parentheses to indicate where site-specific modes and locations eight be substituted to make the question more relevant. The same principle applies to the recommended response categories; the above list is suggestive and needs to be adjusted to site-specific concerns such as measuring the number of auto passengers for evaluation of a carpool encouragement program. *The use of "will you" or "did you" depends on whether the survey is filled out while the passenger is on board the vehicle or is completed later and returned by mail. A-40 EXHIBIT A-l5 RECOMMENDATIONS FOR QUESTIONS ON WHEN PRESENT MODE WAS FIRST USED QUESTION FORMAT For User Surveys "When did you begin to use (specify services regularly for the trip you are now taking?" . not applicable, or __________month_________year For Non-user Surveys "When was the last time you regularly used (specify service) for the trip you are now taking?" . not within the last 5 years, or ________month________year For Non-user Surveys in Which Carpoolers and Those who Drove Alone are Given Separate Questionnaires For Carpoolers: "When did you begin to regularly use this carpool for the trip you are now taking" . not applicable, or ________month_______year For Those Who Drove Alone: 1. "When was the last time you regularly used (specify service) for the trip you are now taking?" . not within the last 5 years, or _________month________year 2. "when was the last time you regularly used a carpool for the trip you are now taking?" . not within the last 5 years, or _________month__________year A-41 EXHIBIT A-16 RECOMMENDATIONS FOR QUESTIONS ON FORMER TRANSPORTATION MODE QUESTION FORMAT How did you make this trip before (specify service) was available?" RESPONSE CATEGORIES U.S. Census Recommended Private auto, driver Same (indicate total number of occupants) Private auto, passenger Same (indicate total number of occupants) Bus or streetcar Same Subway, elevated train, railroad Same Walked Same Worked at home Omit Taxi Same Bicycle, motorcycle Other Other COMMENTS The responses will have to be tailored to include particular local transportation alternatives. For instance, it might be desirable to obtain information on former auto occupancy levels for ex-drivers/passengers. A-42 EXHIBIT A-117 SET OF ATTITUDINAL QUESTIONS ON TRAVEL BY TRANSIT AND AUTO YOUR OPTIONS OF TRAVEL BY DIFFERENT MODES 4. On the scales below, please indicate your general opinion of car and bus travel for local travel. Base your opinion on what you have experienced or have heard about local travel by each mode from the user’s viewpoint. Even though you may not use the bus, you probably have some perceptions of what this form of travel is like, you don’t need to have tried something in order to be able to express some general opinions. To indicate your opinion, look at the descriptive scales below, each of which allows for a range of opinions on a particular characteristics, such as COMFORT. Then, mark what you consider to be the single most appropriate description on each scale by circling the relevant number. For instance, on the COMFORT scale, if you thought cars were a very comfortable forms of travel for local travel, you would circle "1" on the scale on the line for cars; however, if you thought they were a slightly uncomfortable form of travel, you would circle "4" and so forth. Click HERE for graphic. A-43 EXHIBIT A-18 SET OF QUESTIONS ON GENERAL ATTITUDES OF THE POPULATION YOUR OPINIONS ABOUT LOCAL GOVERNMENT Everyone has different ideas about the kinds of things local government should be most concerned about. Below is a list of different things the government might do. Please indicated your feeling about how much the government should do of each activity. About Much Slightly the Slightly Much Government Activities more more same less less I) Reduced Crime M m s 1 L ii) Reduced environmental pollution M m s 1 L iii) Provide low-cost medical care for all M m s 1 L iv) Control population growth v) Provide more housing for low to medium income families. M m s 1 L vi) Insure equal opportunity for women M m s 1 L vii) Provide a consumer protection M m s 1 L viii) Add to and improve the freeway system M m s 1 L ix) Increase direct aid to the poor M m s 1 L x) Improve bus service and other forms of public transportation M m s 1 L xi) Have more parks and outdoor recreation areas M m s 1 L xii) Improve the public schools xiii) Reduce taxes M m s 1 L (a) Which one of the activities do you feel is the most important for the to do? Just give the letter. Most important______ (b) And which do you feel is the next most important? Next most important______ A-44 Click HERE for graphic. A-45 EXHIBIT A-19 RECOMMENDATIONS FOR QUESTIONS ON RESPONDENTS' SEX QUESTION FORMAT A. For Self-Administered Surveys "Are you -- . Male . Female or "Please indicate your sex" . Male . Female B. For Interview Surveys Respondent’s sex is noted by the interviewer. A-46 EXHIBIT A-20 RECOMMENDATIONS FOR QUESTIONS ON RESPONDENT'S AGE QUESTION FORMAT "To what age group do you belong?" Click HERE for graphic. Click HERE for graphic. COMMENTS The recommended response categories represent the minimum stratification of data to be collected about age. Age responses can be further stratified according to the U.S. Census Categories, depending on the survey objectives and the expected age distribution of the respondent population. It is important to use the phrase "age group" in all questions about age to minimize the respondent’s resistance to this question. A-47 EXHIBIT A-21 RECOMMENDATIONS FOR QUESTIONS ON RESPONDENT'S INCOME QUESTION FORMAT "What is the combined annual income of all members of Your household?" Click HERE for graphic. RESPONSE CATEGORIES (RECOMMENDED) Less than $10,000 $10,000 - $19,999 $20,000 - $29,999 $30,000 - $39,999 $40.000 - $49,999 $50,000 - $59.999 $60,000 - $69,999 $70,000 - $79,999 $80,000 - $89,999 $90.000 - $99,999 $100,000 and greater COMMENTS The recommended response categories represent the minimum stratification of income data. Responses can be further stratified according to the U.S. Census categories, depending on the survey objectives and the expected income distribution of the respondent population. For interview surveys, asking a respondent to point to one of the above categories on a card facilitates handing of this often sensitive question. It is important to use the word "annual" or "yearly" in order to obtain responses on a consistent basis. Moreover, if deemed appropriate, the question can be phrased to refer to the most recently ended calendar year. A-48 EXHIBIT A-22 RECOMMENDATIONS FOR QUESTIONS ON AUTO AVAILABILITY (For User Surveys Only) QUESTION FORMAT "Was a car available to you for this trip?" RESPONSE CATEGORIES "Was a car available....?" . Yes, and without inconvenience to others. . Yes, but with inconvenience to others. . No COMMENTS Information on the availability of a car for a specific trip or time period is the most direct way of determining auto availability and its possible influence on mode used. A-49 EXHIBIT A-23 RECOMMENDATIONS FOR QUESTIONS ON AUTO OWNERSHIP QUESTION FORMAT "How many cars are owned or operated by members of your household?" RESPONSE CATEGORIES U.S. Census Recommended 0 cars . None, or_______auto (s) 1 cars 2 cars 3 or more cars EXHIBIT A-24 RECOMMENDATIONS FOR QUESTIONS ON WHETHER RESPONDENT HAS DRIVER'S LICENSE QUESTION FORMAT "Are you a licensed driver?" RESPONSE CATEGORIES "Are you ...." . Yes . No A-50 EXHIBIT A-25 RECOMMENDATIONS FOR QUESTIONS ON RESPONDENT'S OCCUPATION QUESTION FORMAT 1. "Are you______" . Employed . Student . House Spouse . Retired . Other 2. "If you are employed, describe briefly the kind of work you do." ____________________________________________________ ____________________________________________________ CODING CATEGORIES FOR QUESTION 2 U.S. Census Recommended Professional, technical ant The survey form kindred workers should contain a Managers and administrators, blank space for an except farm open-ended description Salesworkers which can later be Clerical and kindred workers codes using the Craftsmen and kindred workers U.S. Census occupa- Operatives, except transport tional categories. Transport equipment operatives Laborers, except farm Farmers and farm managers Farm laborers and farm foremen Service workers, except private household Private household workers COMMENTS Question 2 should be included in the survey only when there is a very specific reason for using employment data. In order to perform the coding for question 2, it is necessary to obtain description of the type of work actually done as well as job title. A-51 EXHIBIT A-26 RECOMMENDATION'S FOR QUESTIONS ON RESPONDENT'S EDUCATIONAL LEVEL QUESTION FORMAT For All Surveys "What is the last grade (or year) of regular school you (he/she) attended?" (asked for each household member in dwelling unit survey) RESPONSE CATEGORIES U.S. Census Recommended None No Formal Schooling 1-4 5-7 Grade School 8 High School: 1-3 Some High School 4 High School Degree College: 1-3 Some College 4 years or more College degree or higher A-52 EXHIBIT: A-27 RECOMMENDATIONS FOR QUESTIONS ON LENGTH OF RESIDENCE QUESTION FORMAT "When did you (your household) move to your present residence" RESPONSE CATEGORIES U.S. Census Recommended 1989-1990 . Not within the last 5 years, or 1980-1984 _____________month___________year 1970-1979 1960-1969 l959 or earlier COMMENTS The recommended response categories represent the minimum stratification of data to be collected about length of residence. Responses can be further stratified (for greater than 5 years), depending on the survey objectives and the expected residency level distribution of the respondent population. A-53 APPENDIX B STATISTICAL METHODOLOGY This Appendix presents guidelines relevant to determining appropriate sample sizes for data collection as well as the subsequent analyses. The determination of appropriate sample sizes and data analysis requirements is a crucial aspect of planning for data collection, since in general this phase involves scoping the level of activity related to collection of project-specific measures. Just as failure to plan the basic evaluation approach will mean not having the proper framework in which to observe and evaluate the operational test, failure to plan or improper planning of sample size requirements and data analysis procedures will threaten the ultimate statistical validity and usefulness of project results. An insufficient quantity of data, whether due to no planning (i.e., haphazard data collection) or to an underestimate of needs, will be manifested in the loss of potentially valuable analyses and/or a loss in accuracy and validity of the analyses based on the data. On the other hand, excessive quantities of data will mean the unnecessary expenditure of funds and possibly the sacrifice of other data items which could be useful but which are beyond a constrained budget. The intent is to obtain an appropriate balance between analysis requirements and resource availability. It should be remembered that small samples, if they are well planned, can yield useful and interpretable data. B.1 DEFINITIONS To assure a complete understanding of the concepts presented in this Appendix, as well as those identified in the references thereto, the following terms are identified: (1) OBSERVATIONAL ENTITY or ELEMENT - An individual item in a set of items or responses, each of which is identifiable by one or more measures. Examples of observational entities are automobiles, vehicles, persons, time periods. (2) POPULATION or UNIVERSE - A finite or perhaps very large collection of observational entities. A population is usually a group about which inferences are desired. Examples of populations would be all those vehicles on a corridor leading to the central business district during AM peak periods, all those persons within 15 minutes access time of the transit system, or all users of a service. B-1 (3) SAMPLE - A finite subset of observational entities drawn from a population. Samples can be drawn by appropriate procedures which will permit inferences to the population from which the sample was drawn or they may be obtained by non-controlled devices. Examples of samples would be some of the vehicles passing a given screen-line during a specific time period, or a subset of those individuals within a service area. (4) OBSERVATION - One or more measures which describe the observational entities included in the sample either directly or derived from measurements, such as travel times or passenger counts. (5) POPULATION PARAMETER - A specific descriptive characteristic of a population assumed to be constant at any moment or period in time. (6) SAMPLE STATISTIC - A summary value obtained from a sample observation, usually descriptive of the sample but desired for purposes of making inferences about the population or changes in the population parameter. B.2 DATA ANALYSIS DETERMINATION It should be evident that a major intent of using samples is to make inferences about changes in transit system characteristics or in the attitudinal/behavioral characteristics of the community being served. Before estimating sample size requirements, it is necessary to determine the appropriate types of analyses to be performed (i.e., What will be done with the data once they have been collected?). Types of statistical analyses which can be performed are numerous. As a general guideline, it is essential that the evaluations for APTS projects be confined to fairly fundamental types of analyses (i.e., involving the calculation of means, standard deviations or variances, proportions, ratios, and ranges). Suggested statistical techniques for performing these analyses are discussed later in this Appendix. More sophisticated statistical methods, such as multiple regression, factor analysis, and discriminant analysis may also be applicable in the current generation of APTS projects. As more experience is gained with the data collected during these projects, it may be possible to institute some of the referenced multivariate techniques. The use of a simple analytical framework will have three main advantages: (1) the results will be expressed in numerical terms that have a direct relation to specific project B-2 objectives; (2) the evaluation results will be meaningful to a wide audience; and (3) the results of a particular project can be more easily compared with those of other projects. The types of statistical analyses which can be performed and the appropriate equations and tables to be used in performing these analyses and determining sample sizes are presented in an organized, thorough manner in M.G. Natrella, Experimental Statistics, National Bureau of Standards Handbook 91, August, 1963.B-1 Included in this handbook are procedures for estimating average performance from a sample, estimating variability of performance for a sample, comparing two or more samples with respect to average performance or variability of performance, characterizing the functional relationship between two variances, and comparing samples with respect to discrete classifications such as income, mode of travel to work, etc. Two other excellent references are given at the end of this Appendix. Since most of the specific equations to be employed in dealing with these situations are clearly presented in Natrella and other commonly used statistics reference books, the remainder of this section will be devoted primarily to a discussion of some of the statistical considerations by the contractor. Of the numerous cases presented in Natrella, the following basic set of underlying questions is considered applicable for APTS projects: If estimates of population parameters only are required: (1) What is an estimate for the average value (mean) of the measure (Let X represent the measure)? (2) What is an estimate for the variability (variance or standard deviation) of the measure? (3) What is an estimate of the proportion of units that have a given characteristic? If comparisons between two groups (e.g., before vs. after, test vs. control) are involved: (1) What is the difference between the average value of the measure, X, for group A and the average value of the measure, X, for group B? _________________________ [B-1] The contractor is encouraged to obtain a copy of this book, since it is referenced throughout this section of the guidelines as a source for tables, equations and other material. It is available through the Government printing office and was reprinted in 1983. B-3 (2) Same question as (1) except applied to the variability of the measure in groups A and B. (3) Same question as (1) except applied to proportions of some discrete measure in groups A and B. The same types of questions can be asked when there are more than two groups (time periods) involved in the comparisons. Here, however, the methods for analysis become more complex, and greater care must be exercised in selecting and applying statistical techniques. In connection with addressing the question "What is the value...?" or "What is the difference...?", it is recommended that results be given in terms of confidence intervals rather than tests of significance. By presenting a confidence interval (an interval which contains the true parameter, or difference between two parameters, with a known probability), the decision-maker can interpret the magnitude of this interval whether it be for an estimate of a population parameter or for the difference between two parameters. On the other hand, if a test of significance is used, the interpretation of non-significance and significance becomes somewhat more difficult in terms of relating these inferences back to the project objectives. In some instances where sample sizes are fairly large, differences that can be significant from a statistical viewpoint, may have little practical significance attached to them. Statements on statistical significance may be made but the practical implications must be considered. It will generally be adequate for the contractor to report two-sided confidence intervals for a stated confidence level. B.3 SAMPLE SIZE DETERMINATION As long as appropriate sampling methods are applied, the accuracy of a statistic computed from a sample will be greater with a larger sample size. However, this relationship can be one of diminishing returns for very large sample sizes. Moreover, there is a cost, in time and money, which serves as a constraint on sample sizes in each APTS project. The key aspect of sample size determination is finding the proper balance between desired accuracy and cost: on the one hand, the sample should not be so small that the results lack the required accuracy; conversely, the sample should not be wastefully large. B-4 In Section 3.3, variable stratification (the categorization of collected data by such factors as time of day) was discussed. It was mentioned that the data collection activities should be planned with the finest level of stratification consistent with constraints of time, cost, and acceptable accuracy and confidence. It is important that this determination of desired level of stratification be made as early as possible, since, from the statistical point of view, the sampling plans must include sufficient data in each category of interest for which cross-tabulations are to be performed. The formulas for determining sample size must be applied with respect to each category, so that the appropriate quantity of data is collected for each one. Clearly, an attempt at further stratification after the data has been collected would reduce the accuracy and/or confidence associated with these new sub-stratifications. The appropriate sample size formula depends on the type of statistical analysis to be performed. Sample size formulas applicable for calculating means, variances, proportions, etc., are given in the references at the end of this Appendix, so the following discussion will be somewhat general. The sample size calculation process should be viewed as providing input for the broad scoping and planning of the data collection effort. The specific sample size values obtained from the formulas should be taken as rough indications of lower limits for data collection, rather than as precise targets or cut-off points. Prudent expansion factors should be applied to the calculated sample size values so that the ultimate amount of usable data (i.e., the net sample size after the collection activities and editing) is sufficient to yield results with the desired level of precision and statistical accuracy, and allows for unforeseen stratification. As data is collected, it should be possible to modify sample requirements for subsequent phases of a project. As has been mentioned earlier, it is desired to have results presented in the form of confidence intervals. Determining the sample size for calculating a confidence interval requires three input factors: (1) The desired confidence level, (2) An estimate of the variability in the-population, and (3) The desired precision of the results. B-5 The confidence level of a statistical calculation (l-) can be defined as the proportion of samples of size n for which the calculated confidence interval may be expected to contain the true value of the population parameter being estimated. For purposes of obtaining a conservative sample size estimate, it is recommended that the value = .05 be used. An estimate for variability is usually taken as the standard deviation. It is desirable initially for this value to be an overestimate to allow for a conservative determination of sample size. White it is preferable to have some prior knowledge about the variability of those measures to be collected, Natrella (pages 2-8 to 2-10) gives an excellent approach for cases where the true standard deviation is unknown. Determination of an acceptable level of precision is perhaps the most difficult input factor. In the case of estimating means, variability measures, and proportions, the task is to determine the acceptable accuracy, say d, for each confidence interval. The sample size calculated on the basis of a prescribed d and = .05, reflects an acknowledged (permissible) risk that 5 times in 100 the real precision will be worse than d. In the case of estimating the difference between means or between other statistics, the analogous task is to specify the absolute value of a minimum desired detectable average difference . Here, too, if = .05, then the sample size wilt reflect an acknowledged risk that 5 out of 100 times the true difference between the two groups being compared will exceed . In establishing values for d and , consideration must be given to the problem of trading off the cost vs. benefits of increased precision. The cost of increased accuracy can be seen as the marginal amount of time and money needed to collect an additional sample unit. The benefits of increased accuracy can be viewed in terms of additional confidence in the results of a particular project and the consequent willingness of FTA to make policy and funding recommendations on the basis of these results. Clearly, FTA does not want to encourage cities to implement APTS innovations which have only a negligible impact on the quality or usage of transit service; this would argue in favor of setting relatively large values of d and . On the other hand, there is a desire to learn whatever possible about the effects of implementing new techniques; if the minimum detectable difference is set too large, the resultant sample size may be too small to detect the existence of minor, possibly unanticipated changes which might be of interest. B-6 Working with the cognizant FTA and Volpe Center professionals, the contractor should indicate the value of d or selected for each measure to be collected, and should explain the rationale for choosing the particular value in terms of the cost-benefit considerations discussed above. Issues concerning sample size determination and precision are discussed in Sampling Techniques, by W. G. Cochran. B.4 DATA COLLECTION Once the minimum sample size for each stratification category of each sampled measure has been determined using the appropriate formula and the above three prescribed input factors, the data collection phase can be implemented. As was mentioned above, the contractor should apply a prudent expansion factor to the minimum sample size to obtain a target sample size. Field observations should be scheduled for a sufficient number of days to collect the target quantity of sample units. In most cases, the scheduling of data collection will present no particular problems: the required number of "representative" days can be designated, as well as alternate dates to be used in the event of unusual weather conditions or other atypical occurrences on the planned dates. However, there may arise a situation where the day-to-day variability is known or suspected to be significant in relation to the variability within a day. In this case, arbitrary spreading of the data collection phase over several consecutive days may adversely affect the inferences to be made. Depending upon the project objectives, it may be more appropriate to schedule data collection for consecutive weeks on a particular day of the week (the most representative day).B-2 B.5 ANALYSIS METHODS Since numerous statistical methods are available, the balance of this Appendix discusses a family of statistical techniques which will be appropriate for project analyses. The measures which will be collected and utilized to assess achievements of project objectives can be classified _________________________ [B-2] The preceding discussion deals with day-to-day variability with a known pattern. In the unusual situation of day-to-day variability which exceeds within-day variability and does not follow a particular pattern, the target sample size must be calculated according to different procedures, which give a number of sample days as well as number of sample per day. B-7 as discrete or continuous. A discrete measure is one which can assume only a fixed and known set of values. Examples of such measures would be counts of numbers of vehicles and passengers, responses to qualitative questions and classifications of survey responses into categories such as yes/no. Continuous measures may assume (in theory) an infinite set of values. The accuracy of these measures is con
