BTS Navigation Bar

NTL Menu


Advanced Public Transportation Systems: Evaluation Guidelines January 1994



			Advanced Public
			Transportation Systems:
			Evaluation Guidelines
			January 1994

Click HERE for graphic.




			Advanced Public
			Transportation
			Systems:
			Evaluation Guidelines

			Final Report
			January 1994


			Prepared by
			Robert F. Casey and John Collura
			John A. Volpe National
   			   Transportation Systems Center
			Kendall Square
			Cambridge, MA 02142

			Prepared for
			Office of Technical Assistance
			Federal Transit Administration
			400 Seventh Street SW
			Washington, DC 20590


			Distributed in Cooperation with
			Technology Sharing Program
			U.S. Department of Transportation
			Washington, D.C. 20590

			DOT-T-94-1 0


			PREFACE

	This document was prepared by the Office of 
Research and Analysis, Volpe National Transportation 
Systems Center, under the sponsorship of the Advanced 
Public Transportation Systems (APTS) Program, Federal 
Transit Administration (FTA) and with the guidance of Mr. 
Ronald Fisher, FTA's Director of Training, Research, and 
Rural Transportation.  The Volpe Center operates under 
the auspices of DOT's Research and Special Programs 
Administration (RSPA).  The major contributors were Mr. 
Robert Casey, RSPA/Volpe Center Operations Research 
Analyst, and Dr. John Collura, RSPA/Volpe Center Faculty     
Fellow and Professor of Civil Engineering at the 
University of Massachusetts, Amherst.  Technical 
assistance also was provided by Ms. Judith Schwenk and 
Mr. Lawrence Labell of the RSPA/Volpe Center and Dr. 
Thomas Horan of the Institute of Public Policy at George 
Mason University.  The summaries of the breakout sessions 
at the recent National Workshop on APTS Evaluations also 
were useful in the completion of the guidelines.  The 
summaries were prepared by Ms. Katherine Turnbull of the 
Texas Transportation Institute, Mr. John Mason of Science 
Applications International Corporation, Mr. Joel 
Markowitz of the Metropolitan Transportation Commission 
(San Francisco), and Mr. Philip Shucet of Michael Baker 
Jr., Inc.

	The preparation of this document was also 
facilitated by prior evaluation work by Mr. Mark 
Abkowitz, Ms. Carla Heaton, Mr. Chester McCall, Mr. 
Howard S. Slavin, and Mr. Robert Waksman as part of the 
Federal Transit Administration's Service and Methods 
Demonstration program.

	The document consists of evaluation guidelines for 
use by contractors responsible for evaluating APTS 
operational tests.  Although these guidelines are 
intended for the APTS Program, their potential 
applicability extends beyond the evaluation of 
FTA-sponsored operational tests to the evaluation of any 
innovative use of advanced technology in public 
transportation.

	It is anticipated that this document will be 
modified periodically to reflect additional experience 
gained in evaluating APTS operational tests.

iii

	
Click HERE for graphic.


TABLE OF CONTENTS

											
							PAGE

1. OVERVIEW OF EVALUATION GUIDELINES 			1

2. BACKGROUND 						3
	
   2.1 Overview of the Evaluation Process 		7
       2.1.1 Evaluation Frame of Reference 		7
       2.1.2 Evaluation Planning 			12
       2.1.3 Evaluation Implementation 			13
       2.1.4 Potential Evaluation Spin-offs. 		15
   2.2 Coordination of APTS Evaluations 		16

3. GUIDELINES FOR PLANNING EVALUATION ACTIVITIES 	19

   3.1 Determination of Site Data Requirements and 
      Sources 						21

  3.2 Determination of Measures and Collection/
      Derivation Techniques Required to Address 
      APTS; Program Objectives and Other Relevant 
      Project Objectives/Issues 			24

      3.2.1 Basic Set of Measures 			25
            3.2.1.1 APTS Costs and Functional 
                    Characteristics 			27
	    3.2.1.2 User Acceptance 			28
            3.2.1.3 System Efficiency and Effectiveness 28
	    3.2.1.4 Impacts 				29
            3.2.1.5 Relationship Between APTS Program 
                    Objectives and the Categories of
                    Measures				29
	    3.2.1.6 Other Objectives and Measures	34
      3.2.2 Data Collection/Derivation Techniques 	34

  3.3 Planning Considerations Relative to Data 
      Collection  and Analysis 				38
      3.3.1 Basic Data Collection/Analysis Design 	39
      3.3.2 Measure Stratification 			43
	    3.3.2.1 Categorization of a Measure Into 
                    Additive Components 		44
            3.3.2.2 Categorization of a Measure 
                    According to Target Market, 
                    Operational, Geographic, 
                    or Time Categories 			48
     3.3.3 Grouping of Raw Data Into Class Intervals    49
     3.3.4 Sampling Requirements			51
     3.3.5 Timing of Data Collection			52

v


4. GUIDELINES FOR PERFORMING EVALUATION ACTIVITIES	55

  4.1 Monitoring/Performance of Data Collection		55

  4.2 Data Reduction, Analysis, and Presentation	59

5. CONTENT AND ORGANIZATION OF REPORTS			75

  5.1 	Evaluation Plan					75

  5.2 Monthly Evaluation Progress Reports		77

  5.3 Interim Evaluation Reports			78

  5.4 Final Summary Evaluation Report			78

  5.5 Quarterly Project Progress Reports		79

APPENDIX A - SURVEY EXECUTION AND DESIGN		A-1

  A-1 Defining the Survey Universe			A-2
	
  A-2 Sampling the Survey Universe			A-3

  A-3 Techniques for Surveying the Samples Selected	A-7

  A-4 Survey Design Principles				A-10
      A.4.1 Organization				A-11
      A.4.2 Length					A-11
      A.4.3 Question Sequence and Wording		A-12
      A.4.4 Standardized Questions			A-14
            A.4.4.1 Behavioral Measures			A-14
            A.4.4.2 Attitudinal Measures		A-15
            A.4.4.3 Social and Demographic Measures	A-16

  A.5 Non-Response Bias					A-17

  A.6 Interviews With Transportation Agency Personnel	A-18

   A.7 References					A-19
	
vi


APPENDIX B - STATISTICAL METHODOLOGY			B-1

  B.1 Definitions					B-1

  B.2 Data Analysis Determination			B-2

  B.3 Sample Size Determination				B-4

  B.4 Data Collection					B-7

  B.5 Analysis Methods					B-7

  B.6 Methodology Documentation				B-10

  B.7 References					B-10

APPENDIX C-GLOSSARY					C-1

APPENDIX D-BIBLIOGRAPHY					D-1

APPENDIX GENERAL WORK TASKS OF EVALUATION CONTRACTOR	E-1

  E.1 Task 1: Evaluation and Task Administration Plans	E-1
			
  E.2 Task 2: Implementation and Analysis of Data	E-2

  E.3 Task 3: Report Preparation			E-3

vii
    

			LIST OF EXHIBITS

							PAGE

EXHIBIT 1: Selected Examples and Applications of APTS 	 4

EXHIBIT 2: Evaluation Relationships 			 6

EXHIBIT 3: Evaluation Process 				 8

EXHIBIT 4: APTS Operational Test Planning, 
           Implementation, and Evaluation Sequence 
           of and Responsibility for Activities 	17

EXHIBIT 5: Basic Site Data Requirements for APTS 
           Operational Tests 				20

EXHIBIT 6: Typical Sources for Site Data 		22

EXHIBIT 7: APTS Program Objectives and Examples of 
           Corresponding Measures 			31

EXHIBIT 8: Examples of Data Collection Techniques for 
           Selected Measures 				36

EXHIBIT	9: FTA/Section 15 Worksheet for Functional 
           Distribution of Expense Object Classes/
	   Level B 					47

EXHIBIT 10: Service Area for the Seattle Project 	63

EXHIBIT 11: Distribution of Park-and-Ride Users for the 
            Seattle Project 				64

EXHIBIT 12: Passenger Volume for the Seattle Project 	65

EXHIBIT 13: Bus Schedule Adherence for the Minneapolis 
            Urban Corridor Project 			66

EXHIBIT 14: Corridor Demographic Characteristics for 
            the Shirley Highway Express Bus-On-Freeway 
            Project 					67

EXHIBIT 15: Charge-A-Ride Usage by Card Type and Time 
            Period for the Merrimack Valley 
            Charge-A-Ride Program 			68

EXHIBIT 16: Comparison of Fare Payment Times Using 
            Different Methods 				69
EXHIBIT 17: Project Effectiveness Measures for the 
            Seattle  Project 				70

EXHIBIT 18: Highway Travel Time Distributions for the 
            Minneapolis Urban Corridor Project		71

viii


EXHIBIT 19: Results of Before and After Analyses for 
            Portland Self-Service Fare Collection	72

EXHIBIT 20: Benefit-Cost Analysis Results of Salt Lake 
            City Rider Information System 		73

EXHIBIT A-1: Summary of Survey Sampling Methods and 
             Applicable Survey Techniques 		A-20

EXHIBIT A-2: On-Board Bus Survey -- Katy Transitway 
             Transit  User Survey 			A-21

EXHIBIT A-3: Carpool/Vanpool Survey 			A-22

EXHIBIT A-4: Freeway Motorist Survey 			A-24

EXHIBIT A-5: Bus Riders Survey 				A-26

EXHIBIT A-6: Bus Riders Mail-Back Survey 		A-28

EXHIBIT A-7: Bellevue Smart Traveler Project Surveys 	A-30
 
EXHIBIT A-8: Washington, DC, Self-Administered Post 
             Card  Bus Survey 				A-34

EXHIBIT A-9: 1979 Downtown Crossing Bus Passenger 
             Survey 					A-35

EXHIBIT A-10: Recommendations for Questions on Boarding 
              and Alighting Points (for user surveys 
              only) 					A-36

EXHIBIT A-11: Recommendations for Questions on Trip 
              Origin 					A-37

EXHIBIT A-12: Recommendations for Questions on Trip 
              Destination 				A-38

EXHIBIT A-13: Recommendations for Questions on Trip
              Start and End Times 			A-39

EXHIBIT A-14: Recommendations for Questions on Access 
              Mode to Transit Vehicle 			A-40

EXHIBIT A-15: Recommendations for Questions on When 
              Present Mode was First Used 		A-41

EXHIBIT A-16: Recommendations for Questions on Former 
              Transportation Mode 			A-42

EXHIBIT A-17: Set of Attitudinal Questions on Travel by 
               Transit and Auto 			A-43

EXHIBIT A-18: Set of Questions on General Attitudes of 
              the Population 				A-44

ix

	
EXHIBIT A-19: Recommendations for Questions on 
              Respondents' Sex 				A-46   

EXHIBIT A-20: Recommendations for Questions on 
              Respondents' Age 				A-47

EXHIBIT A-21: Recommendations for Questions on 
              Respondents' income 			A-48

EXHIBIT A-22: Recommendations for Questions on Auto 
              Availability 				A-49

EXHIBIT A-23: Recommendations for Questions on Auto 
              Ownership 				A-50

EXHIBIT A-24: Recommendations for Questions on Whether 
              Respondent has a Driver's License 	A-50

EXHIBIT A-25: Recommendations for Questions on 
              Respondents' Occupation  			A-51

EXHIBIT A-26: Recommendations for Questions on 
              Respondents' Educational Level 		A-52

EXHIBIT A-27: Recommendations for Questions on Length of 
               Residence 				A-53
x


		1. OVERVIEW OF EVALUATION GUIDELINES

	This document presents guidelines for planning, 
implementing, and reporting the findings of the 
evaluation of Federal Transit Administration's (FTA) 
Advanced Public Transportation Systems (APTS) operational 
tests.  These evaluation guidelines are intended for use 
by organizations engaged by the Research and Special 
Programs Administration/Volpe National Transportation 
Systems Center (Volpe Center) to evaluate the APTS 
operational tests.  In addition, the guidelines will be 
useful to state and local organizations involved in the 
design and evaluation of Advanced Public Transportation 
Systems.

	An objective of these guidelines is to foster 
consistency of evaluation philosophy and techniques, and 
comparability and transferability of results to improve 
the quality and utility of information obtained from the 
APTS program.  The guidelines are designed to emphasize 
the assessment of the APTS Program's national objectives 
as well as the objectives of the local implementing 
agency.

	The various operational tests implemented under the 
APTS Program are meant to serve as learning tools and/or 
as models for other locales throughout the country.  In 
order for these tests to have maximum effectiveness in 
their respective operational capacities, a consistent, 
carefully structured approach to project evaluation is 
desirable.

	This document has been prepared to provide a common 
framework and methodology for developing and then 
executing the evaluation of individual operational tests.  
These evaluation guidelines are by no means 
comprehensive--that is, they do not offer a suggested or 
preferred course of action for every conceivable 
situation that might arise.  Nor are they to be rigidly 
or blindly followed, since each operational test and each 
site wilt be unique and wilt require somewhat tailor-made 
evaluation procedures.

	It is anticipated that these guidelines wilt be 
modified during the course of the APTS Program to reflect 
experience gained in implementing and monitoring the 
evaluations of individual tests.  Although it is not the 
desire to update these guidelines frequently, 
modifications resulting from field experience wilt be 
made where appropriate for enhancement of performance and 
evaluation of the various projects.

1


	In order to put these guidelines into a meaningful 
context, Chapter 2 provides background information on the 
FTA/APTS Program and the operational test evaluation 
process.  Chapters 3 and 4 present guidelines relative to 
planning and executing operational test evaluations.  
Finally, Chapter 5 presents the recommended content and 
organization for each type of report to be prepared in 
conjunction with the evaluation process.

2


			2. BACKGROUND

	The Federal Transit Administration has developed 
the Advanced Public Transportation Systems (APTS) Program 
which is an integral part of the overall U.S. DOT 
Intelligent Vehicle Highway Systems (IVHS) effort.  A 
major aim of the APTS Program is to promote research and 
development of innovative applications of advanced 
navigation, information, and communication technologies.  
These technologies would be designed and tested to 
achieve APTS Program goals directed toward enhancing the 
ability of public transportation systems to satisfy 
customer needs and contributing to the achievement of 
broader community goals and local objectives.  The APTS 
Program goals and objectives will be discussed further 
within the context of the evaluation frame of reference.

	The wide array of new technologies provides a 
unique opportunity to discover innovative and useful 
applications in public transportation.  These operational 
tests and evaluations will be the principal activities of 
the APTS Program.  Real world testing will be done in 
urban and rural areas using those technologies which 
appear to offer promise and represent useful 
applications.

	Major technologies include automated vehicle 
location systems, smart card systems, dynamic ridesharing 
systems, passenger information systems, high occupancy 
vehicle systems, and vehicle component monitoring 
systems. Exhibit 1 provides selected examples of these 
technologies and associated applications.  Tests will 
involve joint ventures with state and local governments, 
and, when appropriate, universities and private vendors.  
Tests may range from 3 - years: 1-2 years to develop 
implementation plans, 1 year to implement service, and 1 
year to evaluate the APTS application and associated 
impacts.

	In order for the APTS Program to encourage 
significant technological innovations by many urban and 
rural areas, the technologies tested and the results 
obtained must be evaluated, well documented, and widely 
distributed. It is important not only that the 
operational tests be structured and evaluated to 
facilitate transferability of results but also that 
evaluation results be disseminated so that prospective 
beneficiaries in other urban and rural areas are made 
aware of the potential of such technologies. Accordingly, 
a significant element of the APTS Program is the 
technology sharing function.

3


    EXHIBIT 1. SELECTED EXAMPLES AND APPLICATIONS OF APTS

APTS Examples				Applications

. Automated vehicle location (AVL)    . controlling and monitoring 
  system using satellite or ground-     the use of vehicles
  based technologies and compute-
  rized dispatching techniques	      . estimating vehicle 
                                        positions to assist 
				        dispatchers in improv-
					ing  on-street 
					schedule adherence

				      . obtaining boarding and
				        alighting information 
					in conjunction with
					automatic passenger 
					counters (APCs)

				      . assisting in the 
					development of more
					realistic schedules

			              . facilitating the 
					assignment of
					individuals to shared 
					ride, demand response 
					services 
				      . assisting in the 
					preparation of daily 
					driver logs

. Smart card systems using a contact  . facilitating the 
   or contactless plastic card with a	collection of fares, 
   microchip and storage and processing the verification of 
   capabilities	                        travel, and the 
					acquistion of 
					information about 
					passengers and vehicle 
					usage

				      . encouraging the 
					coordination of various
					modes including bus, 
					rail, auto, and 
					parking services

				      . aiding in the 
					establishment of a 
					postpayment fare 
					system and the 
					application of employer
					and human service 
					agency-based subsidy 
					programs

				      . assisting in the 
					design of a 
					comprehensive, 
					historical vehicle 
					maintenance and parts 
					inventory  database

4


			EXHIBIT 1 (continued)

APTS Examples

. Dynamic ridesharing systems using   . providing quick 
  real-time communication methods	and easy access to 
  with the aid of touch-tone telephone, up-to-date information
  television, radio, and videotex       to aid an individual 
  systems		                in arranging a carpool
					or vanpool the same 
					day or evening before 
					a trip

. Passenger information system	      . supplying passengers 
  using audio, visual, and/or hard      with real time infor-    
  copy methods such as digitized        mation on routes,       
  voice, interactive television,        schedules, cancella-       
  videotex, automated map displays,     tions, delays, rerout-    
  computer monitors and printers, and   ing, and other aspects
  other devices located in terminals,   of service to make
  stations, vehicles, places of         travel easier and to
  employment, and at home; also         facilitate intermodal
  could be provided in conjunction      transfers
  with a traffic management center
  (TMC).

. High occupancy vehicle systems      . providing traffic 
  (HOVs) including preferential	        control signal
  treatment methods and park and ride   preemption capabilities
  facilities

				      . monitoring vehicle
					occupancy remotely to 
					enforce HOV lane 
					restrictions

. Vehicle component monitoring	      . assisting in the early
  systems                               detection of problems 
					with vehicle components
					(e.g. engine, exhaust 
					system) to avoid 
					component failure while
			  		vehicle is in 


The exact number, general content, and location of 
the APTS operational tests are yet to be determined.  For 
each fiscal year program, a series of primary objectives 
will be selected, and a group of proposals corresponding 
to each objective, and in keeping with total budgetary 
constraints, will be developed.  Then, following an 
investigation, analysis, and negotiation

5


process involving FTA, the Volpe Center, and candidate 
sites, a final set of operational tests and respective 
sites will be agreed upon.  Once final negotiation and 
transfer of funds between FTA and the APTS local sponsor 
are completed, the operational test can be implemented 
and evaluated.

	As part of its responsibility to evaluate the 
operational tests implemented under the APTS Program, the 
Volpe Center shall engage contractor support to 
participate in all phases of the evaluation process.

	Exhibit 2 shows the interaction among FTA, the 
Volpe Center, the local sponsor, the evaluation 
contractor, and the APTS vendors involved in the 
operational test.

	FTA/APTS staff is responsible for overseeing and 
guiding all aspects of the operational test including 
planning, site selection, negotiations with the site, 
implementation, and evaluation.  The local sponsor is 
responsible for planning and implementing the actual 
conduct of the operational test as well as performing 
most of the data collection.  The Volpe Center assists 
FTA in the activities for which FTA is responsible, and 
directs and monitors the efforts of the evaluation 
contractor.  The Volpe Center, the evaluation contractor, 
and the vendors interface with the local sponsor (or the 
implementing agency, if different from the local 
sponsor). While being directly responsible to the Volpe 
Center for its activities, the evaluation contractor will

6


maintain an informal association and relationship with 
the local sponsor, the APTS vendors, and the cognizant 
FTA Project Manager.  The APTS vendors, as deemed 
appropriate by FTA and the Volpe Center, may participate 
in a review of the evaluation plan, data reduction and 
analysis, and the interim and final reports.  The APTS 
vendors may serve on the local evaluation
review team as discussed in Section 2.2.

2.1 OVERVIEW OF THE EVALUATION PROCESS

	The evaluation process can be thought of 
conceptua11y as a link between the operational tests and 
technology transfer portions of the APTS Program. That 
is, it serves as a bridge between the conduct of an 
operational test at a particular site and the 
understanding of its actual performance at that site as 
well as its potential effectiveness in other locales.  
The quality of the evaluation process directly influences 
the accuracy and perceptiveness of the operational test 
assessment and ultimately affects the applicability and 
transferability of tat findings.

	Exhibit 3 is a flow diagram representing the 
evaluation process for an APTS operational test.  The 
diagram is divided into four major sections: the 
evaluation frame of reference, evaluation planning, 
evaluation implementation, and potential evaluation 
spin-offs.  (The specific organizational responsibilities 
associated untie the various aspects of each APTS test 
are given later in this chapter.)  The first and fourth 
sections can be thought of, respectively, as input to and 
output from the active phases of the evaluation process, 
which are planning and implementation.  A discussion of 
each of the four sections follows.

2.1.1  Evaluation Frame of Reference

	The evaluation frame of reference consists of four 
elements: the operational test application(s); APTS 
Program objectives; external influences; and local 
issues, objectives and site characteristics.

	An APTS operational test will consist of one or 
more technological applications introduced individually 
or sequentially.  For example, a test might include the 
use of a smart card to facilitate automatic fare 
collection.  Another example could consist of an 
automated vehicle location (AVL) system to determine 
vehicle position, followed by the installation of an 
automated passenger counting (APC) system and a 
computerized dispatching and scheduling

7


		EXHIBIT 3: EVALUATION PROCESS*
8


system which work in conjunction with the AVL system.

     Each APTS operational test also is intended to meet the 
goals of the APTS Program which are: 1) to enhance the 
ability of public transportation to satisfy customer 
needs; and 2) to contribute to broader community goals by 
providing information on innovative applications of 
available IVHS technologies.  These goals can be 
translated into the following set of objectives:

Objective #1: Enhance the Quality of On-Street Service to 
              Customers
  . Improve the quality, timeliness, and availability of 
    customer information,
  . Increase the convenience of fare payments within and 
    between modes,
  . Improve safety and security,
  . Reduce passenger travel times, and
  . Enhance opportunities for customer feedback.
 
Objective #2: Improve System Productivity and Job 
              Satisfaction
  . Reduce transit system costs,
  . Improve schedule adherence and incident response,
  . Increase the timeliness and accuracy of operating data 
    for service planning and scheduling,
  . Enhance the response to vehicle and facility failures,
  . Provide integrated information management systems and 
    better management practices, and
  . Reduce worker stress and increase job satisfaction.

Objective #3: Enhance the Contribution of Public 
             Transportation Systems to Overall Community Goals
  . Facilitate the ability to provide discounted fares to 
    special user groups (e.g., disabled persons or 
    employees eligible for tax-free employer subsidies),
  . Improve communication with users having disabilities 
    (e.g., visual or hearing impairments),
  . Enhance the mobility of users with ambulatory 
    disabilities,
  . Increase the extent, scope, and effectiveness of 
    Transportation Demand Management programs,
9


  . Increase the utilization of high occupancy vehicles, 
    with an emphasis on reducing the use of single 
    occupant vehicle, and

  . Assist in achieving regional air quality goals and 
    mandates established in the Clean Air Act Amendments 
    of the Intermodal Surface Transportation Efficiency 
    Act (ISTEA).

Objective #4: Expand the Knowledge Base of Professionals 
              Concerned with APTS Innovations

  . Conduct thorough evaluations of operational tests,
  . Develop an effective information dissemination process,
  . Showcase successful APTS innovations in model 
     operational tests, and
  . Assist system design and integration.

      Objective #1 relates primarily to the riders and their 
desire for improved transit service.  Objective #2, on 
the other hand, deals in part with management aspects 
regarding system costs, service planning, scheduling, and 
operations.  Objective #3 concerns broader impacts in 
terms of the degree to which an APTS application 
contributes to local community goals and national issue 
pertaining to, for example, the special needs of disabled 
persons, congestion management activities, user-side 
subsidy initiatives, energy, air quality, and 
accessibility.  In section 3.2.1, measures are presented 
to examine the level to which these first three 
objectives are attained in each operational test.

         The fourth objective is directed at expanding the 
knowledge base of policy-makers, engineers, planners, 
researchers, and other individuals interested in the 
application of advanced technology to improve public 
transit.  Because this objective is a broader, 
overarching aim of the entire evaluation program, its 
level of achievement will not be assessed using measures 
such as those discussed in section 3.2.1.  Instead, an 
effort will be made to cull information from interim and 
final evaluation reports prepared as part of each 
operational test, and this information will be 
disseminated in publications such as FTA's APTS Briefs, 
IVHS America's Newsletter, and technical journals and 
conference proceedings of other organizations.  In 
addition, selected evaluation results will be summarized 
on electronic bulletin boards commonly available to 
transportation professionals, and results will be 
presented at national and

10


international meetings.  Finally, where appropriate, the 
findings and conclusions of the evaluations will be used 
as a basis for discussion in focus groups, meetings, and 
seminars.

	It should also be emphasized that for any given 
operational test, there may be objectives, over and above 
the APTS Program objectives, which are important 
evaluation considerations.  These might be state or local 
objectives which other participants (e.g. transit 
operator, state transportation agency, community group, 
or local government) are striving to attain (e.g., to 
encourage ridesharing into the downtown area for the 
purposes of reducing parking requirements or traffic 
congestion in the central business district, to preserve 
the stability, cohesion, and authenticity of 
neighborhoods).  The extent to which these state and 
local objectives relate to the APTS program objectives 
should be identified by the contractor.

	The operational test site can consist of anything 
from a corridor in a city to a group of cities or towns, 
and can be at any point along the population and density 
spectrum.  An understanding of the unique demographic, 
economic, geographic, and transportation characteristics 
of the site, as well as prevailing attitudes toward 
transportation, is a useful and necessary adjunct to 
knowledge about the APTS application and associated 
objectives.

	To the maximum extent possible, external influences 
on the project should also be identified and, if 
necessary, appropriate strategies should be designed to 
reduce the likelihood that such influences will have 
adverse effects on the operational test.  For example, if 
the APTS application has radio frequency (RF) spectrum 
requirements, such requirements should be analyzed, and 
political negotiation with authorized communication 
agencies should be initiated as early as possible.

	Information on the planned APTS innovations, 
project objectives, other issues and site 
characteristics, and external influences will generally 
be available from the application submitted to FTA by the 
site prior to approval of the project.  Depending on the 
timing of the evaluation contractor's initial involvement 
in the project, a more detailed description of the 
project may be available in the form of a Project 
Implementation Plan.  Further background on the 
operational test (e.g., genesis of the project concept, 
recent history of transit/para-transit developments at 
the site) can be obtained through discussions with the 
PTA Project Manager, the Volpe Center staff, and the 
local sponsor.

11


			2.1.2 Evaluation Planning

The evaluation planning phase of the evaluation 
process is the period during which the contractor 
interacts with FTA, the Volpe Center, and various 
agencies at the local level to transform the evaluation 
frame of reference into a detailed, structured program 
for conducting the evaluation. This phase sets the stage 
for the entire evaluation effort and, in addition, 
provides an opportunity to reassess and, if necessary, 
restructure the planned operational test.

The planning phase begins with the preparation of 
an Evaluation Strategy for the particular project, which 
describes:

(1) Pertinent information on the APTS application and 
    site (in particular, an indication of what features 
    of the operational test are unique and merit emphasis 
    in the evaluation).

(2) APTS Program objectives addressed by the operational 
    test.

(3) Relevant local, state and/or national objectives and 
    issues addressed (and the relative emphasis to be 
    placed on these objectives vs. APTS objectives).

(4) Key issues to be resolved.

(5) External influences to be addressed.

(6) Recommended scope and focus of the evaluation 
    including a discussion of the APTS costs and 
    functional characteristics and a review of the 
    potential efficiency, effectiveness and other impacts 
    anticipated.

The Evaluation Strategy may be prepared by the Volpe 
Center or the contractor.  The contents of each 
Evaluation Strategy will vary from test to test depending 
on the nature and timing of the project.

	The Evaluation Strategy becomes the basis for the 
more detailed Evaluation Plan1 which is developed by the 
contractor.  While the Volpe Center will provide a 
general evaluation strategy including suggestions 
regarding measures to be used, data to be collected, and 
analytical techniques to be employed, it is generally the 
contractor's responsibility to refine and elaborate on 
the Volpe Center's suggested strategy by developing 
specific procedures for collecting and analyzing data 
relative to project objectives, issues, and the site.

____________________
[1] Chapter 3 presents guidelines relative to the 
evaluation planning phase.  The recommended content and 
 organization of the Evaluation Plan presented in Chapter 5.

12


	In developing the Evaluation Plan, the contractor 
is encouraged to propose changes to the approach 
recommended by the Volpe Center, particularly if the 
proposed modifications have significant potential to 
improve the objectivity, accuracy, completeness, and/or 
efficiency of the project evaluation effort or to enhance 
the transferability of project findings.  In addition, 
total evaluation costs relative to potential findings 
must be borne in mind at all times.  Throughout the 
process of developing the Evaluation Plan, the contractor 
is urged to keep in close contact with the local sponsor 
or project team responsible for implementing and 
operating the test and performing data collection.  This 
continuing liaison with the local sponsor will ensure 
that the proposed methods of data collection are 
consistent with the resources available at the local 
level, with the operational implementation plan developed 
by the site, with important local objectives, and with 
reasonable costs for the evaluation contractor efforts.

	As is apparent from the preceding discussion, the 
evaluation planning phase entails substantial and 
continued interaction among all parties involved in the 
operational test.  Ideally, planning of the evaluation 
effort should be coordinated, and take place concurrently 
with the planning of the project itself.  This 
coordination between the implementation/operation and 
evaluation planning cycles permits optimum flexibility in 
the conduct of the overall test.  Where possible, 
operational aspects of the test will be planned to 
conform to requirements of the evaluation, rather than 
the evaluation having to be integrated into a 
pre-existing, rigid operational structure.  The 
concurrence of the two planning cycles ensures that the 
Evaluation Plan is completed prior to the implementation 
of the project.  Early development of the Plan, in turn, 
allows the necessary lead time for "before" data 
collection -- that is, observations of phenomena such as 
transit system performance prior to the introduction of 
the APTS application(s) as well as possible information 
on community awareness and attitudes prior to project 
implementation. Throughout this phase of the project, it 
is critical to recognize that the FTA Project Manager is 
the final authority in negotiating any operational test 
modifications with the local sponsor.

2.1.3 Evaluation Implementation

	The evaluation implementation phase is the period 
during which the approved Evaluation Plan is executed.  
Activities during this phase include collection/analysis 
of data relative to

13


project objectives and issues, collection/analysis of 
data on site characteristics, compilation of a chronology 
describing the implementation and operation of the test, 
and recording of external factors which might influence 
operational test findings and results.  Contractor 
functions during this phase include monitoring and in 
selected instances, supervising the data collection 
process (generally to be performed by the local sponsor), 
any data collection not performed by the local sponsor, 
data reduction and analysis, subjective analysis of 
information relative to project issues, and synthesis of 
project findings into one or more Interim Evaluation 
Reports and a Final Summary Evaluation Report.2

	This phase not only generates information on which 
the final assessment of the operational test is based but 
also provides feedback information relative to ongoing 
transit operations.  The ongoing evaluation activities, 
while adding to the cumulative body of quantitative and 
qualitative information regarding the project impacts, 
provide interim indications of costs and functions of 
APTS applications and the preliminary effects of these 
applications on transit system efficiency and 
effectiveness.  These interim findings serve as useful 
input to the local agency responsible for implementing 
and operating the test by suggesting the need for 
operational modifications.

	During this phase, modifications may be made to the 
evaluation procedures originally specified in the 
Evaluation Plan.  For instance, examination of interim 
findings may reveal certain gaps or redundancies in the 
originally planned data collection program.  Still other 
reasons for modifying the evaluation procedure might be 
changes in the operational test, unanticipated 
developments or institutional factors at the site, or 
discovery of an improved evaluation procedure.  
Procedural steps to accomplish this necessary update for 
the Evaluation Plan appear in Chapter 5.

	The culmination of the evaluation implementation 
phase is the Final Summary Evaluation Report, which 
presents the following types of findings:
_________________
[2] Chapter 4 presents guidelines relative to the evaluation 
implementation phase.   Chapter 5 gives the 
recommended content and organization of the various 
contractor reports prepared during this phase, 
including the Monthly Evaluation Progress Report, 
the Annual Project Status Summary, the Interim 
Evaluation Report, and Final Summary Evaluation 
Report.  In addition, Chapter 5 describes the 
content of local sponsorís quarterly Project 
Progress Report to FTA, which can serve as useful 
input to the contractorís work.

14


(1) Evaluation of the project in terms of its attainment 
    of relevant APTS Program objectives and other (local 
    and/or national) project objectives.
   
(2) Insight into project issues associated with 
    operational feasibility and  characteristics of the 
    applications.

(3) Assessment of the influence of site-specific 
    characteristics and external factors on the outcome 
    of the operational test.

(4) Lessons learned, based on practical experience, 
    relative to the implementation and operation of the 
    APTS applications (possibly to include 
    recommendations for project modifications in the test 
    site or for future applications in other locales).

(5) Appraisal of the evaluation procedures employed in 
    terms of effectiveness, cost, accuracy, etc.

In essence, this report presents an assessment of the 
impact of the APTS applications at the site and provides 
guidance for the transferability of results to other 
locales.

	The body of the Final Summary Evaluation Report 
includes both narrative and graphic exposition, while 
detailed quantitative data and documentation of 
procedures are provided in technical appendices.  Since 
the report is intended for a variety of audiences -- 
including transportation planners; transit operators; 
federal, state, and local officials; and private industry 
- - it contains an Executive Summary which highlights the 
salient project findings.

2.1.4 Potential Evaluation Spin-Offs

	It is anticipated that each operational test will 
give rise to potential implementation and analytical 
spin-offs.  The Final Summary Evaluation Report, while 
essentially documenting the history and effects of a 
single project, also serves the broader function of 
increasing the understanding of and stimulating the 
application of the demonstrated APTS technologies in 
other localities.  Information presented in the report 
provides a versatile basis for comparing the effects of a 
particular APTS application with those of other similar 
projects, suggesting modifications to the applications 
for future use, and predicting the effectiveness and 
utility of the APTS applications in other cities.  
Moreover, the report's assessment of project evaluation 
procedures can serve as a stimulus for improving the 
state-of-art of evaluation techniques.  Since these 
broader functions of the Final Summary Evaluation Report 
generally materialize after

15


the test period and are not within the purview of the 
evaluation contractor assigned to a particular project, 
they are shown in Exhibit 3 as potential evaluation 
spin-offs.

2.2 COORDINATION OF APTS EVALUATIONS

      Exhibit 4 summarizes the various activities involved in 
planning, implementing, and evaluating an APTS 
operational test and indicates the allocation of 
responsibility for these activities.  The sequence of 
activities ranges from overall APTS Program definition, 
to the operation and evaluation of an individual test, to 
the spin-off uses of the project.  It can be seen that 
the entire stream of activities, especially those 
comprising the evaluation process, involves extensive 
interaction among FTA, the local sponsor, the Volpe 
Center, the evaluation contractor, and the APTS vendors.  
Moreover, it should be noted that the activities shown do 
not always occur in a fixed sequence.  Time constraints 
may require that some of the steps be performed in 
parallel, and there will ideally be considerable 
interaction and feedback between the project planning and 
evaluation planning phases.  The review functions of the 
Volpe Center, the local sponsor, and the APTS vendors 
associated with the data analysis provide a mechanism to 
identify, on a continuing basis, major problems (if any) 
so that APTS operational changes can be made (if 
necessary) during the course of the test. Evaluation 
spin-offs, while arising out of individual tests, will 
result in activities which extend beyond the FTA, Volpe 
Center, local sponsor, and evaluation contractor.

	The diversity of activities and generally long 
(three to four years) time frame for an individual test 
necessitate close and continual coordination among the 
groups involved.  To facilitate communication among local 
test participants and the contractor concerning the 
evaluations, FTA will encourage the establishment of a 
local evaluation review team consisting of 
representatives of transit providers, metropolitan 
planning agencies, human service organizations, 
environmental groups, APTS vendors, and the general 
public.  It may also be appropriate to include faculty 
from local colleges and universities on the evaluation 
review team.  The contractor will meet with- the local 
evaluation review team to discuss the project objectives 
and the emphasis to be placed on each objective in the 
evaluation; to determine the roles and responsibilities 
of all parties involved in the anticipated data 
collection activities; to review problems encountered (if 
any) during the conduct of major data collection 
activities and overall

16


EXHIBIT 4: APTS OPERATIONAL TEST PLANNING, IMPLEMENTATION, AND
EVALUATION: SEQUENCE OF AND RESPONSIBILITY FOR ACTIVITIES

Click HERE for graphic.



KEY:

P = Primary role   M = Monitoring role   R = Review function

a  	Includes local evaluation review team.
b	Local evaluation review team will be established as 
        part of negotiations.
c	Primary role may also be assigned to the 
	contractor.  It may be necessary to have the 
	contractor on-site to monitor the conduct of some 
	data collection efforts such as an on-board survey 
	to ensure that such efforts are carried out properly 
	and that appropriate personnel are available to 
	address unanticipated problems and questions.
d	FTA will disseminate information from these reports, 
	where appropriate.  Such information will appear in 
	FTAís  APTS Briefs, IVHS Americaís Newsletter, 
	professional conference papers, and electronic 
	bulletin boards.  The final evaluation reports 
	themselves will also be published.

17


operational test implementation; to present preliminary 
findings and results of the data analyses; and to seek 
the team's input.  However, equally as important as 
coordination within a particular project is coordination 
across test sites, so as to maximize the effectiveness of 
the APTS Program in encouraging the application of new 
innovations.  This coordination across sites is 
essentially important with respect to the evaluation 
process.  Given the multiplicity of sites, operational 
tests, and participating organizations within the APTS 
Program, there is a strong need for coordination of the 
evaluation process so as to achieve consistency in the 
planning, implementation, and output of individual 
project evaluations.

	With respect to the conduct of the evaluations, 
such coordination will ensure that: (1) the scope of each 
evaluation effort is consistent with the importance of 
that particular APTS test relative to other APTS tests; 
(2) the technical approaches used to evaluate tests are 
consistent with the current state-of-the-art of 
evaluation techniques; (3) common data and definitions 
are employed; and (4) statistical reliability is 
maintained.

	With respect to evaluation output, such 
coordination will ensure that the Final Summary 
Evaluation Reports associated with individual projects 
are consistent in terms of content, format, perspective, 
and level of detail. This consistency in output will, in 
addition, enhance the spin-off potential of the 
evaluations.  The achievement of a basic data set of 
uniform quality across operational tests will make 
possible inter-project comparisons in terms of rider 
characteristics, site characteristics, user acceptance, 
and system efficiency and effectiveness and associated 
criteria.  These types of comparisons will be especially 
significant in the case of multiple applications of a 
particular APTS technology in several locations, or in 
the case of operational tests involving alternative APTS 
technologies directed towards a particular APTS Program 
objective.

	The coordination of the individual evaluation 
efforts will be achieved through the Volpe Center's 
active and continual participation in the program, with 
functions ranging from initial planning of each project 
evaluation effort, to monitoring of the contractor team, 
and finally to the synthesis of individual operational 
tests, evaluation reports and results. This document 
constitutes the first stage of the Volpe Center's 
evaluation coordination function, in that it describes 
general procedures to be followed by each contractor in 
performing the various evaluation tasks speeded in the 
contract.

18


	3. GUIDELINES FOR PLANNING EVALUATION ACTIVITIES

	This chapter presents guidelines for planning the 
evaluation activities associated with a particular APTS 
operational test.  As was mentioned in Chapter 2, the 
evaluation planning phase of the evaluation process is 
that period during which the contractor prepares a 
detailed Evaluation Plan based on the Volpe Center's 
Evaluation Strategy.  The Evaluation Plan contains, among 
other things, a listing of relevant quantitative and 
qualitative measures related to various APTS, local, and 
national objectives and relevant issues, associated data 
collection and analysis procedures, and site specific 
data requirements and sources (both one-time and 
recurring).  As such, the Evaluation Plan constitutes a 
structured, time-phased program for subsequently 
conducting the evaluation.

	The chapter is organized into three sections, 
corresponding to the basic decision-making elements 
shown in Exhibit 3:

  . determination of site data requirements and sources,
  . determination of measures and collection/derivation 
    techniques required to address APTS Program objectives 
    and other relevant objectives/issues, and
  . planning considerations relative to data collection 
    and analysis.

The organization of the chapter is not meant to imply a 
highly ordered time-sequencing of activities, since the 
evaluation planning phase is in fact highly iterative and 
dynamic.  Moreover, it is important to realize that the 
guidelines comprise a basic set of ground rules for 
planning evaluations.  The evaluation contractor will, in 
all probability, need to depart from these guidelines 
during the actual planning phase, so as to conform to the 
unique conditions surrounding a given operational test.

	The contractor should recognize his responsibility 
in working with the local sponsor and the Volpe Center to 
assure that an objective assessment of the project is 
achieved.  One or more site visits during the evaluation 
planning phase is desirable to establish working 
relationships and channels of communication among the 
involved organizations and to uncover any constraints 
which may have a significant bearing on the development 
of the Evaluation Plan.  During this planning effort, 
clarification must be made regarding responsibilities for 
performing and/or

19


EXHIBIT 5: BASIC SITE DATA REQUIREMENTS FOR APTS OPERATIONAL 
           TESTS
 1. Population

 2. Square miles

 3. Population density, persons per square mile

 4. Number of persons in the labor force

 5. Number of households, by type

 6. Age, sex, education, occupation, income distributions

 7. Household auto ownership

 8. Number of persons with no drivers license

 9. Modal split, by trip purpose or time of day if available

10. Existing (Pre-operational test) transit service 
    characteristics

  . Organizational arrangements
  . Route miles (fixed route systems)
  . Tour area (non-faced route systems)
  . In-service vehicles per square mile of service 
    area (non-faced route system)
  . In-service vehicles per hour within service area
  . Time of service operation throughout day
  . Days of service operation throughout year
  . Service frequency (fixed route systems)
  . Fare schedule

11. Description of para-ransit service characteristics

  . Data on taxi operations
  . Information on carpool promotion/matching programs

12. Map of the site showing:

  . The APTS project service area - note that this 
    might be a contiguous area served throughout by 
    the APTS transit system, or it might be two or 
    more non-contiguous areas linked by the APTS 
    service through a travel corridor
 
  . The existing transportation network - major 
    highways, transit lines, commuter rail lines
  . Air quality attainment and non-attainment areas
  . Major topographical features such as rivers
  . The central business district
  . Any other important activity centers

13. Description of relevant site features such as: 
  . Weather conditions
  . Seasonal population variations
  . Institutional/political climate
  . Economic conditions
  . Cost indices (e.g., cost of living index, prevailing 
    transit wage rates)
  . Population/employment growth rate, land use 
    development patterns
  . Residential mobility
  . Air quality conditions concerning ozone, lead, 
    carbon monoxide, PM10, and other environmental concerns

20


overseeing various activities.  The Evaluation Plan 
should indicate the finally agreed upon allocation of 
responsibility between the contractor and local 
evaluation review teams.


3.1 DETERMINATION OF SITE DATA REQUIREMENTS AND SOURCES

	The purpose of the site data is to provide an in-depth   
understanding of those characteristics of the site which 
might in some way influence the outcome of the project or 
the interpretation of project results.  Obviously, the 
APTS operational test will not be implemented in a static 
environment, but rather it will affect the surrounding 
area.  Thus, an examination of certain site 
characteristics is necessary in order to assess fully and 
accurately the impacts of the APTS application.

	An additional function of site data is to enhance 
the comparability and transferability of APTS project 
findings.  Specifically, if conclusions drawn from one 
project are to be compared with findings of other similar 
projects or "transferred" to other potential sites, there 
must exist an objective approach for such a comparison or 
transfer.  This requires the identification of a set of 
site-specific measures which permit one to classify sites 
in terms of meaningful similarities or to identify 
significant areas in which sites differ.  Such measures 
might employ data pertaining to demographic and land use 
attributes, transportation facilities, and vehicle travel 
characteristics, both intra and inter-urban. In addition, 
information on the political/institutional climate of the 
area and prevailing attitudes toward 
transportation-related issues might be helpful in 
anticipating or understanding any problems regarding 
implementation and evaluation of the project.

	A review of past transit project evaluations 
indicates an inconsistency in both the amounts of and 
details concerning reported site-specific data.  To some 
extent this inconsistency reflects a lack of standardized 
site data requirements, but more significantly it 
reflects deficiencies in knowledge regarding the 
interplay between site characteristics and test results.  
In an attempt to shed further light on the subject, a 
basic set of data requirements has been developed for use 
in APTS operational test projects (see Exhibit 5).

	Contractors are encouraged to propose additions or 
deletions to this list, in the context of particular 
projects, if it is felt that the nature and scope of the 
project call for a wider or narrower set of site 
descriptors.  Contractors are also encouraged to propose 
permanent

21


EXHIBIT 6: TYPICAL SOURCES FOR SITE DATA

DATA NEEDED				TYPICAL SOURCES

Demographic			U.S. Bureau of the Census
				City of County Clerk
				State Department of Labor
				State Department of Internal 
				  Revenue
				City or County Planning 
Board

Air Quality			Environmental Protection 
Agency

Land Use Characteristics	City Directories
				Local, Regional and State 
				   Planning Agencies
				Tax Assessor's Records
				Planning Studies

Motor Vehicle Travel		State Highway department 
				   (or State DOT)
				U.S. Census (Journey-to-work)
				Local Traffic Department
				Earlier Travel Surveys
				State Registration Records
				Gasoline Tax Collection Records

Public Transportation Travel	Private Transit-Paratransit 
				   Companies
				Transit Authorities
				State Highway Department  
				   (or State DOT)		
				Local Planning Agency
				U.S. Census (Journey-to-work)
				Earlier Travel Surveys

Travel by Intercity Modes	Federal Agencies such as:
(air, rail, bus)		Federal Aviation Administration
				  Interstate Railroad 
				     Administration
				  Federal Railroad 
				    Administration
				  Department of Commerce
				State Regulatory Agencies
				Earlier Travel Surveys
				Private Carriers

22


additions, deletions, or changes to this minimum list 
based on their cumulative experience in conducting APTS 
evaluations.

	Aside from the site data requirements in Exhibit 5, 
it may be desirable in certain instances to collect a 
standardized set of attitudinal measures to obtain a 
profile of the community.  Examples would be general 
opinions regarding the role of government, environmental 
issues, adequacy of transportation facilities, and 
desirability of travel by alternative modes. Since the 
value of this type of data for evaluation and 
transferability purposes has not yet been fully explored, 
community profile data will be collected only in selected 
operational tests (to be identified by the Volpe Center).  
Appendix A contains sample questionnaires which might be 
used to obtain such data.  As experience is gained in 
this area, a standardized approach to developing an 
attitudinal profile of the test site may be formally 
incorporated into these guidelines.

	It is anticipated that the data set and descriptive 
information shown in Exhibit 5 will be available from 
secondary sources or from the local sponsor and will not 
involve specialized data collection activities (an 
exception being attitudinal profile data, which will 
entail surveys). Exhibit 6 indicates typical sources for 
various categories of site-specific data.3

	Once the contractor has determined the type of site 
data required and the appropriate sources, two decisions 
remain: (1) the geographic scope of the area, and (2) the 
time period (s).

	Regarding the geographic scope, it was indicated 
above that a basic data set should be assembled for the 
APTS service area.4   In some cases, data conforming 
exactly to the service area boundary may be unavailable 
or may be obtained only by aggregation of fine-grained 
data (e.g., Census tract).  If data is available for an 
area approximating the service area, the contractor may 
choose to use this pre-existing data base rather than 
deriving a special data base, provided that such a 
substitution will not be misleading and bias the 
evaluation. O n the other
_______________________________
[3] Adapted from Heaton, Carla; McCall, Chester; and 
    Waksman, Robert;  "Evaluation Guidelines for Service and 
    Methods Demonstration Projects",  USDOT/UMTA-SMD; 
    Washington, DC, 1976.

[4] A definition of the APTS service area may not be 
    available at the outset of the project, but rather will 
    need to be developed during the evaluation implementation 
    phase on the basis of user surveys.

23


hand, the use of fine-grained data may be appropriate if 
the service area is large and heterogeneous and thus 
should be divided into zones.

	The time period(s) for which data is to be assembled 
depends on the time period of the operational test and 
the rate at which conditions at the site are changing.  
If the project spans a fairly long period it may be 
desirable to gather site data for periods before, during, 
and after the project.  In the case of a rapidly changing 
area or a staged project, data for even more points in 
time may be necessary.  Moreover, if an historical 
perspective on the site is deemed relevant to the 
evaluation, it may be desirable to obtain 1980 as well as 
1990 Census figures or recent trend data for key 
variables such as population, employment, and modal 
split.  Since original data collection by the contractor 
is not anticipated, the number and exact timing of site 
data periods will be constrained by the collection cycles 
of existing sources.

3.2 DETERMINATION OF MEASURES AND COLLECTION/DERIVATION 
    TECHNIQUES REQUIRED TO ADDRESS APTS PROGRAM OBJECTIVES AND 
    OTHER RELEVANT PROJECT  OBJECTIVES/ISSUES 

	It was pointed out in Chapter 2 that the Evaluation 
Strategy will set forth a recommended set of APTS Program 
objectives, relevant project objectives (of local and 
national significance), and project issues to be 
examined.  The contractor, in developing the Evaluation 
Plan, is  responsible for reviewing this recommended set 
in the context of the local sponsor's Project 
implementation Plan and the various national and local 
perspectives, and then proposing appropriate 
modifications to the list of objectives and issues.

	Once the set of project objectives and issues has 
been finalized (which involves obtaining concurrence from 
the Volpe Center), the contractor must associate with 
these items a set of germane measures and identify 
suitable techniques to derive each measure and to collect 
necessary data.  It is important to note that certain 
issues may not lend themselves to the use of quantitative 
measures but may rather involve qualitative analysis of 
pertinent information.

	The material presented below is intended to guide the 
contractor in developing appropriate measures and 
associated collection/derivation techniques.  It is 
important to recognize that this material will 
undoubtedly be modified as information is gained through 
the consistent application and analysis of evaluation 
techniques on the operational tests. Therefore, because 
revisions to data program requirements in terms of basic 
data sets, collection and

24


analysis procedures, and presentation techniques can be 
expected, the fundamental value of this section of the 
guidelines lies in the manner in which it structures the 
approach to the selection of measures and the selection 
of techniques for collecting/deriving them.

	In preparing this material, considerable documentation 
was reviewed (see Bibliography).  In addition, direct 
observance and participation in many previous and ongoing 
Federally-funded projects has permitted those preparing 
this document to identify not only a logical structure 
for project evaluation but also to highlight problem 
areas of which all potential project evaluators should be 
aware.  The specific projects which contributed the 
greatest amount of insight were the evaluation plan 
development for the APTS/AVL operational tests and the 
Service and Methods demonstration projects.

3.2.1 Basic Set of Measures
	To assist the evaluation contractor and the local 
evaluation team in the selection of measures to assess 
operational test objectives, six categories of measures 
are suggested:
 
  . APTS costs,
  . APTS functional characteristics,
  . user acceptance,
  . transit system efficiency,
  . transit system effectiveness, and
  . impacts.

The first three categories of measures relate directly to 
the costs, functional aspects, and utility of the APTS 
application and associated equipment.  The next two 
categories pertain to transit system performance in terms 
of actual delivery and usage of the transit services 
provided.  The final category of measures addresses 
project impacts related to critical transportation issues 
and societal goals and concerns.

	While many operational tests will be designed to 
achieve the same (or similar) objectives, some tests 
might be particularly unique in their ability to address 
certain objectives.  Consequently, "priority objectives" 
should be identified in these unique tests, and a 
corresponding set of measures should be formulated so 
that these "priority objectives" are given

25


proper attention, emphasis and evaluation resources.  
Furthermore, the type of measure and the method of 
measurement should be considered as discussed below.

  . Type of measure

    Quantitative -- a measure which is expressed in 
    terms of counts, measurements, dollars, or other physical 
    units

    Qualitative -- a measure which is expressed in terms 
    of people's attitudes, perceptions, or observations

  . Method of obtaining measure

    Collected -- obtained by measurement (vehicle travel time),
    counting number of passengers), surveying (perceived 
    reliability), or from records (daily revenue)

    Derived -- calculated from collected measures either by 
    simple arithmetic procedures (passenger miles per seat 
    mile) or through use of analytic models (reduction in air 
    pollution or fuel consumption)

In reviewing the basic set of measures, it is important 
to note that some of these measures would be more 
meaningful if stratified by time of day (beak versus 
off-peak), location (corridor versus arterial), person 
time segments (waiting, access, transfer, in-vehicle), 
route type (fixed route versus demand responsive), and 
vehicle tour segments (in-service, non-service). Because 
such a classification of measures would have needlessly 
extended the list, the subject of stratification, or 
categorization, with respect to specific data collection 
plans is discussed separately in Section 3.3.2 of this 
chapter.

	The above categories of measures are not to be 
construed as a minimum requirement for every APTS 
project, since an evaluation need only encompass measures 
corresponding to the APTS Program objectives and other 
project objectives/issues addressed by the particular

26


operational test.  Rather, the categories of measures 
should be used by the contractor as a checklist from 
which the most germane measures can be selected and to 
which other relevant measures can be added as 
appropriate.

	It will be noted that for each of the APTS Program 
objectives, it is possible to measure attainment of some 
objectives from two vantage points: the actual and the 
perceived attributes of the transit system (as 
represented by quantitative and qualitative measures, 
respectively).  In the case of transit travel time, it 
might be appropriate to measure actual changes in travel 
time and then to compare with perceived travel time. 
Similarly, in the case of APTS equipment reliability and 
user acceptance measures, comparisons with user 
perceptions and attitudes might also be appropriate.

	Until more is learned about the interrelationship 
between actual measurements and attitudinal data, it is 
not possible to set forth hard and fast rules for when to 
supplement quantitative measures with qualitative 
measures.  Clearly, it may be prohibitively expensive to 
employ this two-pronged procedure for each area of 
interest; on the other hand, mere reliance on 
quantitative measures may result in overlooking what is 
in fact the major behavioral determinant -- people's 
perceptions of the system.  For the time being, the 
contractor must exercise sound judgment in deciding which 
situations are unique and instructive enough to warrant a 
two-pronged data collection effort.  In no case should an 
attitudinal measure ever be used in place of a 
quantitative measure, where both are available.

	The rationale underlying each category of measures 
and their association with operational test objectives is 
discussed in Sections 3.2.1.1 to 3.2.1.6.  Further 
discussion of data collection/derivation techniques 
appears in Section 3.2.2.

3.2.1.1 APTS Costs and Functional Characteristics

	Central to an operational test evaluation is the 
performance of the APTS system and its individual 
components.  Questions surrounding the costs and 
functional characteristics (including reliability, 
usefulness, maintainability, adherence to specifications) 
should be addressed, and the relationship between these 
APTS characteristics and overall operational test 
objectives should be examined.  Examples of such 
questions are:

27


  . What are the life cycle costs (including fixed and 
    recurring expenses) of the APTS system and its 
    individual components? Which are "start-up" costs 
    associated with the newness of the system and might be 
    avoided in future applications?

  . Is the automated vehicle location system easy to use 
    and are vehicle positions determined quickly and 
    accurately so that on-time scheduling can be carried 
    out and that passengers are provided with timely 
    information?

  . Is the smart card system reliable, and does the system 
    meet the required design specifications?

	To the extent possible, the objective (or 
objectives) related to a particular APTS component should 
be clearly articulated and the specific component costs 
and associated functionality should be determined.  This 
will facilitate the comparison of APTS costs and 
associated benefits.  It is recognized, however, that 
individual component costs may be difficult to determine 
if the procurement process allows lump sum bids.

3.2.1.2 User Acceptance

	The extent to which various APTS applications are 
actually utilized will be an extremely important 
dimension of performance in each operational test.  The 
percentages and numbers of riders using a smart card for 
fare payments are just examples of quantitative 
measurements in this category.  In addition, qualitative 
measures of user acceptance (or utility) would be 
employed, examples of which include the attitudes of 
riders regarding the usefulness of AVL-based pre-trip 
information and the perceptions of dispatchers concerning 
the benefits of component monitoring equipment.

3.2.1.3 System Efficiency and  Effectiveness

	Transit system performance is typically viewed in 
terms of efficiency and effectiveness, both of which may 
be influenced by the use of the APTS application and 
other technology.  Efficiency is related to the extent to 
which system inputs such as vehicles, personnel, fuel, 
and funds are employed to produce outputs; examples of 
outputs include the actual number of vehicle miles or 
vehicle hours of service.  For example, reductions in 
unit operating costs would be

28


examined in part with the use of efficiency measures such 
as the operating cost per vehicle mile or operating cost 
per vehicle hour.

	Effectiveness concerns the users and actual demand for 
service and relates to financial aspects such as revenue 
and cost effectiveness service utilization, quality, 
convenience, safety,	security, and reliability.  In 
addition, non-financial aspects of effectiveness include 
service utilization, safety, security, and service 
reliability.

3.2.1.4 Impacts

	To examine the extent to which the operational test 
responds to critical transportation issues and national 
mandates such as the Americans with Disabilities Act, the 
Clean Air Act, and other Federal legislative efforts, 
both quantitative and qualitative impact measures are 
required. Such impacts may be anticipated or 
unanticipated and positive or negative. These impacts 
relate to, for example, the transit agency and its 
internal activities and administrative procedures; 
aspects of human factors; privacy; and matters dealing 
with equity, social, energy, traffic congestion, air 
quality, special mobility needs, institutional and 
political concerns. For example, the use of a smart card 
might facilitate the implementation of a more equitable 
and efficient fare policy as may have been anticipated, 
but it unexpectedly required a reorganization of the 
transit system's finance department and the existing fare 
collection and accounting activities and procedures.  
Another example concerns the use of an automated vehicle 
location (AVL) system which, as intended, may improve 
on-time scheduling; however, such scheduling improvements 
will only be realized after the transit dispatching staff 
has been properly trained and has learned to use the AVL 
system for the purpose of communicating with the bus 
operators.

3.2.1.5 Relationship Between APTS Program Objectives and 
        the Categories of Measures

	While the six categories of measures discussed 
above are not meant to be exhaustive, they do provide 
structure and guidance in the selection of measures to 
evaluate the APTS program objectives #1$ #2$ and #3 to 
the extent that they are associated with each operational 
test.

29


	The first APTS program objective, as stated in 
Section 2.1.1, focuses on enhancing the quality of 
on-street service to riders in terms of safety, security, 
convenience, ease of travel, and travel time.  These 
concerns fall largely under the categories of APTS 
functional characteristics, transit service efficiency 
and effectiveness, user acceptance, and impacts as 
discussed above.  Examples of corresponding measures 
appear in Exhibit 7.

	The second APTS program objective is to improve 
system productivity and job satisfaction.  Anticipated 
system productivity improvements might result from 
reductions in system costs; better schedule adherence; 
quick and effective responses to incidents and vehicle 
and facility failures; and information management systems 
to provide reliable and accurate operating data in a 
timely manner.  Job satisfaction pertains directly to 
another group of potential APTS beneficiaries; that is, 
the employees, such as drivers, dispatchers, and data 
analysts.  An APTS application may lead to a change in 
the day-to-day activities of such employees and may, in 
turn, lead to reductions in worker stress and increases 
in job satisfaction.  Examples of measures to evaluate 
the association of each test with this objective are 
given in Exhibit 7.

	The third APTS program objective centers around the 
contribution of public transportation to larger societal 
issues and community goals.  These issues and goals 
relate to such elements as special mobility needs, 
traffic congestion, air quality, energy, privacy, equity, 
and other concerns. Appropriate measures to assess this 
APTS objective are mainly included in Exhibit 7 under the 
categories of user acceptance, effectiveness, and 
impacts.

	As discussed in Section 2.1.1., the fourth APTS 
program objective is a somewhat broader objective than 
the other three and consequently, the above measures will 
not be used to measure its level of achievement in each 
test. However, as mentioned in Section 2.1.1., to expand 
the knowledge base, results of tests will be disseminated 
in journals, conference proceedings, electronic bulletin 
boards, technical meetings, and seminars.

	Each category of measures includes criteria 
associated with various aspects of APTS applications 
ranging from their costs and functional characteristics 
to their association with overall transit system 
efficiency and effectiveness and other broader societal 
issues, such as air quality, energy, and special mobility 
needs.  The results of each evaluation will be widely 
disseminated as discussed in Chapter 2, so that 
professionals have access to the knowledge they need 
regarding the actual performance of APTS technologies and 
the use of the analytical

30

Click HERE for graphic.

31


Click HERE for graphic.

32


Click HERE for graphic.


33


techniques employed in the analyses.  The availability of 
such knowledge will lead to the design of improved APTS 
applications, in the conduct of more thorough 
evaluations, and the utilization of enhanced evaluation 
analysis tools.

3.2.1.6 Other Objectives and Measures

	The six categories of measures in Exhibit 7 are 
also useful in the selection of measures for other 
operational test objectives.  As pointed out in Section 
2.1.2, there will likely be state or local objectives in 
addition to the APTS program objectives.  For example, a 
state objective might be to reduce the amount of 
financial operating assistance needed.  This would imply 
that either operating costs must decrease or operating 
revenues (e.g., fares) must increase.  Measures 
associated with this objective relate to system 
efficiency and effectiveness.  Another example might be a 
desire to revitalize the central business district.  
Measures for this objective would fall under the area of 
economic concerns in the impacts category.

3.2.2 Data Collection/Derivation Techniques

Once the relevant measures for project evaluation 
have been determined, it is necessary to identify 
appropriate collection or derivation techniques. 
Collected measures can be obtained through the following 
four basic methods:

(1) By measurements, using various instruments, such as 
    stopwatches, odometers, speedometers, and lap-top 
    computers.  The accuracy of the recorded data is a 
    function of the accuracy of the measuring instrument 
    itself. Typical measurements include travel times and 
    vehicle velocities.

(2) By counts or observations involving tallies either 
    from discrete digitized recording equipment, lap-top 
    computers, or manual counts. Typical counts would be 
    numbers of passengers in vehicles.

(3) By surveys or interviews which provide information 
    relative to the individual being questioned, said 
    information to include such items as origin, 
    destination, income level, previous travel modes, 
    observations of how the service is functioning, and 
    attitudes towards transit amenities.

(4) By searching records such as those available through 
    the transit system, local sponsor, and other local 
    planning agencies and Census records.

34


Derived measures can be calculated either through the use 
of simple arithmetic processes or special analytic 
models.  This form of measures builds upon basic data 
collected through some of the above means.  An 
illustration of a simple derived measure might be 
dividing passengers per day by vehicle miles per day to 
obtain passengers per vehicle mile.  Examples of the 
latter type of derived measures resulting from analytic 
models might be the use of a time-delay curve to estimate 
vehicle speeds or the calculation of reductions in fuel 
consumption and air pollution based on a model using 
changes in traffic volumes as input.

	In view of the large number and variety of measures 
in Exhibit 7 and the even larger number which are likely 
to arise during the course of the APTS Program, it would 
be very difficult to specify in these guidelines a 
preferred method of data collection for each measure.  
Moreover, it would be inappropriate to attempt to choose 
a set of "best" methods from among the techniques already 
tried; rather, it is desirable to encourage the continual 
development and implementation of novel techniques with 
potential for increasing the efficiency or accuracy of 
evaluations.  Finally, there is really no requirement for 
uniformity among data collection techniques, but rather 
there is a need for consistency and comparability of the 
data obtained by these collection techniques.  The 
techniques can differ from project to project, as long as 
they are comparable in terms of accuracy and yield data 
in a form suitable for analysis both within the project 
and among projects.

	For the above reasons, it is not the intent here to 
prescribe a standardized approach to data collection.  
However, it is appropriate to discuss the potential 
applicability of some of the specific techniques, drawing 
where possible from previous experience.

	Exhibit 8 illustrates the range of techniques 
employed for selected measures in past transportation 
projects.5   Specific comments on these techniques and 
general recommendations applicable to collecting the 
measures follow:

(1) Travel time, speed, and vehicle volume data collection 
    techniques can range from manual to automatic.  In general,
    automatic techniques are effective only where the 
    magnitude of data requirements or some -other special 
    circumstances warrant their use.  Some of the more 
    sophisticated automatic procedures are subject to 
    reliability problems.  Failure of

____________________

[5] For further details on collecting transit date, see 
    "Review of Data Collection Techniques," prepared by 
    Booz-Allen & Hamilton, Inc. for  FTA, March 1985

35


EXHIBIT 8. EXAMPLES OF DATA COLLECTION TECHNIQUES
FOR SELECTED MEASURES

Travel times for transit vehicles:
  . On-board checkers or on-street checker with stop watches or
    lap-top computers
  . Time referenced equipment connected to bus

Speeds for transit vehicles and autos:
  . On-street checkers with radar units or other equipment
  . Test vehicle with use of odometer, clock, and other 
    equipment
  . Real-time surveillance system with image processing 
    capabilities

Counting auto occupants:
  . On-street counts recorded on paper, counters, or lap-top 
    computers

Counting transit vehicle passengers:
  . On-board checkers or on-street counts recorded on 
    paper, counters, or lap-top computers
  . Bus drivers recording passenger load
  . Automatic Passenger Counters

Travel times for autos:
  . On-street checkers at selected locations recording license 
    plates and times; calculation of elapsed time by matching 
    plates; possibly in conjunction with video camera and image 
    processing technology 
  . Time lapse aerial photographs or video
  . Floating car with observers to record travel time and 
    stopped time delay using stop watches or other equipment

Counting of transit vehicles and autos:
  . Permanent or temporary tube counters or loop detector in 
    lanes or zones of interest
  . Visual counts recorded by persons
  . Time lapse aerial photographs or video
  . Real-time surveillance system with image processing 
    capabilities
  . Electronic detectors

Demographic/behavioral/attitudinal data on users/non-users/
operators:
  . Post cards distributed to auto drivers at exit ramps, to 
    boarding and on-board passengers, and at park-n-ride 
    facilities
  . Forms, usually no longer than one page, distributed and 
    returned by mail or collected on buses
  . Sampling of autos by noting license plates and subsequent 
    identification through Department of Motor Vehicles files; 
    possibly with video camera and image processing technology
  . Interview conducted either at home, work, or within the 
    transit system itself (on board, at stations, etc.) or with 
    transit or local officials

36


    these devices can cause loss of vital data, which 
    will in turn delay the evaluation, and considerably 
    increase costs.  In addition, the measurement 
    accuracy of automatic or semiautomatic devices may 
    be questionable, particularly if they have not been 
    used extensively before.  In cases where definitive 
    information on devise accuracy is not available, it 
    is essential to confirm the accuracy of 
    automatically collected data by periodic use of 
    manual devices.

    Simple manual devices can be deployed so as to 
    maximize utilization of roadside personnel.  For 
    example, in one project, the use of special 
    counters by each observer permitted keeping track 
    of the auto occupancy of each vehicle counted, with 
    the result that two measures were obtained at once.  
    In other projects, special manual devices were used 
    to obtain vehicle counts and occupancy data 
    simultaneously.

(2) Past experience has shown that there is a lack of 
    consistency between passenger counts recorded by 
    transit personnel and counts by on-board or roadside 
    observers.  For instance, in one project, it was 
    found that bus drivers tend to overestimate the 
    passenger load and that on-board and on-street 
    counters tend, on the average, to be consistent with 
    the other.  If transit personnel are to record such 
    data, it is essential that verifications be made 
    during the project to detect any potential bias or 
    unusual variability in this data.

(3) In utilizing transit system records and service area 
    records, such as census data, it is critical to 
    ascertain accuracy of these data. Usually, 
    discussions with personnel who initially record these 
    data will provide an assessment of accuracy.  
    Further, where special data are collected for the 
    project by a local organization, monitoring 
    procedures will be established to assure that no 
    modifications in procedures or notations have 
    occurred which might have an impact on the evaluation 
    process.

(4) Demographic, behavioral, and attitudinal data on 
    users and non-users of the services provided as part 
    of the operational test, as well as attitudinal 
    information from transit operators, can be collected 
    through a wide variety of survey and interview 
    techniques, with varying degrees of respondent 
    cooperation, accuracy, and cost.  In view of the 
    large amount of documented survey experience relating 
    to both transportation and general market research 
    contexts, and in view of the large anticipated role 
    of surveys in APTS evaluations, Appendix A has been 
    devoted to a discussion of survey design and 
    execution.

	In evaluating the array of existing and potentially 
innovative collection techniques relative to a particular 
measure, some of which are included in Exhibit 8 as 
examples, the contractor should consider factors such as 
the cost and accuracy of each method, the availability of 
local resources to implement each method, the ease of 
implementation, and the ultimate data analysis 
requirements.

37


	With respect to cost, the contractor should apply 
sound judgment in determining whether the anticipated 
cost of using a particular technique is justifiable in 
terms of the contribution to the overall project 
evaluation of the specific measure being collected.  
Clearly, the total project expenditure for data 
collection should be allocated among individual measures, 
taking into account each measure's contribution to the 
project evaluation.  The contractor should make special 
note of any data item which is relevant to the evaluation 
but whose collection cost appears to be 
disproportionately high in relation to other items.

	The contractor should determine whether the 
accuracy of a particular technique is consistent with the 
accuracy requirement for the measure, which in turn is 
dependent on the relative importance of the measure.  A 
very accurate technique is probably not warranted for a 
relatively insignificant measure, especially if that 
technique would be expensive to implement.  In addition, 
a high degree of accuracy for some measures may be 
inconsistent with a lesser degree of accuracy for others.  
The contractor should also evaluate alternative 
techniques in light of the available local resources--
labor resources as well as equipment.  An attempt should 
be made to utilize existing equipment or rental equipment 
arrangements wherever feasible, rather than opting for 
techniques which require the purchase of new equipment 
(which might not be needed by the locality after the APTS 
evaluation).

	The contractor's Evaluation Plan should contain 
justification for selecting the particular technique 
applicable to each measure in terms of these 
considerations.  In the case of a novel technique, it is 
required that the contractor demonstrate acceptable 
accuracy before it can be used as the sole source for 
data collection.  It is further required that the 
evaluation contractor document his experience with those 
data collection methods employed in an evaluation, as 
explained below in Chapter IV.  As this further 
experience develops, the Volpe Center will make this 
information available via updates to this Guidelines 
document.

3.3 PLANNING CONSIDERATIONS RELATIVE TO DATA COLLECTION 
    AND ANALYSIS

	The preceding section contained-guidelines relative 
to specifying appropriate measures and 
collection/derivation techniques for addressing APTS 
Program objectives and other project objectives and 
issues.  This section completes the discussion of 
evaluation planning activities with general guidelines 
for data collection and analysis procedures.  The 
material in this section,

38


while intended to be applied to individual measures 
selected for inclusion in the evaluation, is presented in 
a general context.  The following topics are included: 
basic data collection/analysis design, measure 
stratification, sampling requirements, and the timing of 
data collection.

3.3.1 Basic Data Collection/Analysis Design

	A significant aspect of the evaluation process for 
APTS operational tests is determining the basic data 
collection and analysis design to be employed relative to 
specific project objectives.  There are a great variety 
of potential design approaches, ranging from an 
"after-only" design (a one-shot case study approach 
involving a single set of measurements taken after the 
project is operational) to a "before-after with control 
group" design (involving a comparison of multiple 
measurements).  A General Accounting Office (1991) Report 
entitled, "Design Evaluations," presents guidelines with 
the use of a "decision tree" to assist in the selection 
of an evaluation design including case studies, 
cross-section or panel surveys, comparative group 
analyses, or a before and after study.  A comprehensive 
discussion of the specific utility and the relative pros 
and cons of the various design approaches can be found in 
Donald T. Campbell and Julian C. Stanley, Experimental 
and Quasi-Experimental Designs for Research, 1968, and L. 
Mohr, Impact Analysis for Program Evaluation, 1988.  The 
information which follows is intended to discuss the 
relative advantages of various approaches in the context 
of the APTS program and to highlight the major 
considerations involved in selecting the appropriate 
design for each APTS evaluation, or for individual 
measures included in the evaluations.

	In general, a single set of measurements (for 
example, taken while the test is in operation) will be 
insufficient for assessing the impact of the test, since 
it will not provide any yardstick with which to interpret 
the measurements.  It is recommended, therefore, that 
every data  collection/analysis program be structured 
around some form of comparison.  If such an approach is 
for some reason infeasible, the contractor must indicate 
the reason(s) in the Evaluation Plan.

	Given that the basic data collection/analysis 
design will generally be in the form of a comparison of 
multiple measurements, the next question to be considered 
is what types of comparison are appropriate.  The two 
main forms of comparison are before vs. after and test

39


vs. control.  In a before-after comparison, a given 
measure is collected on a given system element before the 
experimental or exemplary operational test technique is 
instituted and then again while the technique is 
operational.6 In a test-control comparison, a given 
measure is collected on a system element which has been 
affected by the introduction of a technique (test unit) 
and also on an equivalent system element which has not 
been similarly treated (control unit).  Each type of 
comparison is somewhat limited: the before-after 
comparison fails to show what portion of the change in 
the measure is due to external factors; the test-control 
comparison shows the difference between "after" measures 
and hence accounts for external factors, but fails to 
indicate the degree of change from the before state to 
the after state.  Accordingly, it is desirable, where 
feasible, to conduct a before-after comparison in 
conjunction with a test-control comparison.  In other 
words, the data design should, if possible, involve 
collection/analysis observation of both a control and 
test unit before and after the institution of the APTS 
application.

	To make the foregoing discussion more concrete, 
consider a large area with many bus routes and suppose 
that a certain fraction of them are treated in some 
manner (i.e., an APTS application is implemented which 
can be expected to reduce bus travel time).  If 
pre-application and post-application measures of travel 
time are made only on the treated routes and a reduction 
in time is indicated, there is no way of knowing the 
extent to which the improvement is attributable to 
external factors (for instance, a decrease in auto 
traffic on the streets where the buses operate).  In 
order to account for, in a quantitative fashion, these 
known or unknown factors which have arisen during the 
interval between the before and after measurements, it is 
necessary to make before and after measurements of bus 
travel time on routes which are comparable to the test 
routes and therefore susceptible to the same set of 
external factors.  The difference between the travel time 
reduction on the test vs. control routes can then be 
taken as the true change due to the application.  To make 
these statements, it is necessary to be fairly

___________________
 
[6]  As is discussed below, a before-after comparison does not 
     necessarily imply a single measurement before the 
     operational test is implemented and another measurement 
     while it is operation.  Rather, this type of comparison 
     can take the form of a series of measurements prior 
     to, during, and after the operational phase of the 
     operational test.  If the project is implemented is 
     stages, there will be a series of measurements 
     corresponding to each stage.
40


confident that conditions affecting both control and 
experimental units are reasonably similar a requirement 
which is sometimes difficult, if not impossible, to 
assure.

	To reiterate, the proper use of the combined 
before-after/test-control approach guarantees to the 
greatest extent that any observed improvement is indeed 
due to an operational test application.  Thus, the 
contractor should employ both types of comparisons 
wherever appropriate and feasible.  The determination of 
appropriateness of the combined approach involves a
consideration of the time span of the operational test.

	Regarding the scope of the project, the larger the 
geographic area encompassed by or affected by the 
project, the greater the possibility that no control 
units can be identified (i.e., the entire population is 
composed of test units).

	Regarding the time span of the project, no 
generalizations can be made since tests will vary in 
length depending on a variety of factors.  As a general 
rule-of-thumb, the desirability of the combined 
before-after/test-control approach increases with the 
time span of the project, since this approach reveals 
internal as well as external changes occurring over the 
project's duration.  The determination of feasibility of 
the combined approach involves questions of data 
availability and project timing.  If there is a known 
deficiency in either type of comparison, then only the 
valid comparison should be employed; it is generally 
better to do without a before observation or a control 
observation than to settle for unsuitable before or 
control data.  In the event that only one type of 
comparison is feasible, there are alternative techniques 
and precautionary measures available to the contractor to 
compensate for the absence of the other type of 
comparison.

	If no control group exists (e.g., if the 
operational test affects the entire population of 
observation units, making each one a test-unit) or if no 
suitable group can be found (each test unit is unique), 
then the contractor should be especially observant 
throughout the evaluation period of possible external 
factors which might influence the interpretation of 
project results.  Any statistics regarding the before vs. 
after change due to the applied technique should be 
examined very carefully in the context of these observed 
external factors, and any conclusions based on such 
statistics should be qualified accordingly.

	If, due to project timing, there is no opportunity to 
perform before measurements, or if it is known beforehand 
that the units to be observed will undergo considerable 
change between

4l


the before and after periods, the contractor should 
attempt to obtain surrogate data for the before period.  
Possible sources of surrogate data would include: (1) 
surveys conducted after the test is operational which 
question people about conditions or their behavior prior 
to the implementation of the technique; and (2) 
demographic and travel data collected by the local 
highway department, planning agency, or transit operator 
some time prior to the operational test.  The surrogate 
data can be used to provide some indication of the 
magnitude of the before-after change experienced by the 
test and control groups.

	In using the before-after and/or test-control 
approach, one of the key steps is identifying comparable 
units.  To as great an extent as possible, the units 
observed for the before case must be equivalent to the 
units observed for the after units.  Returning to the 
previous example of bus routes, before-after 
comparability is not a difficult problem, since the same 
routes can be observed for both time periods.  The only 
note of caution is that the routes should be unchanged 
(with respect to length, number and location of stops, 
etc.)  from one measurement period to the next.

	Test-control comparability, on the other hand, 
raises some interesting problems.  Theoretically, the 
test and control units should be as nearly alike as 
possible to rule out any chance of the observed change 
being a result of something other than the operational 
test application.  Test and control units should be 
chosen which are similar in terms of variables assumed to 
be related to the particular measure.  Again, using the 
example of bus routes and the measure travel time, 
matching of test and control routes could be done on the 
basis of such descriptors as route length, total trips 
along the route, peak headway, and average speed.  The 
Volpe Center's Evaluation Strategy will generally suggest 
the basic data  collection/analysis design to be employed 
for each project as a whole or for particular measures 
(e.g., before-after comparison, test-control comparison, 
both types of comparison, or a single set of 
measurements.  The contractor should determine the 
feasibility of such suggestions in terms of the data 
availability and time frame of the particular project and 
site.  The contractor's Evaluation Plan should then 
elaborate on the approach finally selected for each 
measure, indicating information such as the specific 
units chosen for the control and test groups.
42


3.3.2 Measure Stratification

	Measure stratification refers to the categorization 
of individual measures for collection/derivation and/or 
analysis purposes.  Examples of measure stratification 
are:

(1) peak versus off-peak time periods,
(2) day of the week,
(3) revenue (in-service) versus non-revenue service,
(4) waiting, access, transfer and in-vehicle travel times, and 
(5) fixed route versus demand responsive.

Measure stratification improves the quality of the 
evaluation by allowing an assessment of how changes in 
measures relate to the stratification categories, hence 
facilitating the formulation of more specific findings 
and conclusions.

	Whereas collection of an unstratified measure 
provides only a single, average reference point, the use 
of a stratified measure provides a series of reference 
points, each of which may be significant to the analysis 
and interpretation of results.  Knowledge of 
inter-category differences in results enhances 
transferability; for instance, if a particular 
operational test proves to be especially beneficial in 
congested areas but of limited value in sparsely traveled 
areas, then other sites considering implementation of the 
service will know to focus their efforts in congested 
areas.

	Stratification can take the following forms:

(1) categorization of a measure into additive components 
    (e.g., measuring person trip time in terms of trip 
    components such as access time, line-haul time);

(2) categorization of a measure, and possibly its components, 
    according to target market, operational, geographic, or 
    time categories (e.g., measuring trip time for peak and 
    off-peak periods); and

(3) grouping of raw values of a measure into class intervals, 
    with class intervals determined either before or after data
    collection (e.g., determining the distribution of early, 
    late, and on-time arrivals).

	It is not possible apriori to present a 
standardized approach to be used for each measure.  
Clearly, the appropriate type and level of stratification 
depend on the particular measure and on the 
characteristics of the site and project.  However, in 
order to provide the contractor with

43


some guidance in this area, examples of possible types 
and levels of stratification are presented below.

3.3.2.1 Categorization of a Measure Into Additive Components

	This form of stratification involves collecting and 
reporting data separately for specific components, or 
sub-breakdowns, of a measure.  The purpose of 
categorizing in this manner is to single out the effect 
of an APTS application on these specific components.  
Examples of this form of stratification are available for 
measures relating to travel time, reliability, and 
productivity.

Person transit trip time for fixed route systems 
can be broken into segments as depicted in the following 
diagram:

Origin                                        Destination



where:

  Segment A = Access time
  Segment W = Waiting time for first vehicle or for subsequent 
                      transfer
  Segment T = In-vehicle transit time
  Segment E = Egress time
  ti        = Time for ith trip segment

	If further amplification is desired, access time 
and egress time can be subdivided into walking, riding, 
and other portions; or in-vehicle transit time can be 
subdivided into collection, line-haul, and distribution 
phases.

	In the case of demand-responsive systems, some of 
the trip time components might take	on a different 
definition: for example, access time would be zero, and 
waiting time would refer to the difference between the 
caller's requested time of pick up and the arrival time 
of the vehicle at the origin. In cases where the caller 
is told that pick up can only be made later than

44


the requested time,7 wait time can be further divided 
into the time between the requested pick-up time and the 
promised pick-up time, and the time between the promised 
pick-up time and the arrival time of the vehicle at the 
origin.  This latter travel time component, is, in 
itself, a basic transit system reliability measure in the 
category of effectiveness measures summarized in Exhibit 
7.  In-vehicle transit time, if desired, can be divided 
into the direct routing travel time (the time between the 
person's origin and destination if no other pick-ups or 
drop-offs are made) and the detour travel time (the time 
spent detouring to make other pick-ups and drop-offs).

Transit vehicle time is always to be broken into 
in-service time and non-service time.  However, if 
desired, these two prime categories can be further 
divided as indicated below.

For fixed route systems:

	In-service
		In motion
		Loading
		Non-productive -- waiting for lights, metering, 
		                  or other obstacles to motion

	Non-service
		Garage to first service point
		Last service point to garage
		Dead turnaround time
		Deadhead time
		Other

	For demand responsive systems:

	In-service
		In motion with one or more passengers onboard
	   	In motion with no passengers onboard and in the 
		   act of picking up one or   more passengers
  		Loading

	Non-service
   		Garage to first pick-up point
   		Last drop-off point to garage
   		Between first pick-up point and last drop-off 
		  point with no passengers onboard and not in 
		  the act of picking up one or more passengers
________________________
[7]  Due to the potential ambiguity associated with requests for
     immediate service, the contractor should note how the 
     particular transit operator maintains data on requested 
     and promised pick-up times.

45


These time segments are depicted in the following 
diagram:


where:
Point A = Garage
      B = First pick-up point
      C = Drop off point -- no passengers on vehicle but 
	                    driver is instructed to proceed 
                            immediately to pick up a passenger

      D = Pick-up point
      E = Drop-off point -- no passengers on vehicle and 
			    there are no requests for 
			    immediate pick-up; driver is 
                            instructed to proceed to 
			    central waiting point

     F = Point enroute to central waiting point-- driver is
                                                  instructed 
						  to proceed 
						  immediately 
						  to  pick 
						  up a passenger

    G = Pick-up point
    H = Last drop-off point of day
    I = Garage

Note that in segments BC and GH pick-ups and drop-offs 
are being made and at least one passenger is always 
onboard.  Also, all pick-up and drop-off points include 
time spent waiting for riders to board and deboard 
vehicles.


For operating costs of APTS operational tests, it 
has been decided that the aggregation of cost items 
should be consistent with FTA Section 15 expense 
categories.  Exhibit 9 is a matrix showing the 
distribution of expense object classes into functional 
areas under Section 15.

46


Click HERE for graphic.

47


	Because of possible differences in current internal 
accounting practices, it is essential that any techniques 
for disaggregation and allocation of costs be described 
in the Evaluation Plan.  In addition, because of 
different funding mechanisms, it is important to review 
in depth individual transit authority practices.  It is 
also recognized that the reporting of operating costs 
should be carried out using a consistent time frame for 
reporting periods.

3.3.2.2 Categorization of a Measure According to Target Market, 
        Operational, Geographic, or Time Categories

       The primary purpose of this form of stratification is to 
evaluate the effect of APTS applications in different 
contexts.  As in the case of categorization into additive 
components, this form of stratification involves 
collecting and reporting measures separately for each 
category.	Examples are as follows:

	Target Market:
		Trip purpose -- work/non-work
		User group -- commuters/non-commuters
		Mode -- auto/transit/other

	Operational:
		Type of transit service -- express/local; fixed 
		  route/demand responsive
		Direction of traffic flow - inbound/outbound
		Type of thoroughfare -- freeway/arterial

	Geographic:
		Within/outside central business district
		Zones with different demographic characteristics

	Time:
		peak/off-peak
		weekday/week-end

	Finer stratification in the above examples is also 
possible.  For instance, within the target market 
category, the trip purpose "non-work" can be divided into 
medical, social, recreational, etc.; non-commuter can be 
stratified into elderly, disabled (ambulatory and 
non-ambulatory) unemployed, etc.; and mode can be divided 
into solo driver auto, carpool auto, chauffeured auto, 
and specific local transit service options.  Types of bus 
service can be divided into local feeder, local 
line-haul, and express line-haul, and further divided 
into individual routes, and

48


beyond that into route segments.  Time of day can be 
refined into the four Section 15 categories (A.M. peak, 
midday, P.M. peak, night) or even further into hour, 
half-hour, or 15-minute segments within certain 
categories.

	In general, in some instances it will be desirable to 
partition collected data into various target market 
categories, since most operational tests will probably 
consist of specific innovations aimed at particular user 
groups.  The decision as to whether to stratify collected 
data by operational and geographic categories depends on 
the nature of the project and thus will have to be made 
on a case-by-case basis.  However, it is recommended that 
serious consideration be given to using a minimum time of 
day stratification (peak/off-peak) for every measure, 
since many transit system operating characteristics as 
well as general traffic conditions vary widely between 
peak and off-peak periods.  The decision as to 
stratification of data collection within the peak period 
(i.e., morning vs. evening peak) and within the off-peak 
period (i.e., midday vs. nighttime) should be made in 
accordance with the time of APTS service operation 
throughout the day and the variability of travel 
conditions and other relevant factors between the 
different categories.  It is important to note that the 
peak period may be a changing period depending upon 
distance from the CBD and type of transit system. Other 
issues regarding data stratification and analysis are 
discussed in Section 4.2.

3.3.3 Grouping of Raw Data Into Class Intervals

	Measure stratification can also refer to the 
grouping of raw data into intervals, with intervals 
determined before or after data collection. Whereas the 
first two forms of stratification involve collecting and 
reporting a measure separately for each category (e.g., 
change in travel time during peak periods, off-peak 
periods), this type of grouping produces a frequency 
distribution for the particular measure.

	Survey data on traveler behavior, characteristics 
and attitudes is a good example of pre-collection 
determination of intervals.  For instance, comparisons of 
users and non-users of an APTS test can be made using 
distributions of-such measures as age, income, auto 
availability, and attitudes toward transit, with the 
particular response categories of each measure having 
been determined beforehand.  Appendix A contains 
recommended response categories for selected

49


demographic and travel behavior measures, as well as 
sample questions and response categories for selected 
attitudinal data.

	Reliability measures provide examples of intervals 
that can be determined after data collection.  The 
difference between scheduled and actual arrival time at 
an access point would be collected in its raw form (i.e., 
each vehicle's time difference in minutes), but would be 
reported as a frequency distribution. A suggested minimum 
stratification of this measure is:

	% early
	% on time (vehicles arriving within +x or -y 
	  minutes of scheduled time) 
	% late

The contractor should be aware of differences in transit 
company standards with respect to schedule adherence, and 
the potential impact on data collection and analysis 
procedures.

	Vehicle delays due to breakdowns can be grouped 
according to the following minimum stratification:

	% No delay (delay of 2 minutes or less) 
	% Delayed
	% 96 Total disruption of service

	If further detail is desired, the late category under 
schedule adherence and the delayed category under vehicle 
reliability can be divided into categories such as: 1-5 
minutes delay, 6-10 minutes delay, over 10 minutes delay.

	The basic intent of grouping is to summarize the 
raw data without masking the real form of the 
distribution for a given measure.  In addition, the 
extent of grouping may also depend upon the specific 
analyses which are planned.

	Interval grouping can be used in conjunction with 
either of the two forms of stratification previously 
discussed.  For instance, person trip time can be 
stratified into components (access time, etc.), and time 
period (peak vs. off-peak) can be grouped into 5 or 10 
minute intervals to obtain a frequency distribution.

	As was stated above, it is not possible in these 
guidelines to present a standardized approach to 
stratification for each measure.  The contractor will 
therefore have to rely on judgment and past experience to 
determine which types of variable stratification are most 
likely to enhance understanding of specific areas of 
project effectiveness and potential application.

50


The contractor should plan data collection activities 
with the finest stratification which can be justified as 
appropriate for the APTS objectives.  Since the ultimate 
sample size will be directly related to the number of 
categories employed, the contractor should make sure that 
the available sample units are sufficient to support the 
level of stratification deemed desirable.  The Evaluation 
Plan developed by the contractor should contain 
justification for the type(s) and level of stratification 
selected, as well as evidence that such stratifications 
are feasible from the standpoint of data and sample size 
availability.

3.3.4 Sampling Requirements

	Once the contractor has determined the basic data 
collection/analysis design for the project evaluation and 
the type(s) and level of stratification for each measure, 
the final question to be addressed is sampling 
requirements.

	In general, data required from records maintained 
by the transit operator or other organizations should be 
available on a continual basis over the entire lifetime 
of the experimental test and such data should not require 
sampling.  On the other hand, data obtained from 
measurements, counts, and surveys will generally not be 
available on a continual basis but will have to be 
collected in the form of samples.  There may also be 
situations where measurements or counts yield continual 
data, but sampling is desired in order to reduce data 
processing expenses.

	When collection of a particular measure involves 
sampling, an estimate of the minimum sample size must be 
made prior to the initiation of the data collection 
effort.  In estimating sample size requirements, the 
objective is to have a large enough sample to be able to 
draw valid inferences about the population from which the 
sample is drawn.  As might be expected, the determination 
of appropriate sample sizes involves trade-offs between 
the desired level of precision and the cost of data 
collection.  These trade-off decisions in turn require a 
determination, during the evaluation planning phase, of 
the appropriate types of analyses to be performed 
(e.g.,-estimates of population parameters, 
comparisons-between two or more groups of sampled data).

	Appendix B presents specific guidelines relevant to 
estimating required sample sizes.  Included in the 
discussion are: (1) references to statistics books 
containing sample size

51


equations, (2) recommendations regarding values for the 
three input factors in the sample size equation, and (3) 
suggestions regarding implementation of the field data 
collection effort based on the calculated sample size 
values. Appendix B also contains a section on the basic 
types of possible statistical analyses, appropriate 
confidence levels, and desirable reporting formats.

	The contractor should follow the guidelines in 
Appendix B to develop appropriate sample sizes for each 
measure.  The Evaluation Plan should contain the sample 
size values, along with an explanation of any assumptions 
or special procedures underlying these values (e.g., 
equations, input factor values used).

	3.3.5 Timing of Data Collection

	For measures based on sampling, another issue to be 
addressed by the contractor is the timing of data 
collection.  The exact periods during which measures are 
collected have a significant effect on the validity and 
representativeness of evaluation results, since the 
operation and effectiveness of a transportation system 
are sensitive to various factors associated with time.

Four basic questions arise concerning the timing of data 
collection:

(1) The appropriate season(s) of the year and day(s) of the 
    week to  include in the sample,
(2) The appropriate duration of each data collection period,
(3) The proper time to initiate data collection, and
(4) The appropriateness of "one- shot" vs. periodic monitoring

	The particular season(s) and day(s) depend largely 
on the assumed sensitivity of the APTS application to 
each time unit.  If it is deemed appropriate to assess 
the impact of the APTS application under reasonably 
normal conditions, data collection should be performed 
during the fall and spring, when weather conditions are 
not severe, schools are in session, and few people are on 
vacation.  To the extent that the experimental test 
evaluation involves measures related to travel patterns 
and transit usage, the contractor should attempt to 
schedule data collection activities during those two 
seasons which are most representative of normal 
conditions.  On the other hand, if severe weather 
conditions or other atypical conditions are an inherent 
feature of the site and it is desirable to examine the 
experimental test under a full range

52


of possible conditions, the contractor should schedule 
data collection throughout the year so the sample 
observations include extreme as well as normal 
conditions.

	If a particular transit service operates seven days 
a week, then the sample of days should include both 
weekdays and week-end days (in fact, the data should be 
stratified by weekday vs. week-end day to highlight the 
differences during these two periods).  Regarding which 
day(s) to include in the weekday sample, similar logic 
applies as in the case of seasons.  If the aim is to 
observe the project under typical weekday conditions, 
then any day(s) with abnormal traffic patterns should be 
avoided.  In some cities, there is a difference between 
Monday and/or Friday conditions vs. 
Tuesday/Wednesday/Thursday conditions; if this is known 
to be the case for a particular test site, then data 
collection should be scheduled for the three "typical" 
days rather than either of the typical days.  The 
contractor should consider the special characteristics of 
the operational test and the site in deciding which days 
are appropriate.  If a large number of days is going to 
be involved, and there is no particularly significant 
distinction among days of the week, then a randomly 
selected sample of days would be preferable.

	The duration of each data collection period should 
be determined based on the degree of day-to-day 
variability and on the required sample size.  If the 
particular item being measured is suspected to vary in 
behavior from one day to the next, then the data 
collection period should include several days; if it has 
been determined that only Tuesdays, Wednesdays, and 
Thursdays can be used, then several weeks may be 
necessary to achieve the required sample of days.  
Moreover, if the sample size required for a particular 
variable is large, then several days of data collection 
may be appropriate to obtain the minimum sample of 
observations.

	The choice of initiation time for each data 
collection period is dependent on a number of 
considerations, the chief one being that the "after" data 
collection not begin until the use of the APTS 
application is fully operational and its performance has 
stabilized.  In general, it will probably take at least a 
few months for an APTS application to become fully 
operational, with all the "bugs" worked out, and possible 
behavioral influences associated with the application are 
eliminated.  The desire is to achieve a "steady state" 
for the system after the application has been 
implemented.  The time to achieve this "steady state" 
undoubtedly will vary from project to project.  Thus, 
data collection related to the test should not commence 
until these adjustments and modifications are completed.  
Other factors determining the initiation date for data 
collection

53


are the desire to avoid summer and winter months 
and the overall schedule of the operational test.

	In most instances, data collection will be 
performed for discrete phases of the operational 
test (i.e., before the project is implemented, 
while the project is operational, and possibly 
after the project is terminated).  
Post-operational test data collection would only 
be performed if there was a desire to see whether 
operation of the APTS experiment for a limited 
period had led to permanent changes in people's 
travel patterns or attitudes.  However, if 
operational test elements are by nature changing 
continually or if it is expected that the APTS 
application will cause gradual but continual 
changes in transit performance measures, then a 
periodic process of data collection would be more 
appropriate than merely "before," "during," and 
"after"  data collection.  The multitude of data 
points obtained from a periodic monitoring 
process will make possible the examination of 
functional relationships either among measures of 
interest or in a time series.   Moreover, 
monitoring of certain measures during the early 
months following introduction of the 
application(s) may be useful in determining when 
the effects have stabilized enough to initiate 
full-scale data collection.  It should be noted 
that if periodic data collection is appropriate, 
then a sequential analysis procedure (similar to 
control charts) may be useful to permit 
reductions in sampling requirements.

The contractor's Evaluation Plan should 
indicate the exact timing of data collection for 
each measure involving sampling.  This 
information should be presented in a schedule 
which also shows the projected implementation 
dates for the various elements of the project.

54


4. GUIDELINES FOR PERFORMING EVALUATION ACTIVITIES

	This chapter presents suggestions relative to 
implementing the evaluation of an APTS operational test.  
During the evaluation implementation phase of the 
evaluation process, data collection/analysis relating to 
site characteristics, quantitative measures, and 
qualitative measures is undertaken according to the plans 
and procedures laid out in the Evaluation Plan.  In 
addition, information is gathered relative to the 
project's operational history and external events which 
may have some bearing on the project outcome.  This 
information is eventually incorporated into the analysis 
and interpretation of project results.

	Contractor functions during the evaluation 
implementation phase include monitoring and/or 
performance of data collection activities, data reduction 
and analysis, subjective analysis of information relative 
to project issues, and synthesis of project findings into 
a Final Summary Evaluation Report.  In accordance with 
these contractor functions, this chapter of the 
guidelines is organized into two sections: (1) 
monitoring/performance of data collection and (2) data 
reduction, analysis, and presentation.  The recommended 
content and organization of the various contractor 
reports prepared during this phase are presented in 
Chapter 5.

	During this phase, the contractor must maintain a 
sensitivity to the relationships among the organizations 
involved in the project -- in particular the local 
sponsor or project team, FTA, and the Volpe Center (see 
Chapter 2).  The contractor must work closely with these 
groups at the appropriate times, while maintaining the 
role and perspective of an external, objective 
organization assessing the impact of the operational 
test.

4.1 MONITORING/PERFORMANCE OF DATA COLLECTION

	Since much of the data required for evaluations 
will be unavailable from pre-existing data bases and 
secondary sources, each operational test will undoubtedly 
involve significant data collection efforts.  Given the 
considerable amount of time and money which will be spent 
on data collection; careful management and oversight of 
the data collection process are essential.  Where 
possible and appropriate, data collection may involve the 
use of students from local colleges and universities.

55


	The contractor is responsible for ensuring that 
data collection is performed according to the Volpe 
Center/FTA-approved Evaluation Plan.  There are three 
potential alternatives associated with data collection.  
One of these occurs when the local sponsor or operator 
collects all data (under FTA/APTS and/or local funding), 
and the contractor acts in a monitoring role to assure 
the quality and timeliness of data collected, as well as 
adherence to procedures laid out in the Evaluation Plan.  
A second alternative occurs when the contractor collects 
the data, and coordinates the timing and performance of 
these activities through the local sponsor or operator.  
The third possibility is one in which both collect 
various elements of the data.

	In order to monitor and/or perform the data 
collection activities called for in a given evaluation, 
the contractor will need to maintain open channels of 
communication with the site, in the form of visits, 
telephone and written correspondence with the appropriate 
local agencies as well as subscriptions to local 
newspapers.  In the rare instance where day-to-day 
contact with the site is necessary, the contractor should 
arrange to base a member of the firm at the site.

	Whether data collection is being performed by the 
contractor or by the local sponsor, the contractor must 
stay closely involved in all phases to make sure the 
procedures specified in the Evaluation Plan are followed. 
In cases where the local sponsor or other local agency is 
collecting data, the contractor should meet frequently 
with the agency to discuss progress and problems, work 
out solutions to the problems, and observe key phases of 
field data collection.  In addition, the contractor 
should occasionally perform independent spot checks, 
especially in the case of measures for which the local 
agency has limited experience in data collection.

	The contractor is expected to inform the Volpe 
Center of the status of data collection in its Monthly 
Evaluation Progress Reports (see Chapter 5 for the 
recommended content and organization of this type of 
report).  Should there be an unacceptable degradation of 
quality or timeliness of data collected by the local 
sponsor, the contractor should notify the Volpe Center in 
writing.  The Volpe Center will in turn take steps 
through the FTA Project Manager to rectify the situation.

	Over and above monitoring data collection 
activities, the contractor should keep abreast of the 
status of the operational test. This awareness of project 
operational status is important so that: (1) data 
collection activities can be smoothly coordinated with 
ongoing project activities (causing minimum disruption of 
day-to-day operations), and (2) evaluation results can be

56


interpreted in the context of project history.  The local 
sponsor's quarterly project progress reports to FTA/Volpe 
Center (see Chapter 5 for recommended content and 
organization) will be a useful source of information on 
the project's operational evaluation.  However, the 
contractor is encouraged to obtain a more detailed 
account of progress/problems relative to implementing and 
operating the APTS test by talking with the local sponsor 
at the site.

	In addition to keeping abreast of project 
operations, the contractor should be continually watching 
at the site for unexpected (external) events which might 
affect the validity of project results.  In any 
implemented operational test, no matter how well 
controlled or planned, the possibility remains for 
unexpected events to occur that may have an impact on 
measures of the project's performance.  These unexpected 
occurrences are classified as threats to the validity of 
the operational test.

	Unanticipated developments at the site can take the 
form of temporary events such as a driver strike or 
longer-term phenomena such as the closing of a major 
thoroughfare.  The following are examples of unexpected 
factors that have been experienced in earlier FTA 
projects, along with an indication of the compensatory 
action taken to counteract the external event:

(1) Changes in employment. There were thousands of 
    unemployed in Seattle due to the high number of 
    layoffs in the aerospace industry.  (No compensatory 
    action was taken.)

(2) Changes in freeway traffic volumes.  Shirley Highway 
    experienced a shift from arterials to the freeway 
    upon completion of new lanes and sections. 
    Minneapolis on the other hand, noted a shift to the 
    freeway due to arterial street construction and land 
    developments within the project. Seattle noted volume 
    shifts on the freeway entrance and exit ramps where 
    new lanes had been added or preferential treatment 
    was given to buses. Seattle also experienced a 
    queuing problem onto the freeway from autos that were 
    diverted from converted ramps.  (An adjustment in 
    queuing sequence was made where necessary.)

(3) The national energy crisis.  Minneapolis experienced 
    a drastic change in traffic volumes from auto to 
    transit during the energy crisis.  Although it cannot 
    be determined whether the shift in volumes was 
    directly attributable to this factor, the timing of 
    the initiation of the project during this period may 
    have had some impact on data interpretation. 
    (Extended routes and an increase in the frequency of 
    service were the immediate modifications made to 
    facilitate transporting such large number of people.  
    Also, their marketing campaign was modified -- slowed 
    down in view of the large numbers.)
	
57


	As previously noted, the use of a test-control 
evaluation design will, in certain cases, mitigate the 
impact of these unplanned events on the validity of the 
project results.  For further information regarding the 
phenomena that can jeopardize internal and external 
validity, see Campbell and Stanley.

	The contractor is responsible for informing the 
Volpe Center of any unplanned phenomena which arise 
during the course of the evaluation.  The contractor's 
Monthly Evaluation Progress Report should describe the 
potential effects on validity of any phenomena noted, as 
well as propose changes in the project and/or evaluation 
to compensate for the unplanned occurrences.

	Although data collection should generally proceed 
according to the Evaluation Plan, there may be instances 
where modification to the originally planned procedures 
is warranted.  The previous paragraph indicated that 
external events at the site might be cause for modifying 
the evaluation.  Two additional reasons for deviating 
from the planned approach are discussed below, namely, 
operational changes in the project, and availability of 
improved evaluation techniques.

	Operational changes in the project can come about 
as a result of contractor recommendations (transmitted in 
the Monthly Evaluation Progress Reports) or decisions by 
FTA and the local sponsor.  Whatever the impetus for 
these changes in the scope or operation of the 
operational test, the evaluation will have to be modified 
accordingly.  The contractor is responsible for assessing 
the impact on the evaluation of any forthcoming or 
proposed operational changes, and recommending 
appropriate modifications of the Evaluation Plan to the 
Volpe Center.

	As new data collection techniques are developed in 
the course of the APTS program, it may be appropriate to 
modify certain aspects of a project's Evaluation Plan.  
The contractor will have to assess, on a case-by case 
basis, whether the potential benefits of the new 
techniques are sufficient to justify modification to the 
planned evaluation activities, and then recommend the 
appropriate course of action to the Volpe Center.

	In order to further the state of the art of transit 
evaluation, the contractor is responsible for performing 
an ongoing assessment of data collection procedures used.  
The evaluation contractor should maintain close control 
over data collection procedures used and summarize

58


findings with respect to certain techniques reflected by 
the Volpe Center for further examination.

These findings will include, as a minimum:

(1) a narrative description of how the collection procedure was 
    and implemented,
(2) an indication of areas in which the technique outperformed 
    expectation,
(3) an indication of areas in which the technique was deficient,
(4) some summary of the inherent variability in collecting 
    project measures due to the technique itself, as opposed to
    variability due to other operational test factors,
(5) an estimate of the cost of implementing the technique, and
(6) where two techniques have been employed to collect 
    the same basic measures, cross-comparisons and a 
    recommendation as to which technique should be used 
    in similar future operational tests.

This information will ultimately be incorporated into an 
appendix of the Final Summary Evaluation Report.

4.2 DATA REDUCTION, ANALYSIS AND PRESENTATION

	The contractor is responsible for performing all 
data reduction and analysis, regardless of which agency 
has collected the data.  Data reduction involves the 
processing of raw data, either manually or using a 
computer, to yield statistics such as means, standard 
deviations, ratios, ranges, frequency distributions, 
coefficients of determination, correlation coefficients, 
ratios, "t" statistics, and elasticities.  The specific 
statistic to be calculated and the need to control for 
other variables will depend in pan on the type of measure 
and type of comparison involved. Quantitative measures 
such as travel time and vehicle passenger counts might be 
processed into average values for each level of 
stratification used.  If a comparison of two time periods 
is involved, the percentage change from the earlier to 
the later period might be calculated, or two multiple 
regression equations might be calibrated and their 
coefficients compared.  Quantitative measures relating to 
schedule dependability might be summarized into average 
values as well as standard deviations, with comparisons 
calculated as ratios of standard deviations.  Some 
qualitative measures for example, might be obtained 
through surveys and

59


might be presented to yield frequency distributions for 
the response categories.  It should be stressed that the 
level of analytical sophistication and choice of 
quantitative and qualitative measures will vary from test 
to test depending largely on the objectives being 
evaluated.

	Data reduction may involve the use of statistical 
inference techniques.  If the data are based on a 100% 
data collection effort (i.e., no sampling), then exact 
values of the statistics listed above can be calculated.  
However, if the data has been obtained by sampling (as 
will usually be the case), results cannot be presented as 
precise values, since there is a certain probability that 
the calculated values are different from the true 
population values.  It is recommended that data based on 
samples be processed into two-sided confidence intervals 
using two confidence levels:
 Ķ =.01 and Ķ =.05.  Appendix B presents further 
guidelines relative to calculating confidence intervals. 

	The contractor should arrange for smooth transfer 
of collected data from the collection site (e.g., buses, 
transit company, roadside stations) to the processing 
site.  Special attention should be paid to details such 
as labeling and dating of forms, tapes, etc. to make sure 
that valuable data is not lost or altered.  

	The basic data which are collected during an 
operational test should be maintained either on 
appropriate storage devices (e.g., hard discs, floppy 
discs).  While the raw data may not be immediately 
utilized, it should remain with the contractor (or 
eventually the Volpe Center) for potential future uses.  

	Data analysis involves the interpretation and 
synthesis of the processed data and other information to 
draw conclusions relative to the attainment of project 
objectives and issues and relative to project 
transferability.  Statistics such as those cited above, 
which range from the simple to the complex, are carefully 
examined and pulled together to obtain a comprehensive, 
in-depth understanding of the effects of the operational 
test, and the underlying reasons for observed changes.  
The contractor must apply sound judgment as well as 
knowledge and experience relative to transit system 
operations, traffic operations and travel behavior in 
order to interpret the collected data and place it in 
proper perspective.  To the extent possible, the results 
of the APTS applications at the site should be 
supplemented by an assessment of the influence of 
site-specific and external factors on project outcome, so 
that conclusions can be made regarding the potential 
applicability and effects of implementing the operational 
test in

60


other sites across the country.  In order to further 
enhance project transferability, the analysis/synthesis 
phase should provide a compilation of lessons learned 
regarding the operation of the test.

	The contractor should understand and be aware of 
the importance that the use of appropriate statistical 
techniques can attach to the analysis and interpretation 
of project results.  In view of the fact that most 
aspects of an urban transportation system tend to be 
dynamic and variable from hour-to-hour, day-to-day, and 
month-to-month, observed differences could be 
attributable only to this inherent variability and not to 
the APTS applications.  Furthermore, factors other than 
the planned and controlled innovations could also be 
directly related to the observed changes in those 
measures being collected.  It is important to note that, 
while no single technique exists for removing the 
potential influence of these external factors, it is 
possible by careful analysis, to at least point out the 
occurrence of such events and create an awareness for 
those who review the project's conclusions and/or 
recommendations.  Hence, it is important to be able to 
specify whether the observed differences in, for example, 
travel time are within reasonable bounds of expected 
variability inherent in the given transportation system, 
or whether the observed differences cannot be accounted 
for just by system random variability.  If the latter 
case were true, taking into consideration the potential 
external influencing factors, one could conclude that the 
application has in fact provided a real change in the 
measures being considered.  It is to this capability for 
making valid inferences that the specific statistical 
techniques apply.

	Presentation of project results in Annual Project 
Status Summaries, Interim Evaluation Reports, and Final 
Summary Evaluation Reports should be in the form of 
quantitative and qualitative exposition, with exhibits 
such as tables, graphs, and bar charts serving as the 
focus for narrative discussion.  In no instance should 
there be an excessive narrative describing all the 
elements of an exhibit.  This tends to be redundant and 
masks the really important findings.

	Chapter 5 provides some guidance relative to 
overall content and organization for the 
aforementioned-reports.  With respect to the format for 
exhibits, creative techniques for displaying information 
are encouraged, so long as the information is presented 
in a clear and accurate manner.  In order to provide the 
contractor with some indication of the types of exhibits 
that are acceptable, some examples are presented on the 
following pages.

61


	Exhibits 10-13 are clear and informative.  While 
they do not present detailed information, they are useful 
in highlighting the project findings appearing in an 
executive summary, which is designed to convey rapidly to 
the decision-maker the significant conclusions of the 
project.  Back-up exhibits which contain significantly 
more detail of simple statistical results, multiple 
regression analysis, and benefit-cost analysis should be 
contained within the body of the evaluation report or in 
technical appendices.  Exhibits 14-20 fall into this 
category.

	The contractor should perform data reduction and 
analysis as data are collected, so that interim results 
are available throughout the project evaluation.  These 
interim findings will not only satisfy general curiosity 
regarding the project's effects, but will also provide 
feedback information relative to ongoing project 
operations and evaluation.  Examination of preliminary 
evaluation results may suggest opportunities for 
modifying the project and/or evaluation procedures so as 
to increase the utility of the operational test.  To 
facilitate inter-project analyses, contractors will be 
required to deliver all survey data in an ASCII file 
format in machine-readable form.

62


Click HERE for graphic.

63


Click HERE for graphic.

64


Click HERE for graphic.

65


Click HERE for graphic.

66


Click HERE for graphic.

67



	EXHIBIT 15. CHARGE-A-RIDE USAGE BY CARD TYPE

AND TIME PERIOD FOR THE MERRIMACK VALLEY CHARGE-A-RIDE PROGRAM

CARD TYPE       AM PEAK	       MIDDAY	PM PEAK     TOTAL

ADULT	         1,761		650	1,430     3,841
COLUMN%		    69		 65	   76	     70
ROW%		    46		 17	   37	      -

HANDICAPPED	   472	        118	  331	    921
COLUMN%		    18		 12	   17	     17	
ROW%		    51	         13	   36	      -

ELDERLY		   235		199	  105	    539		
COLUMN%		     9		 20	   06	     10
ROW%		    44		 37	   19	      -

STUDENT		    96		 29	   26	    151
COLUMN%		     4		 03	    1	     03	
ROW%		    64		 19	   17	      -

TOTAL		 2,564		996	1,892     5,452
ROW%		    47		 18	   35	      -

68


			EXHIBIT 16.
COMPARISON OF FARE PAYMENT TIMES USING DIFFERENT METHODS


PAYMENT TYPE	   MEAN	   STANDARD      NO.OF     COMPARATIVE
		TIME(Secs) DEVIATION	OBSERVS.  DIFFERENCE1	
CHARGE-A-RIDE			
AUTOMATED	  6.80	    3.30	 131	     -
MANUAL		 10.34	    5.58	  72	   SLOWER

EXACT FARE CASH
READY TO PAY	  6.18	    4.02	  23	   NONE
NOT READY	  7.09	    5.46	  37       NONE
BOTH		  6.74	    4.98	  60	   NONE

CASH WITH CHANGE
PASS PRESENTED	 12.57	    5.61	 21	   SLOWER
DOLLAR BILL	 11.70	    2.24	 20	   SLOWER
INCORRECTIONS	 13.64	    5.34	 30	   SLOWER
ANY COMB. OF	 12.86	    4.47	 50	   SLOWER
 BILLS OF COINS

TEN-RIDE TICKETS  5.77	    5.80	46	  FASTER

____________________

1 REFERS TO DIFFERENCES  OF MEAN OF METHOD COMPARED TO 
AUTOMATED CHARGE-A-RIDE.  THE FARE PAYMENT METHODS ARE 
COMPARED TO AUTOMATED CHARGE-A-RIDE USING A SIMPLE T-TEST 
OF DIFFERENCE BETWEEN THE MEAN TIMES

69


EXHIBIT 17.
PROJECT EFFECTIVENESS MEASURES FOR THE SEATTLE PROJECT

 
70


Click HERE for graphic.


71


    EXHIBIT 19 RESULTS OF BEFORE AND AFTER ANALYSES FOR 
             PORTLAND SELF-SERVICE FARE COLLECTION


SPRING 1982 (PRE-SSFC):

DWELL = 5.95 + 1.18 (ALIGHTINGS) + 2.46 (BOARDINGS); R2 = 0.82
(TIME/BUS STOP)     (.064)               (.052)


SPRING 1983  (SSFC):

DWELL = 8.26 + 1.58 (ALIGHTINGS) + 1.93 (BOARDINGS) R2 = 0.66
(TIME/BUS STOP)     (.064)               (.052)

72


	   EXHIBIT 20 BENEFIT-COST ANALYSIS RESULTS

	  OF SALT LAKE CITY RIDER INFORMATION SYSTEM

YEAR INCURRED BY UTA1

COST/BENEFIT CATEGORY	     1983   1984    1985      TOTAL

SYSTEM COSTS
TELERIDE CONTRACT	$ 485,918  $36,263  $ 203,070  $ 725,251
UTA LOAN FROM TELERIDE	 -165,050	-	-	  -
UTA SUPPORT COSTS
  STAFF COSTS		   60,671   27,578	-	  88,249
  INSTALLATION/START UP	   29,083      -       22,917	 -52,000
  TELEPHONE		   34,286   15,584	-	  49,870
  HARDWARE & SOFTWARE	   33,000   15,000	-         48,000
  MAINTENANCE

TOTAL INITIAL COSTS	$ 477,908  $94,425  $ 225,987  $ 798,320

SYSTEM BENEFITS
REVENUE INCREASES DUE TO    5,753    2,876	6,967	  15,596
TRANSIT MARKETING

COSTS AND BENEFITS IN 
1983 DOLLARS2
INITIAL COSTS		$  77,908 $ 90,570  $ 207,913  $ 776,391
INITIAL BENEFITS	    5,753    2,759	6,410     14,922

__________________
1. EXCEPT THE LAST TWO ROWS, ALL COSTS AND BENEFITS ARE 
   IN 1985 DOLLARS.  COSTS ARE AS SPECIFIED IN THE UTA-TELERIDE
   AGREEMENTS

2. BASED ON ACTUAL 1983-1984 AND ASSUMED 1984-1985 INFLATION 
   RATE OF 4.3  PERCENT
	
73


		5. CONTENT AND ORGANIZATION OF REPORTS

	At various stages in the evaluation process the 
contractor for each APTS project is responsible for 
submitting specific reports to the Volpe Center.  These 
reports include: an Evaluation Plan submitted prior to 
conducting the evaluation, Evaluation Progress Reports 
submitted monthly throughout the project, Interim 
Evaluation Reports submitted periodically throughout the 
project, and a Final Summary Evaluation Report submitted 
at the conclusion of the project.  The local sponsor for 
each operational test is also responsible for submitting 
quarterly progress reports on project status.  
Appropriate information in these reports will be included 
in the contractor's monthly Evaluation Progress Reports.

	This chapter presents recommendations on content 
and organization which will guide the contractor in the 
preparation of these reports.  The suggested content and 
organization for the local sponsor's quarterly progress 
reports are also presented.

	5.1 EVALUATION PLAN

	The Evaluation Plan is written by the contractor to 
explain, in detail, how the evaluation of the particular 
project will be performed.  The following is a summary of 
the suggested content and organization format for the 
Evaluation Plan:

	(1) Overview of the operational test

  . Operational test including description of APTS application
  . APTS program objectives addressed
  . Other relevant project objectives/issues addressed
  . Project history (events or studies leading up to test)
  . Project schedule
  . Project funding (total operational test costs by 
    source of funding, capital costs by application)
  . Project local sponsor/operating agency

75


 (2) Description of the operational test evaluation

  . Overview of basic evaluation design, constraints 
    affecting development of the Evaluation Plan
 
  . Timing of evaluation stages as related to project 
    implementation schedule
 
  . Site data collection plans and sources
 
  . Quantitative measures, qualitative measures, and/or 
    information to be collected in connection with each 
    project objective and issue
 
  . Proposed data collection/derivation and analysis 
    techniques for each measure
 
  . Schedule of data collection activities associated 
    with the evaluation, and identification of which 
    organization (contractor, local sponsor, other 
    local organization) is to perform each activity.  
    Schedule should indicate submittal dates for any 
    Interim Evaluation Reports and the Final Summary 
    Evaluation Report

(3) Technical management and cost information

  . Estimate of contractor person-hours by labor 
    category (e.g., senior, middle, regular, 
    administrative) and task (i.e., management and 
    coordination of evaluation plan preparation and 
    updating, data collection/monitoring of data 
    collection, data reduction/data analysis, and 
    report preparation) for the project
 
  . Estimate of contractor direct costs by category 
    of cost (travel, computer, etc.), task and 
    evaluation stage
 
  . Estimate of total contractor evaluation costs by task
 
  . Estimate of person-hours and costs for data collection to 
    be performed by other organizations (by activity, if 
    possible)

	To facilitate the incorporation of modifications, 
the Evaluation Plan should be submitted in looseleaf form 
and on a WordPerfect file.  As modifications are made, 
each page will have the date of modification indicated.  
Modifications may result from the initial review of the 
Plan by the Volpe Center, FTA, and the local sponsor or 
they may occur during the evaluation implementation 
phase.  As an example of the latter situation, 
examination of interim findings may reveal certain gaps 
or redundancies in the originally planned data collection 
program.

76


Other reasons for modifying the Evaluation Plan during 
the implementation phase might be operational changes in 
the project, unanticipated developments at the site, or 
identification of a refined evaluation procedure.  The 
mechanism for obtaining Volpe Center approval to modify 
the Evaluation Plan procedures is described below under 
Monthly Evaluation Progress Reports.

5.2 MONTHLY EVALUATION PROGRESS REPORTS

	The Monthly Evaluation Progress Reports are written 
by the contractor to keep the Volpe Center and FTA 
abreast of the status of the project evaluation the 
contractor is performing.  These reports are intended to 
be as concise as possible.  The following is a summary of 
the suggested minimum content and organization for the 
Monthly Evaluation Progress Report.

(1) Review of evaluation activities during the past 
    month. Evaluation-related problems encountered and 
    actions taken to rectify them.  Narrative highlights 
    of project and/or evaluation related external factors 
    and other events which appear to be significant and 
    might influence the evaluation of the project.  A 
    review of the implementation process, including a 
    discussion of problems and issues encountered, steps 
    taken to resolve such issues, and associated 
    operational delays, difficulties, and other 
    consequences, if any.

(2) Status of data collection and analysis activities 
    that have taken place in the past month (performed by 
    both local sponsor and evaluator).  Any contractor 
    documentation on preliminary results which have been 
    generated in this area should be appended to the 
    Progress Reports or could be submitted separately as 
    special technical memoranda.

(3) An indication of whether the evaluation is proceeding 
    according to schedule, and, if not, reasons for the 
    departure.

(4) A brief discussion of anticipated activities to be 
    covered during the succeeding report period.  
    Forthcoming Interim Reports, if any.
 
(5) Comparisons of cumulative budget to actual 
    expenditures.  Estimate of costs to complete 
    evaluation tasks.

(6) Recommendations for changes, if any, to the 
    Evaluation Plan, and the reasons such changes are 
    recommended.  (Volpe Center concurrence is 
    needed before any changes to the Plan can be made.)

77


5.3 INTERIM EVALUATION REPORTS

	Interim Evaluation Reports are written periodically 
by the contractor to present interim findings relative to 
all or some of the operational test objectives and to 
evaluate those aspects of the project where it is 
applicable to do so.  Although submitted to the Volpe 
Center, they will also be further disseminated to other 
interested parties.

	If the evaluation process is divided into distinct 
stages whose durations fall roughly within a month time 
frame, then interim reports should be written at the end 
of each stage.  Otherwise, interim reports should be 
written annually, except that no interim report is needed 
at the end of the operational test.  The suggested 
content and format for Interim Evaluation Reports is 
similar to that presented for the Final Evaluation Report 
described next.

5.4 FINAL SUMMARY EVALUATION REPORT

	The Final Evaluation Report is structured by the 
contractor early in the operational test and completed at 
the conclusion of the project.  Its purpose is to 
synthesize the findings relative to each of the APTS 
Program objectives and other relevant objectives/issues 
into an evaluation of the overall project.  Although 
submitted to the Volpe Center, it is also meant for 
dissemination to a technical audience.  The suggested 
content and organization for the Final Summary Evaluation 
Report are given below.

(1) Executive Summary
  . Should be capable of standing on its own and 
    being published separately.

(2) Project Overview
  . Description of project innovations, the APTS 
    Program objectives addressed, and other relevant 
    project objectives/issues.  A brief overview of 
    the operation of the project over its life, and 
    highlights of project related external factors 
    and other events that have been significant 
    enough to influence the project.  A review of the 
    implementation process, including a discussion of 
    problems and issues encountered, steps taken to 
    resolve such issues and problems, and associated 
    operational delays, difficulties, and other 
    consequences, if any.

(3) Site Overview
  . Description of the site, presentation of pertinent site 
    data, and highlights of site related external factors that 
    may have been significant enough to influence the project.

78


(4) Evaluation Overview
  . Description of the basic evaluation procedure and 
    the timing of evaluation stages.

(5) Project Results
  . Assessment of the project in terms of its 
    attainment of relevant APTS Program objectives 
    and other (local and/or national) project 
    objectives; and insight into project issues 
    associated with operational feasibility and 
    characteristics of the APTS application being 
    tested.  Relevant data are analyzed and presented 
    in the forms of charts, graphs, and/or narrative.

(6) Implications Regarding Transferability

  . Assessment of the influence of site-specific 
    characteristics and external factors on the 
    outcome of the operational test.

  . Lessons learned, based on practical experience, 
    relative to the implementation and operations of 
    the APTS application.  Can include suggestions 
    for project modifications at the test site or for 
    future APTS applications in other locales.

(7) Appendices
 
  . Project costs
 
  . Data Collection: Site data, quantitative measures, and 
    qualitative measures collected.
 
  . Assessment of evaluation procedures employed (e.g., 
    effectiveness of particular survey approaches used, cost/
    accuracy of innovative data collection techniques).

5.5 QUARTERLY PROJECT PROGRESS REPORTS

	The Quarterly Project Progress Reports are written 
by the local sponsor to keep FTA/the Volpe Center abreast 
of the status of the operational test project 
implementation for which the local sponsor is 
responsible.  The following is a summary of the suggested 
minimum content and organization for the Quarterly 
Project Progress Reports to be prepared by the local 
sponsor.

(1)  Review of operational test activities during the past 
     quarter. Project-related problems encountered and actions 
     taken to rectify them.  Narrative highlights of 
     external factors and other events which appear to 
     be significant and might influence the project.  A 
     review of the implementation process and an 
     indication of whether the project is proceeding 
     according to schedule and, if not, the reasons.

(2) Status of planned data collection activities.

79


(3) Comparisons of budgeted to actual expenditures.  Estimate 
    of costs to  complete project.

(4) Recommendations for changes, if any, to the conduct of the 
    project, and the reasons such changes are recommended to 
    the Volpe Center.

80


			   APPENDIX A
		 SURVEY EXECUTION AND DESIGN

	It is anticipated that the evaluation of every APTS 
operational test will require data that can be obtained 
only from surveys, and will therefore require some form 
of survey data collection.  Among the possible survey 
respondents are APTS service users, auto users, service 
area residents who do not use transit, and transit 
company personnel.  Typical survey objectives might 
include: determining user and non-user characteristics, 
attitudes toward transit service, and past and present 
travel behavior; measuring modal shift; and assessing the 
experience of transit officials with regard to 
implementing a new APTS technique.  Although the specific 
contexts in which the surveys are conducted may differ, 
there is still a need for consistency of procedure in 
survey design and data collection to insure comparability 
of results.

	In surveys, the researcher is collecting data from 
real life situations, which means that many 
unanticipated, spontaneous, and unusual situations will 
arise.  Some of the unanticipated situations are briefly 
mentioned in Section 4.1, and more detailed discussion 
regarding these situations and their association with the 
validity of the evaluation results is presented in 
Campbell and Stanley.  To compensate for the survey 
researcher's lack of control of the experimental 
situation, the need for consistency and the establishment 
of general policies or guidelines to handle a great 
variety of possible developments is most important.

	This Appendix contains guidelines for use in 
formulating and carrying out surveys.  It discusses how 
to define the populations to be sampled (i.e., the survey 
universes), describes how to select samples that will be 
representative of that universe, examines techniques for 
surveying the samples selected, presents suggestions as 
to survey content and format (including a list of 
standardized questions and, in some instances, 
standardized responses to serve as a basic set for most 
surveys), and discusses the problem of non-response bias.

	A separate section at the end of this Appendix 
contains guidelines for conducting interviews with 
transit company personnel (e.g., drivers, management, 
mechanics).

	It should be stressed that this Appendix presents 
no hard and fast rules which must be followed by each 
contractor.  It merely guides the contractor in designing 
and executing surveys.  In determining survey 
methodology, the contractor should consider potential 
alternatives and

A-1


give the rationale for decisions made in terms of the 
survey objectives, site characteristics, and any other 
relevant factors which have influenced the decision.

A.1 DEFINING THE SURVEY UNIVERSE
	The first step in executing surveys is to define 
the survey universe (i.e., the groups about which the 
surveys are seeking knowledge).  It is apparent that 
knowledge about project service users' travel behavior, 
characteristics, and attitudes toward transit is needed 
in an evaluation of project service.  Moreover, an 
evaluation of project service will usually not be 
complete without some data on non-users, particularly to 
identify who they are and why they do not use the project 
service.  Accordingly, there are two survey universes 
which will be relevant for APTS projects: users of the 
transit service employing the APTS application, and 
non-users of this service.  Users are defined as those 
who ride this service at least occasionally but still on 
a regular basis, e. g., regularly twice a month. 
Non-users are defined as those using alternate modes 
(i.e., other than the APTS project service) who make 
trips that could be made on the project service.

	Occasionally, there will be a third survey universe 
of interest, the general population of the region in 
which an APTS project is being implemented.  Attitudinal 
surveys of this universe will be used to obtain a profile 
of the community in which the transit service is being 
provided.  It should be apparent that many of the 
questions asked users, non-users, and the general 
population will be different.

	Definition of the term APTS project service area 
allows a more precise definition of non-users and the 
general population.  The project service area is defined 
as the area that comprises on the order of 90 to 95 
percent of the origins and destinations of the users of 
the service.  Since non-users are potential users, the 
origins and destinations of non-users should be 
comparable to those of users.  Non-users can now be 
defined as persons not using the APTS project service who 
make trips that begin in the origin portion of the 
service area and end in the destination portion of the 
service area at the same times as users make these trips.  
The general population in the region of the operational 
test can now be defined as the population residing within 
the service area.

A-2


	The operational test service area is usually not 
well defined at the outset of the project and must 
initially be estimated.  In some projects, specifically 
demand-responsive projects, the origin and destination 
portions of the service area are given.  At the other 
extreme, in projects in which park-and-ride is a 
significant access mode, it may be impossible initially 
to estimate the service area accurately.  A 
conservatively estimated area that includes all possible 
park-and-riders would have to be initially defined as the 
origin portion of the project service area.  Once survey 
data on the origins of park-and-riders is obtained, a 
more accurate estimate of the service area can be made, 
and non-users can then be identified.

A.2 SAMPLING THE SURVEY UNIVERSE

	The next step in executing surveys is selecting an 
appropriate sample for surveying users, and, where 
applicable, selecting appropriate samples for surveying 
non-users and the general population.

	The purpose of sampling is to reduce the amount of 
data collection required.  Rather than obtaining 
information from every member of the universe, the 
principles of sampling provide ways to obtain information 
from a very small portion of the universe.  Sampling 
procedures also indicate the accuracy with which the 
characteristics of the universe have been represented.

	A key assumption in sampling is that, prior to 
drawing a sample, the complete universe has been 
identified.  Therefore, every members of that universe 
has a known probability of being  selected for inclusion 
in the sample.  The quality, or representativeness, of 
any sample is directly derived from the completeness of 
the identification of all members of the designated 
universe.

	For these reasons, careful definition of the 
universe and selection of a source from which to draw a 
sample is very important.  If the listing of the 
universe, or the sampling source, is biased through 
failure to include affected members, whether deliberate 
or random, the sample may magnify the bias and may not 
represent the universe.

	A sample of users can be selected from among those 
onboard the transit vehicles or among those at transit 
collection points (i.e., stations), park-and-ride lots, 
or transfer points.  For APTS projects in which all users 
are registered (e.g., demand responsive or subscription

A-3


service), a sample can be selected from among the 
registration lists.  For projects which serve specific 
employment or activity centers, a sample can be selected 
at these centers.

	Selecting a sample of non-users is considerably 
more involved than it is for users.  While the user group 
is identifiable (and can be directly sampled), the 
non-user group cannot explicitly be identified before it 
is sampled.A-1   A larger group must first be sampled, and 
then the trip origins and destinations of the survey 
respondentsA-2  examined in order to identify non-users 
(i.e., those whose trip origins and destinations are 
within the project service area).  A definition of the 
project service area (as previously discussed) is a 
prerequisite for identifying non-users.

	In a project in which travel by users and non-users 
is in a specific direction through a corridor, non-users, 
specifically auto users, can be sampled from license 
plate matches. A  screenline is selected which intercepts 
the main arterials carrying autos between the origin and 
destination portions of the project service area.  A 
sample of the license plate numbers of the autos crossing 
the screenline is recorded and a list of names and 
addresses of the owners of these autos is obtained from 
Department of Motor Vehicle records.  This list (or a 
subset of this list) constitutes a sample in which a 
large percentage are project service non-users.  Some of 
those crossing the screenline do not make trips that 
begin and end in the project service area, and are, 
therefore, not non-users.  However, the entire sample 
must be surveyed because it is not known who the 
non-users are until the trip origins and destinations of 
all those in the sample who completed their surveys are 
examined.  In certain very specific cases, samples can be 
selected directly from the traffic stream (e.g., at toll 
booths, at off-ramps, or from among carpoolers assembling 
at parking lots).

	In operational tests where travel by users and 
non-users is not in a specific direction nor through a 
corridor, the non-user universe cannot be sampled using 
the above methods.  In such cases, a sample may be drawn 
from households in the origin portion of the particular 
project's service area. Lists of households from which to 
select a sample could be obtained from utility

____________________________

[A-1] There may be APTS projects directed at carpooling.  
      In this, carpools would be "users" as defined in 
      this appendix.  However, the population of 
      carpoolers is not explicitly identifiable; therefore, it 
      must be sampled by the same methods used for non-users.

[A-2] This information is requested in the survey.

A-4


records, insurance company records, census block 
statistics, telephone books,A-3 property tax records, etc.  
Many of the people in these households do not make trips 
ending in the destination portion of the project service 
area, and are, therefore, not considered non-users.  As 
previously discussed, the entire sample must still be 
surveyed because the non-users cannot be identified until 
after the entire sample is surveyed.

	If the preceding method is used for obtaining a 
sample of non-users, it should be noted that the 
households selected constitute a sample in which a 
moderate percentage of the people are users.  It may be 
desirable to identify users before they are surveyed (by 
asking all those sampled if they are users) in order to 
ask them questions pertaining to their use of the project 
service.

	In all samples of households, an attempt is made in 
each household to survey only that individual in each 
household who makes a trip ending in or near the 
destination portion of the project service area.A-4   More 
than one household member is surveyed only when more than 
one makes this type of trip.

       For operational tests which serve specific employment or 
activity centers (e.g., handicapped and elderly service 
or subscription service), a sample of non-users does not 
have to be drawn from among households.  A sample can be 
selected from among people at these centers which would 
include non-users (and users also).  If users are 
surveyed, they should be identified before they answer 
any questions in order that the questions asked pertain 
to their use of the project service.

	Where a sample of the general population of a 
region is needed, the sample will always be selected from 
among the households in the project service area.  Again, 
lists of households can be obtained from utility records, 
insurance company records, census block statistics, etc.
_____________________________

[A-3] Where the telephone book is used as the sampling 
      source, there is considerable danger of obtaining a 
      biased sample.  Many households choose to have 
      unlisted telephones.  Also, lower income people are 
      less likely to have telephones, as are residents of 
      boarding houses.  Random digit dialing not only poses 
      potential bias problems but also will be costly because 
      business ant non-residential phones will be selected.

[A4] This comment is also applicable to surveys that are 
     sent to registered automobile owners whose names 
     were obtained from license plate matches.

A-5


	Regardless of the methods chosen for 
selecting samples of both users and non-users 
(and possibly of the general population), every 
effort should be made to assure that samples 
selected are unbiased and large enough for the 
desired statistical confidence.  Such an approach 
involves estimating the percent of persons 
surveyed who are in the universe (i.e., who make 
applicable trips in the service area), estimating 
the response rate, and developing a random 
selection process that aims at the desired number 
of samples.A-5

	In developing a random selection process to 
sample users onboard vehicles, examination of 
vehicle operating schedules and recent passenger 
counts, if available, will be necessary to design 
where and when to select the vehicles on which to 
sample users.  However, the following sources of 
bias in vehicle operating schedules must be 
considered when deciding on the utility of a 
particular schedule for developing a sampling 
source: (1) unscheduled vehicle runs, most likely 
to occur during peak hours, and therefore with 
high passenger loads; (2) schedule delays, 
breakdowns, and accidents, also most likely to 
occur during peak hours when there are high load 
factors; and (3) the occurrence of external 
influences on ridership in the interim, such as a 
strike among people who might have formerly used 
this mode of transportation, the opening of a new 
shopping center or school along the route, or 
unique events such as a concert.  These sampling 
hazards should be kept in mind and some attempt 
should be made to build corrections into the 
research design to compensate, such as 
oversampling on certain routes.

	In many situations, developing a random 
selection process that obtains the desired sample 
size simply involves selecting every Jth person 
going past a given point, or every Jth person on 
a list of users of a given system, or every Kth 
person on a list of employees at a given 
location, or recording the license plate number 
of every Lth auto going past a given point.  To 
obtain a random sample of the households in the 
origin portion of a project service area, every 
Mth household on a list of all of the households 
in the area could be selected; or the random 
clustered household sampling method could be 
used.  This method divides the origin portion of 
the service area into smaller areas (usually 
blocks) of approximately equal population and 
randomly chooses a sample of the resulting 
clusters in which every household in each cluster 
is a part of the sample.
___________________________
[A-5] See Appendix B for a discussion of sample size 
      determination.

A-6


	The possibility of sampling bias occurring through 
use of a particular sampling method should not rule out 
its use.  That sampling method may be very appropriate in 
certain project evaluations.  However, where little can 
be done to minimize the effect of bias, other sampling 
methods should be considered.

	For each survey required for a particular 
evaluation, the contractor must carefully describe the 
universe to which survey research findings will be 
generalized and identify the most complete enumeration or 
sampling source available for that universe.  Actual 
selection of a sampling source must be justified in terms 
of its complete coverage of the affected universe and 
also in light of the survey objectives.

A.3 TECHNIQUES FOR SURVEYING THE SAMPLES SELECTED 

	The final step in executing a survey is determining 
what techniques are applicable for surveying the samples 
that have been selected.  There are five basic techniques 
for surveying these samples:

(1) Self-administered questionnaires handed out by 
    individuals (e.g., survey takers, bus operators, 
    personnel at employment or activity centers), and 
    collected by individuals (not necessarily the same 
    individuals who handed out the questionnaires);

(2) Self-administered questionnaires handed out by 
    individuals and returned  by mail;

(3) Self-administered questionnaires given out by mail 
    and resumed by mail;

(4) Face-to-face interviews; and

(5) Telephone interviews.

A summary of the applicable techniques to be used with 
each possible sampling method is shown in Exhibit A-1, 
which appears at the end of this Appendix with all other 
exhibits.

	With all of these techniques, the greater the 
amount of personal contact between user and survey 
takers, the higher the response rate and the quality and 
detail of the responses.  However, the greater the amount 
of personal contact, the higher the cost.A-6     In fact, the 
face-to-face interview initiated at homes, while 
eliciting the highest response rate, is generally too 
costly to

_________________
[A-6] choosing a survey technique, careful attention 
should be paid to costs associated with the data processing and
analysis of survey   findings.

A-7


be considered in the evaluation process.  It should only 
be used in conjunction with the random clustered 
household sampling method, where the number of personal 
home interviews to be conducted is small and covers a 
small area.  By significantly decreasing the area in 
which a given size sample lies, the cost of using 
personal home interviews is reduced.

	Where a self-administered questionnaire is used to 
survey a sample, the response rate will inevitably be 
lower than where a face-to-face or telephone interview is 
used.  To improve the response rate it may be desirable 
to allow for a wave of follow-up procedures, such as 
phone calls and postcard follow-up.

	Generally, the self-administered questionnaire is 
the most easily conducted and most cost effective survey 
technique.  Self-administered questionnaires initiated 
onboard or at collection points are most widely 
applicable.  If the questionnaires are short enough to be 
completed by all users while they are onboard and there 
are few standees,  the users should be instructed to 
complete the questionnaires while onboard and return them 
as they leave the vehicle.  If the questionnaires are 
initiated onboard and the number of vehicles on which 
users are surveyed is not large, consideration should be 
given to stationing survey takers onboard each vehicle to 
hand out and collect the questionnaires, give 
instructions, and answer any questions.  If the 
questionnaires are initiated at collection points and the 
number of points at which users exit their vehicles is 
small, consideration should be given to stationing survey 
takers at the exit points to collect the questionnaires.  
The additional expense incurred with this degree of 
personal contact generally pays off (i.e., the response 
rate is high and the cost per completed survey is low).

	Where self-administered questionnaires are too long 
to be completed by all users while they are onboard or 
where there are many standees, questionnaires that are to 
be mailed back should be used.  The response rate for a 
mail back questionnaire will be considerably lower than 
for a questionnaire completed onboard.  This should be 
kept in mind when developing the sampling techniques.

	When questionnaires are sent by mail, a cover letter 
giving instructions and explaining the purpose of the 
survey should accompany-each questionnaire as should a 
self addressed, stamped envelope for mailing back the 
completed questionnaire.  It would also be advisable to 
send out "follow-up" letters a few days after the 
questionnaires are sent out as a reminder to complete the 
questionnaires.	

A-8


	There are situations where it is advantageous to 
conduct personal interviews of users onboard vehicles or 
at employment or activity centers rather than to have 
these users complete self-administered questionnaires.A-7 
Where the total user population to be surveyed is small, 
a high response rate may be needed to obtain the desired 
statistical confidence. In such a situation, a 
self-administered questionnaire may not obtain a high 
enough response rate, while personal interviews of users 
onboard vehicles would. Where there may be considerable 
misgivings about answering a self-administered 
questionnaire, as on a crowded bus or train in some parts 
of some large cities, personal interviews conducted 
onboard vehicles may be the only means of obtaining an 
acceptable response rate.  Where the users being surveyed 
are asked about concepts or behavior that are somewhat 
complex, a personal interview will be much more effective 
than a self-administered questionnaire in eliciting 
usable responses. Handicapped and elderly users may have 
difficult writing and it may be difficult for them to 
respond to a lengthy self-administered questionnaire. It 
should be noted, however, that personal interviews are 
relatively expensive and labor intensive.

	Where samples are selected from service 
registration lists, users can be sent self-administered 
questionnaires by mail.  Where it seems that a very low 
response rate would be obtained with the mail back 
questionnaire, or where a high response rate is 
necessary, the telephone interview would be superior.  
Moreover, sampling bias would be minimized because all of 
the users' telephone numbers would be known from the 
registration lists.A-8

	For surveying non-users, no single technique is 
widely applicable. Where a sample of auto users crossing 
a screenline is surveyed, questionnaires could be sent to 
the auto drivers by mail (from license plate matches) or 
these same auto drivers could be interviewed by 
telephone; or auto users selected directly from the 
traffic stream could be given questionnaires to be 
returned by mail.  For example, where autos are selected 
by license plate matches, auto occupancy would be 
recorded along with license plate number, and mail-back 
surveys mailed

______________________
[A-7] When surveying users at collection points, there 
      generally is not enough time to question them by personal 
      interview.

[A-8] It should be noted, however, that it will not be 
      possible to contact all the persons in the telephone 
      survey sample within the survey time frame. Those not 
      contacted may be a  non-random group, with the result 
      that those who are actually interviewed by telephone may
      no longer be representative of the universe.  
      Therefore, great care must be exercised when 
      sampling by telephone interview.

A-9


out according to auto occupancy.  Those who drove alone 
would be mailed one form; carpool drivers would be mailed 
a set of different forms -- a carpool driver form for 
themselves, and carpool passenger forms to be given to 
those who rode with them.

	In some projects, where autos are also selected by 
license plate matches, the owners of the observed autos 
are surveyed by telephone interview.  No carpool 
passengers are surveyed in this fashion.  Carpool 
passengers can be surveyed directly from the traffic 
stream.  In one situation, many carpoolers assembled at a 
parking lot designated partly for that function.  Before 
each carpool left the lot, each member of the carpool was 
given a self-administered questionnaire to be mailed 
back.A-9

	Where a sample of non-users (and users also) is 
surveyed at specific employment or activity centers, 
those techniques which are applicable for user surveys 
initiated onboard or at collection points should be 
considered. This, in general, means that 
self-administered questionnaires should be used.

	Where a sample of households in the origin portion 
of the project service area, which includes non-users 
(and users), is surveyed, no single survey technique is 
widely applicable.  Questionnaires could be sent to those 
households by mail to be returned by mail, telephone home 
interviews could be conducted, or personal home 
interviews could be conducted where the
sample is selected using the random clustered sampling 
method.

	It is anticipated that the Volpe Center will set up 
a "Survey Notebook" in which will be kept a record of the 
survey experience of the contractors during their 
performance of APTS evaluations.  In order for the Volpe 
Center to maintain this notebook, the contractor will 
supply the Volpe Center with a copy of the survey form, 
information on universe size, sample size, cost, and 
response rate, and reasons associated with non-response.

A.4 SURVEY DESIGN PRINCIPLES

	It is apparent that, because different surveys are 
directed at different survey universes using different 
sampling sources and different techniques, surveys will 
vary in content and length.  Nonetheless, all surveys 
should have the same basic organization, sequence, and
____________________________
[A-9] Some carpool drivers might have been surveyed twice 
      if their license plates had been recorded.

A-10


wording of standardized questions.  This section presents 
basic principles on survey organization, length, question 
sequence and wording, and standardized questions that 
should be followed in designing the survey instrument.

A.4.1 Organization

There should be four elements in all surveys, 
whether user or non-user.  They are in order of their 
appearance in a survey:

(1) Introduction - This is a brief statement of the 
    survey's purpose and potential utility and guarantees 
    the respondent's anonymity.  It will be verbally 
    delivered if an interview technique is selected, or 
    will be printed at the beginning of a 
    self-administered questionnaire.

(2) Behavioral and Attitudinal Measures - These refer to 
    the set of questions specifically measuring the 
    survey's objectives, such as modal shift, 
    satisfaction with level of service, etc.

(3) Social and Demographic Measures - These are measures 
    of the respondent's characteristics which are 
    important in interpreting responses to behavioral and 
    attitudinal measures.  Transition to this section of 
    a survey needs to be prefaced by either a verbal or 
    written explanation, as appropriate, such as "Now we 
    need to know a little about you..."

(4) Closing Statement - This is a brief expression of 
    thanks to the respondent for participating, with some 
    indication of the importance of the eventual 
    utilization of his responses, and a request for any 
    additional comments or observations from the 
    respondent.

A.4.2 Length

	The overall length of the survey depends on the 
particular objectives of the survey and the survey 
techniques used.  In general, surveys which are to be 
completed onboard transit vehicles and at employment and 
activity centers should be shorter than those surveys 
completed at home, since they are being administered to 
respondents in a less comfortable and relaxed 
environment.

	Self-administered questionnaires which are handed 
out should be limited in length to one side of a sheet of 
paper or a large postcard. Surveys which are to be 
completed onboard transit vehicles and at employment and 
activity centers (whether in interview or 
self-administered

A-11


format) should be shorter than surveys which can be 
filled out at the respondent's convenience and returned 
by mail.  Moreover, they should be short enough so as not 
to delay the respondent in his trip or current activity.

	The length of surveys which are completed in the home 
varies depending on the method of administration.  
Telephone surveys should be fairly short since it is 
difficult to retain the respondent's attention for any 
longer period given the impersonal nature of the contact.  
Self-administered mail-back questionnaires sent by mail 
can be longer than self-administered mail-back 
questionnaires handed out because there is more 
opportunity to enlist the respondent's cooperation.  
However, mail-back questionnaires given out by mail 
should not be as extensive as personal interviews 
conducted in the home, since the personal contact is 
lacking which might encourage a longer 
attention/cooperation span on the pan of the respondent.

A.4.3 Question Sequence and Wording
	There are several general principles describing 
question sequence and wording that apply to all 
questions.  First, questions should be arranged logically 
to lead the respondent into the frame of reference of the 
issue under study.A-10   It is recommended, following the 
introductory material, to begin the questionnaire or 
interview schedule with behavioral or attitudinal 
measures of responses to transportation alternatives 
because these relate most closely to the announced 
purpose of the data collection effort.  Social and 
demographic data should be collected near the end of the 
survey instrument, reserving any questions about income 
as near to the end of the survey as possible.A-11
________________________________

[A-10] See pages 26 ff in Federal Highway Administration 
       with Urban Mass   Transportation Administration, Urban 
       Mass Transportation Travel Surveys, for an extended 
       discussion of the basic considerations in designing 
       surveys.  Two very practical descriptions of 
       interviewing and coding guidelines helpful in developing
       format are contained in: Survey Research Center, 
       Interviewer's Manual, Institute for Social Research, 
       University of Michigan, May 1969 and Survey Research 
       Center, A Manual for Coders, Institute for Social 
       Research, University of Michigan.

[A-11] Measures of-income are the most difficult to obtain 
       accurately and arouse the greatest resistance in the 
       respondent.  Sometimes a respondent is asked to point to 
       an amount on a card or circle an approximate amount 
       to lessen the resistance.  However, these items 
       arouse such resistance that they must be at the 
       end of the data collection instrument so the 
       hostility produced will not destroy the rest of 
       the data collection.

A-12


	Questions should be as short as possible and in clear, 
concrete language. Visual format is also important.  In 
self-administered questionnaires, it enhances the 
respondent's likelihood of completing the form, and in 
interview format surveys, it makes the interviewer's task 
faster and easier. Questions should be laid out in a 
fashion that ensures ease of coding and processing 
responses and appears attractive at the same time.  
Fill-in questions should be avoided where possible, 
because they often are difficult to code.  Where they are 
used, responses should be anticipated and precoded to 
reduce costs and enhance consistency.  Coding blocks can 
be left at one side of the survey form and the field 
editor can check to insure that the information is 
transferred.  This procedure makes the survey also 
function as a code sheet.

	The survey should be checked to ensure that it is 
as parsimonious and logical as possible.  There are 
several ways to do this.  First, every question ought to 
be evaluated to ensure that it contains a measure related 
to one of the specific project objectives.A-12    Second, 
advance planning of the data analysis, through the 
construction of dummy tables, will ensure that every 
variable measured contributes to the eventual data 
analysis. Finally, pretesting of the survey instrument 
will identify any questions which, because they are 
confusing to the respondent or of limited use in the 
evaluation, should be changed or omitted.  Pretesting has 
even more far-reaching benefits.  It will uncover any 
procedural problems which may arise during the survey 
process and reveal any problems which are particularly 
characteristic of urban areas, such as a sizable number 
of functional illiterates or foreign speaking respondents 
who cannot complete a self-administered questionnaire or 
a systematic refusal to participate by some sectors of 
the population. The pretest of the survey form must be 
conducted with respondents as identical to the proposed 
survey respondents as possible without contaminating the 
sampling source.

	Finally, all survey questions should be checked 
against the provisions of the Privacy Act of 1914 to 
verify that none of the questions violates any person's 
right to privacy as spelled out
_______________________
[A-11 There are several exceptions to dust guideline. One 
      is the deliberate use of one or two meaningless questions 
      in order to lead the respondent into a particular frame 
      of reference.  This is frequently necessary when seeking 
      information on embarrassing, unusual, highly specific 
      or complicated issues.  This technique will increase      
      the validity of the data subsequently collected.  
      A second exception is measuring respondentís opinions 
      of service features that have not changed as part of 
      a set of questions about respondentsí reactions to 
      improved service features.  This combination of 
      questions will measures if a "halo effect"  exists 
      in terms of respondentsí overall positive evaluation of 
      the mode when only several aspects have been changed.

A-13


in the Act.  It is recommended that the contractors 
familiarize themselves with the provisions of the Act.
 
A.4.4 Standardized Questions

	It will be useful to ensure that the data collected 
in different evaluation projects is consistent in format.  
Fostering consistency means that an economical amount of 
data will yield a maximum amount of information.  
Secondly, consistency facilitates comparisons between 
projects, generating a more universally applicable 
understanding of the responses to transit innovations.  
Finally, and most importantly, developing consistent data 
collection categories based on the U.S. Census will mean 
that results of any survey can be corrected for sampling 
error and potentially extrapolated to any other area.  
This section discusses standardized formats for measuring 
behavioral, attitudinal, and social/demographic 
characteristics.

A.4.4.1 Behavioral Measures

	Selecting questions to measure travel behavior is 
very much influenced by the objectives of a particular 
survey.  Some general suggestions regarding ways to 
collect and code such information to increase consistency 
among surveys will be described.

	The following measures of travel behavior are most 
likely to be asked in almost every survey: transit 
vehicle boarding and alighting points (user surveys 
only), trip origin and destination (all described in 
terms of addresses), trip purpose, and trip start and end 
times.  Additional frequently collected data for surveys 
includes access mode to transit vehicle, when present 
mode was first used for this particular trip, former mode 
used for this particular trip (with some attempt to 
control for external influences, such as a residential 
move), reason for switching mode, fare (user surveys 
only), tolls and parking cost (non-user surveys only), 
frequency of use, access time at origin and destination 
(user surveys only), availability of mass transit 
alternatives, back-up mode, and number of transfers 
required (user surveys only).

	Exhibits A-2 through A-9 are examples of bus, 
automobile driver, and automobile passenger surveys.  
These exhibits, together with the preceding discussion, 
indicate the possible range of information which can be 
collected on travel behavior.  Clearly, the determination 
of which particular items to include in a survey depends 
on the survey objective, desired survey

A-14


length, and circumstances under which the survey is 
conducted.  Furthermore, the specific wording of the 
questions relating to travel behavior depends on the 
method of administering the survey and the overall tone 
of the survey and sequence of questions.

	Exhibits A-10 through A-16 present recommended 
question formats and response categories for the measures 
of travel behavior which are likely to be included in 
most user and non-user surveys.  These recommendations 
are based on a review and evaluation of questions asked 
in past surveys (including Census Journey-to-Work) and 
are directed to the five basic types of surveys (See 
Exhibit A-1).  In designing a survey for a particular 
operational test, the contractor should follow these 
guidelines to the extent consistent with the scope and 
objectives of the survey.  Any significant deviations 
from the recommendations, particularly modifications  of 
suggested response categories, should be explained to the 
Volpe Center in a memorandum accompanying the draft 
survey instrument.

A.4.4.2 Attitudinal Measures

	Attitudinal items will be used in many surveys to 
measure the respondent's evaluation of the APTS 
application and the transit service provided, 
specifically in terms of such characteristics as 
reliability, convenience, attractiveness, and safety of 
alternative modes.  Attitudinal questions may also be 
used, if applicable, to determine what factors have 
influenced a modal change.  Construction of such items 
requires careful design and will lengthen the survey's 
administration time.  Occasionally, attitudinal questions 
may be used to obtain a profile of the community in which 
the transit service is being provided.  An entire survey 
would then be designed explicitly for the purpose of 
determining the opinions of the general population in the 
project service area to such things as the role of 
government, environmental issues, adequacy of 
transportation facilities, and desirability of travel by 
alternate mode. 

	Examples of attitudinal questions appear throughout 
the aforementioned Exhibits A-3 through A-9, and also in 
Exhibits A-17 and A-18.  The set of questions in Exhibit 
A-17 can be used both to measure users' and non-users' 
evaluations of the transit service provided and the 
factors that have influenced their modal choices.   This 
set of questions can also be used to learn about the 
opinions of the general population regarding travel by 
alternate modes.  Note that respondents are asked not 
only for opinions about different travel characteristics 
but also for a

A-15


ranking of the relative importance of these 
characteristics.  The latter set of questions is needed 
to put the respondents' opinions about the different 
travel characteristics into proper perspective.  For 
example, if several respondents indicated that "car" had 
a very high status and "bus" had a very low status, it 
might at first appear that the status of the automobile 
might deter the use of bus transit.  However, the 
responses would be considerably less significant if these 
same respondents indicated that the "status" travel 
characteristics was rather unimportant to them.  The set 
of attitudinal questions in Exhibit A-18 can be used to 
obtain a profile of the community in which transit 
service is being provided.

	There are no specific recommendations for the 
format of attitudinal questions, since the design of such 
questions is entirely dependent on the particular 
attitudes being measured (e.g., opinions of a very 
subjective item or perceptions about items which are 
independently measurable) and on the overall survey 
context.  However, the following discussion presents some 
general informative guidelines regarding the treatment of 
responses to attitudinal questions.

	There are three types of response categories which 
can be used for attitudinal questions: nominal, ordinal, 
and interval scales.  Nominal data consists of mutually 
exclusive categories with no implied rating of the 
responses (e.g., questions with "yes," "no" answers).  
Responses such as "like very much," "dislike," "dislike 
very much" represent ordinal level data, with an implied 
rank ordering.  Interval data involves the use of 
numerical scales (e.g., asking people to indicate their 
opinions on a scale of 1 to 5).  Since interval scales 
require prior validation and careful application, it is 
recommended that attitudinal questions be limited to 
nominal or ordinal response categories.  Moreover, it is 
recommended that the survey data be represented in the 
form of frequency distributions, rather than statistics 
such as means which have an implied ranking.

A.4.4.3 Social and Demographic Measures

	The inclusion of certain social/demographic 
questions in surveys serves the dual purpose of (1) 
providing data on respondent characteristics which might 
show a correlation (perhaps even a causal relationship) 
with measured behavioral attributes, and (2) providing 
data about respondents which can be used in conjunction 
with Census data to check survey accuracy, determine 
non-response bias, and extrapolate survey findings to 
other areas.

A-16


	The amount and nature of social/demographic 
information collected depends on a number of factors, in 
particular, the desired length of the survey and the 
extent to which the data will be correlated with 
behavioral data and used for extrapolation purposes.  It 
is recommended that the following items be included in 
every survey: respondent's sex, age, household income, 
the number of autos in the respondent's household, and 
availability of an auto for the particular trip(s) made 
on project service (user surveys only).  Depending on the 
survey objectives, scope, and administration format, the 
following are some of the additional items which might be 
included: whether the respondent has a driver's license, 
the general (regular) availability of an auto for a 
particular trip type (e.g., work, educational level 
completed, occupation, and length of residence and 
employment at present location).

	Examples of questions on social/demographic 
variables appear throughout Exhibits A-3 through A-9.  
Exhibits A-l9 through A-27 present the recommended 
question format and response categories for most of the 
social/demographic measures listed above.  It is 
considered important to collect and code this type of 
data in categories which are equivalent to, or 
collapsible into, Census categories, so as to facilitate 
comparisons with the same type of Census data for the 
survey area (for accuracy check purposes),A-13 or to permit 
the use of other types of Census data to amplify survey 
findings (with the collected data serving as a bridge 
between the survey population and the Census population).  
Special purpose surveys may require a greater amount of 
detail about a particular social/demographic measure, but 
the stratification should be compatible with commonly 
used Census breakdowns.A-14

A.5 NON-RESPONSE BIAS

	Use of the guidelines presented in this Appendix to 
design and execute a survey does not insure that the 
responses obtained will accurately reflect the 
characteristics, travel behavior, and/or attitudes 
towards the operational field test of the entire sample 
selected even though the sample itself is unbiased and 
totally representative of the population from which the 
sample was
_________________________________
[A-13] Census tract or block data on family income will 
       be a good check on reporting accuracy.

[A-14] See U.S. Census, Volume I: Characteristics of the 
       Population. Part II, Appendix B for a detailed 
       discussion on the format of questions.  See also 
       "1980 Census User Guide," U.S. Department of 
       Commerce, Bureau of the Census, Washington, DC, 
       June, 1983.

A-17


selected.  It is possible that the characteristics, 
behavior, and attitudes of the part of the sample that 
did not respond to the survey are different from those of 
the part that did respond, hence producing non-response 
bias.

	Pretesting of the survey instrument may or may not 
reveal this problem when it exists.  Even if pretesting 
does reveal the problem, there may be no effective means 
of eliminating it.  This is especially true if there is a 
systematic refusal to participate in a survey by certain 
segments or personality types in the population.  It is 
recommended here that an attempt be made in every survey 
to determine whether or not non-response bias exists and 
how it might affect the validity of results.

	There are no specific guidelines for ascertaining 
the existence of non-response bias. In general, 
non-respondents can be reached with a very short survey 
containing but a few key questions that is administered 
with considerable personal contact.  Where 
non-respondents cannot be identified, the special survey 
would be given with the regular survey to a part of the 
sample.  Many of those who do not respond to the regular 
survey will respond to the special survey.  There, 
non-respondents can be identified after the regular 
survey has been completed, only they would be given the 
special survey. The responses of respondents and 
non-respondents to the few key questions can then be 
compared to determine whether the responses of 
respondents and non-respondents are significantly 
different, and therefore, whether non-response bias 
exists.

	The contractor should attempt to devise a specific 
methodology for determining whether non-response bias 
exists in the survey responses obtained from the surveys 
being conducted.

A.6 INTERVIEWS WITH TRANSPORTATION AGENCY PERSONNEL

	There are situations where it may be useful to 
conduct interviews with transit company personnel (e.g., 
drivers, dispatchers, mechanics, management personnel 
from the agency operating the project service).  In some 
cases, such interviews could be used to develop ideas for 
questions and sets of responses for surveys of users and 
non-users.  This is useful in situations where changes 
are being hypothesized, and agency personnel could give 
their opinions and insight on measuring these potential 
changes.

	In other cases, such interviews could be used to 
check the validity of collected data and survey 
responses.  In some very specific cases, such interviews 
could provide first-hand data on

A-18


certain APTS applications.  For example, drivers and 
mechanics could provide information on the operating and 
maintenance characteristics of smart vehicle and smart 
card systems.  Management could provide insight into the 
enforcement problems associated with high occupancy 
vehicle lanes. Dispatchers could provide insight into the 
operating characteristics of an automated vehicle 
location system.  The situations discussed are not meant 
to be all-inclusive.  No specific guidelines have been 
put forth.  It is up to the contractor to decide whether 
interviews with transit company personnel would provide 
information needed to perform the particular evaluation, 
and to design the appropriate survey technique.  
Individual interviews and focus groups are practical 
methods of obtaining information from agency personnel.

A.7 REFERENCES

	The following are considered to be excellent 
references on the subject of survey execution, 
experimental design, and associated issues, concepts, and 
techniques:

(1) Campbell, D.T., and J.C. Stanley, Experimental and 
    Quasi-Experimental Design for Research, Rand McNally and 
    Company, Chicago, 1963.

(2) Mohr, L., Impact Analysis for Program Evaluations, 
    Brooks-Cole,  Monterey, 1988.

(3) General Accounting Office, Designing Evaluations, 
    Washington, DC, May   1991.

A-19


A-20


			    EXHIBIT A-2

			On-Board Bus Survey
		KATY TRANSITWAY TRANSIT USER SURVEY

This survey is being undertaken by the Texas 
transportation Institute, the Texas State Department of 
Highways and Public Transportation and METRO in order to 
obtain information about your use of the Katy Transitway.  
Please take a few minutes to answer the question below 
and return this form to the survey taker before leaving 
the bus.

1.  What is the purpose of your bus trip this morning?  
    _____Work   ______School   ____Other

2.  What is the Zip Code of the area where this trip began? 
    (For example, if  this trip began from your home this 
    morning, you would list your home Zip Code._____________

3.  What is your final destination on this trip  _____Downtown  
    ___Galleria/City Post Oak/Uptown  ___Texas Medical Center 
    ___Greenway Plaza ___Other (specify Zip Code____________)

4.  Have you ever carpooled or vanpooled on the transitway?  
    _____Yes, carpooled  ____Yes, vanpooled  ___No

5.  How important was the opening of the Katy Transitway in 
    your decision to ride the bus?
    ____Very important ____Somewhat important ____Not important

6.  If the Katy Transitway had not opened, would you be 
     riding a bus now?
    ____Yes   _____No   ____Not sure

7.  How many minutes, if any, do you believe this bus presently 
    saves by  using the Katy Transitway instead of the regular 
     traffic lanes?  
    _____Minutes in the morning  _____Minutes in the evening

8.  How long have you been a regular bus rider on the 
     Katy Transitway?______

9.  Does your employer pay for any part of your bus pass?
    ____Yes, may employer pays $_____toward the cost of 
    my bus pass and I   pay $___
    ____No I pay the entire amount

10. Was a car (or other  vehicle available to you for this trip?
    (check one)
    ___No, bus was only practical means
    ___Yes, but with considerable inconvenience to others
    ___Yes, but I prefer to take the bus

11. Before you began riding a bus on the Katy Transitway, 
    how did you normally make this trip? (check one)
    ___Drove alone  ___Rode a park-and-ride bus on the 
                        regular freeway lanes
    ___Carpooled    ___Rode a regular route or express bus
    ___Vanpooled    ___Did not make this trip prior to using 
                       the Katy Transitway
	            ___Other (specify_________________________)

12. Do you feel that the Katy Transitway is, at present being 
    sufficiently utilized to justify the project?
    ____Yes   ____No   ___Not sure


13. What is your... Age?____  Sex?___  Occupation?____

14. What is the last level of school you have completed?_______
    
Comments:______________________________________________________
    
_______________________________________________________________
    
_______________________________________________________________
    
_______________________________________________________________


		THANK YOU FOR YOUR COOPERATION

A-21


			EXHIBIT A-3

		  Carpool/Vanpool Survey

              
Click HERE for graphic.             Metropolitan Transit Authority
			         500 Jefferson Street
				 P.O. Box 61429
				 Houston, Texas 77203-1429

				 713-739-4000
Dear Carpooler/Vanpooler

Your vehicle was observed traveling eastbound of the Katy 
Transitway the week of September 11.  Since you have 
first -hand knowledge of the transitway, we need your 
help in a special study being conducted by the Texas 
Transportation Institute, a transportation research 
agency of the Texas A & M  University System.  Because 
the Katy Transitway is one of the first transitways to 
operate in Texas, it is extremely important that we 
determine what effect it has had on your travel.

Please take a few minutes to answer the enclosed 
questionnaire.  Your answers will provide valuable 
information concerning carpooling/vanpooling on the Katy 
Transityway.  Because of the small number of poolers 
contacted, your specific reply is essential to ensure the 
success of the project.  All information you provide will 
remain strictly confidential.

Your cooperation and timely return of the completed 
questionnaire in the enclosed postage-paid envelope will 
be greatly appreciated.  Thank you for your time and 
assistance in this important undertaking.


				METRO


Enclosures

A-22


			EXHIBIT A-3 (cont.)
		KATY TRANSITWAY CARPOOL/VANPOOL SURVEY

Undertaken by the Texas Transportation Institute, The 
Texas A & M University System in cooperation with the 
Texas    Department of Highways and Public 

1.  Is your vehicle a carpool or a vanpool?  
    ___Carpool   ___Vanpool

2.  What is the  primary purpose of your a.m. carpool/vanpool 
    trip?   ____Work    ____School     ____Other

3.  How many members are regularly in your carpool/vanpool 
    (including  yourself)_____________

4.  Who makes up your carpool/vanpool group?  
    ___Family Members  ___Neighborhood friends  ____Co-Workers

5.  Does your carpool/vanpool use a park-and-ride or 
    park-and-pool lot as a staging area?
    ___Yes (please specify which lot you typically use_______)  
    ___No

6.  How long have you been a regular user of the Katy 
    Transitway?___________

7.  Which transityway entrance do you normally use to access 
    the Katy  Transityway in the morning?
    ___I-1-West of SH6   ____Addicks Park-and-Ride Flyover Ramp

8.  What time do you normally enter the transitway in the 
    morning?______a.m.

9.  What is your a.m. carpool/ destination? ____Downtown  
    ___Galleria/City Post Oak/Uptown   ____Greenway Plaza 
    ___Texas Medical Center   ___Other (specify Zip Code______)

10. When did you join your present carpool/vanpool? 
    Month____  Year_____

11. How important was the Katy Transitway in your decision to 
    carpool/vanpool?
    ___Very important  ___Somewhat important  ___Not important

12. If the Katy Transitway had not opened to carpools/vanpools, 
    would you  be carpooling/vanpooling new?
    ___Yes  ____No   ___Not sure

13. Prior to carpooling/carpooling on the Katy Transitway, how 
    did you normally make this trip?
    _____On the transitway
          _____Bus     _____Vanpool    _____Carpool

    _____On the Katy Freeway general purpose lanes
         _____Bus     _____Vanpool    _____Carpool
    
    _____On a parallel street or highway (Street Name ________)
          _____Bus     _____Vanpool    _____Carpool

    _____Did not make this trip

14. How many minutes, if any, do you believe your 
    carpool/vanpool savers by using the Katy Transitway instead 
    of the regular traffic lanes?  
    _____Minutes in the morning
    _____Minutes in the evening

15. Do you feel that the Katy Transitway is, at present, 
    sufficiently  utilized to justify the project?
    ____Yes    ____No    ____Not sure

16. What is your.....  Age?____    Sex?_____  Occupation?_____

17. What is the last level of school you have completed?______

18. What is your home Zip Code?_____________
    
    We would appreciate your additional comments:______________
    
    ___________________________________________________________
    
    ___________________________________________________________


		THANK YOU FOR YOUR COOPERATION

Please return this form at your earliest convenience in 
the postage-paid envelope provided

A-23


			EXHIBIT A-4

		Freeway Motorist Survey


 
Click HERE for graphic.	         Metropolitan Transit Authority
				 500 Jefferson Street
				 P.O. Box 61429
				 Houston, Texas 77206-1429
			
				 713-739-4000

Dear Motorist;

Your vehicle was observed traveling eastbound on the Katy 
Freeway between 6:00 and 9:00 a.m. the week of October 9.  
Since you have first-hand knowledge of traffic conditions 
on the Katy Freeway, we need your help in a special study 
being conducted by the Texas Transportation Institute, a 
research agency of the Texas A & M University System.

To help serve the travel demand, the State Department of 
Highways and Public Transportation and the Metropolitan 
Transit Authority have constructed the Katy Transitway 
for use by buses, carpools and vanpools.  Vehicles using 
the transitway travel inbound toward downtown in the 
morning and outbound in the afternoon.  The Katy 
Transitway has been constructed within the median of the 
freeway and is protected from other traffic by concrete 
barriers.  The location of the transitway in the median 
has not reduced the number of general traffic lanes 
available to motorists.

Because of the Katy Transitway is one of the first 
transitways to operate in Texas, we need your help to 
determine how it is working.  Please take a few minutes 
to answer the enclosed questionnaire.  The questions on 
this survey concern your routing trips made on the Katy 
Freeway in the morning from 6.00 a.m. to 9.00 a.m.  
Because of the small number of motorists contacted your 
specific reply is essential to ensure the success of the 
project.  Your answers will remain strictly confidential.

Your cooperation and timely return of the completed 
questionnaire in the enclosed postage-paid envelope will 
be greatly appreciated.  Thank you for your time and 
assistance in this important undertaking.


			METRO


Enclosures


A-24


			EXHIBIT A-4 (cont.)

		KATY FREEWAY MOTORIST SURVEY

	
1.  What was the purpose of your trip?  
    ___Work  ____School   ____Other

2.  What are your reasons for driving your car on the 
    freeway maintains rather then traveling in a high-occupancy
    vehicle on the transitway?

    ___Need car for job
    ___Car is more convenient and flexible
    ___No convenient bus, vanpool or carpool available
    ___Work irregular hours
    ___Other (specify_________________________________________

3.  How many days per week do you normally make this trip?_____

4.  How do you usually make this trip? 
    ____Drive alone    ____Vanpool   ___METRO regular route or 
                                        express bus
    ____Carpool        ____METRO park-and-ride bus 
                       ____Other (specify______)
         
5.  How many people (including yourself) were in your vehicle 
    for this trip?_________________________

6.  Which on-ramp did you use to enter the Katy Freeway for 
    this trip?______
 
 
7.  What was the destination of you trip?
    ___Downtown ___Texas Medical Center     ___Other (specify 
                                               Zip Code below)
					    _____________
    ___Greenway Plaza   ___Galleria/City Post Oak/Uptown

8.  Based on your observation of the number of vehicles 
    currently using the Katy Transitway, do you feel that 
    it is being sufficiently utilized?  
    _____Yes   ____No     _____Not sure

9.  Based on you perception of the number of persons currently 
    being moved on the Katy Transitway, do you feel that it is 
    being sufficiently utilized?
    _____Yes   _____No    _____Not sure

10. Do you feel that the Katy Tansitway is a good 
    transportation improvement?

11. What is your....   Age?____    Sex?_____  Occupation?______

12. What is the last level of school you have completed?_______

13. What is your home Zip Code?________________
    We would appreciate your additional comments: _____________
    
    ___________________________________________________________
    
    ___________________________________________________________
    
    ___________________________________________________________

		THANK YOU FOR YOUR COOPERATION

Please return this form at your earliest convenience in 
the postage-paid envelope provided.

A-25


		  	   EXHIBIT A-5

			BUS RIDERS SURVEY

  	IF YOU HAVE ALREADY COMPLETED THIS SURVEY, PLEASE 
RETURN THIS QUESTIONNAIRE TO THE SURVEYOR WITHOUT FILLING 
IT OUT.

	The purpose of the following questions is to 
evaluate Tri-Metís new fare collection system.  Your 
answers will help Tri-met understand how well the new 
fare collection system is working for widers like you.

	Since you are part of a relatively small number of 
riders being surveyed, your answers are very important to 
the accuracy of this study.  Tri-Met has hired an outside 
research firm to gather this information.  You can be 
assured that the information you give is confidential, 
and will only be used in combination with the answers 
from others riders.

	We would like you to complete the white part of the 
survey while on the bus and return it to the surveyor or 
place it in the box near the rear door.   The yellow 
portion is to be completed as soon as possible and mailed 
postage free to Tri-Met.

THANK YOU FOR YOUR TIME AND HELP.

1.  How many bus trips on the average do you usually take 
    each week for each  of the following trip purposes?
    (PLEASE COUNT EACH DIRECTION AS SEPARATE TRIP.) 
    (Write your answer on  the line.  Put "0" in none
           NUMBER OF                    NUMBER OF
    _______WORK TRIPS              _____SCHOOL TRIPS
           NUMBER OF                    NUMBER OF
    _______SHOPPING TRIPS          _____SOCIAL/RECREATION TRIPS

2.  At what time do you most often ride the bus? (Circle 
    the one number next to your answer.)
   1.  WEEKDAYS: RUSH HOUR      3.  WEEKDAYS: EVENING/NIGHT
       (7-9 a.m. & 4-8 a.m.)          (6 p.m. - 7 a.m.)
   2.  WEEKDAYS: MID-DAY        4.  SATURDAY OR SUNDAY
       (9 a.m. - 4 p.m.)              (ALL DAY)

3.  What three bus lines do you ride most often?
    NUMBER LINE NAME

    ____  ______________
    ____  ______________
    ____  ______________

4.  How do you usually pay your fare? (Circle the number 
    under the proper column)
          CASH                         BUS TICKET               

   	1 $  .75 (1- or 2- zone)    1 $  5.00 (1-zone) 
   	2 $ 1.00 (3-zone)           2 $  6.50 (2.zone)    
   	3 $ 1.25 (All zone)         3 $  9.00 (3-zone)    
   	4 $  .50 ( Youth)           4 $ 11.00 (All zone) 
   	5 $  .25 (Honored Citizen)  5 24-Hour (All zone)   
   	6 Other                     6 Other              

         PASS

	1 $23 (1- or 2- zone)
	2 $32 (3-zone)
	3 $40 (All zone)
	4 $15 (Youth)
	5 $ 6 (Honored Citizen)
	6 Other
   IF YOU PAY CASH FARES, PLEASE GO TO QUESTION #7

5.  Where do you usually buy your pass or bus tickets? 
    (Circle the one  number next to your answer.)
    1  DRUG STORE                         5 PLACE OF WORK
    2  7-ELEVEN STORE                     6 BY MAIL FROM TRI-MET
    3  BANK OR SAVINGS & LOAN OFFICE      7 SCHOOL
    4  TRI-MET CUSTOMER ASSISTANCE OFFICE 8 OTHER______________
                                                (PLEASE SPECIFY)

6.  Are ticket and pass outlets more or less convenient 
    for you than before  self-service fare collection?
    1  MORE CONVENIENT
    2  SAME
    3  LESS CONVENIENT
    4  DONíT KNOW

7.  How much discount, if any, do you think people should 
    get for purchasing   ten-ride tickets in advance?
    1 NO DISCOUNT                      4 20% (or $ 1.50 
                                          on ten 2-zone rides)
    2 5% (or 37 on ten 2-zone rides)  5 DONíT KNOW
    3 10% (or 75 on ten 2-zone rides) 

A-26


		EXHIBIT A-5 (cont.)

8.  Please circle the rating number below which best describes 
    your opinion of the following statements regarding fare 
    collection.
                                STRONGLY     UNDECIDED STRONGLY
                                 DISAGREE                AGREE

    a. it is a bother to have the       1    2    3    4     5
       correct change.   
    b. i donít like waiting while other
       people search for their fare.    1    2    3    4     5
    c. i am uncertain about time limits
       and when I should pay extra fare.1    2    3    4     5
    d. iím uncertain about where zone 
       boundaries are and when i should
       pay extra fare.                  1    2    3    4     5
    e. I have trouble understanding the 
       information printed by the machine
       on my ticket.                    1    2    3    4     5

8a. What problems, if any, do you have with the method of 
    collecting fares? 
    (Write "none" if you have no problems.)

9.  How many times in the last 30 days has your fare been 
    checked by a TRI-MET Fare inspector?________________

10. Do you think fares should be checked more or less often?
    1  MORE OFTEN
    2  THE SAME
    3  LESS OFTEN
    4  DONíT KNOW

11. Do you think more people or fewer people pay the correct 
    fare with self-service fare than with the old method of 
    collecting fares.
    1  MORE PAY CORRECT FARES
    2  THE SAME
    3  FEWER PAY CORRECT FARES
    4  DONíT KNOW

12. With the new equipment and rear-door boarding, is getting 
    an and off the bus faster or slower for you than with the 
    old fare collection system?
    1  FASTER
    2  THE SAME
    3  SLOWER
    4  DONíT KNOW

13. In general, do you find self-service fare collection 
    more or less confusing than the  old method of collecting 
    fares?
    1  MORE CONFUSING
    2  THE SAME
    3  LESS CONFUSING
    4  DONíT KNOW

15. Are you:
    1   MALE            2  FEMALE

16. What is your age?
    1 15 OR UNDER          4 45 TO 64
    2 16 TO 24             5 65 OR OVER
    3 25 TO 44

17. What was your approximate family income in 1982?
    1  UNDER $ 5,000                 4 $ 15,000 TO $ 24,999
    2  $  5,000 TO $  9,999          5 $ 25,000 OR OVER
    3  $ 10,000 TO $ 14,999
 

AGAIN THANK YOU! PLEASE TEAR OFF THE WHITE FORM AND 
RETURN IT TO THE PERSON WHO GAVE IT TO YOU OR PUT IT IN 
THE BOX NEAR THE REAR DOOR.  PLEASE FILL OUT THE YELLOW 
FORM AS SOON AS POSSIBLE AND MAIL (POSTAGE-FREE) TO TRI-
MET WE APPRECIATE YOUR HELP!

A-27


A-28


A-29


A-30


A-31


A-32


A-33


A-34


A-35


		 	   EXHIBIT A-10

		RECOMMENDATIONS FOR QUESTIONS ON BOARDING
			AND ALIGHTING POINTS
		    (For User Surveys Only)

QUESTION FORMAT

1. "Where did you board this (vehicle)?"
    __________________________________
    Nearest Street Intersection

2. "Where will you (did you) get off this (vehicle)?"
   ________________________________________________
   Nearest Street Intersection

RESPONSE CATEGORIES

Respondent should specify nearest street intersection.  
Coders can then translate street address to codes 
representing bus stops or, if a less fine-grained 
analysis is required, zonal codes.

COMMENTS

Question format contains parentheses to indicate where 
site-specific codes might be substituted.


_________________________
*The use of "will you" or "did you" depends on whether 
 the survey is filled out while the respondent is on 
 board the vehicle or completed later and returned by mail.

A-36


			EXHIBIT A-ll
	RECOMMENDATIONS FOR QUESTIONS ON TRIP ORIGIN

QUESTION FORMAT

la. "Where did this trip begin?"
     ___________________________
     Street Address, City, Zip Code


2. "Is this place --- (check one)"

    .  Home
    .  Place of employment
    .  School
    .  Retail/commercial establishment
    .  Social-recreational facility
    .  Medical facility
    .  Personal business site
    .  Other (specify)


RESPONSE CATEGORIES
Use categories given under "Question Format" or, if the 
main purpose of the question is to distinguish work vs. 
nonwork trips, use the following categories:
		
		.  Home
		.  Place of employment
		.  Other
Respondent should specify street address.  Coders can 
then translate street address to zonal codes, or 
addresses can be decoded using the Census Bureau's TIGER 
files ant address program.

A-37


			EXHIBIT  A-12

	RECOMMENDATIONS FOR QUESTIONS ON TRIP DESTINATION

QUESTION FORMAT

1. "What is (was) the final destination of this trip?"
    _______________________________________________
     Street Address, City, Zip Code

2. "Is this place --- (check one)"
	.  Home
	.  Place of employment
	.  School
	.  Retail/commercial establishment
	.  Social-recreational facility
	.  Medical facility
	.  Personal business site
	.  Other (specify)

RESPONSE CATEGORIES

Use categories given under "Question Format" or, if the 
main purpose of the question is to distinguish work vs. 
nonwork trips, use the following categories:

	.  Home
	.  Place of employment
	.  Other

Respondents should specify street address.  Coders can 
then translate street addresses to zonal codes, or 
addresses can be decoded using the Census Bureau's TIGER 
files ant address progress.

Another option, for interview surveys, is to have the 
interviewer show the respondent a map with numbered zones 
super-imposed, and ask the respondent to identify the 
destination zone.

COMMENTS

The question classifying nature of trip destination, in 
combination with a question classifying nature of trip 
origin, is a better indication of trip purpose than a 
question explicitly asking trip purpose, which can be 
confusing to persons making multiple-purpose trips.

A-38


			EXHIBIT A-13

   RECOMMENDATIONS FOR QUESTIONS ON TRIP START AND END TIMES

QUESTION FORMAT

1. "What time did you begin this trip?" 
             A.M. 
    _________P.M.

2. "What time did you arrive at your destination?" 
             A.M. 
    _________P.M.

RESPONSE CATEGORIES

Depending on the survey objectives, beginning/ending 
times can be used as given to compute total trip times, 
or they can be coded using categories such as A.M. peak, 
midday, P.M. peak, nighttime.



_____________________
* With personal interviews onboard vehicles, it is not 
possible to ask time of arrival at destination.

A-39


			   EXHIBIT A-14
	RECOMMENDATIONS FOR QUESTIONS ON ACCESS MODE
			TO TRANSIT VEHICLE

QUESTION FORMAT

1. "How did you get from the place where this trip began 
    to the (place) where you boarded this (vehicle)?"

2. "How will you (did you) get to your destination after 
   leaving this (vehicle)?"*


RESPONSE CATEGORIES

     	U.S. Census				Recommended
	Private auto, driver			Park 'n' ride
	Private auto, passenger			Carpool
                                        	Kiss Ďní ride

	Subway, elevated train, railroad Same
	walked					Same
	Worked at home				Omit
	Taxi	                                Same
	Bicycle or motorcycle}	Other		Other

COMMENTS

Question format contains several parentheses to indicate 
where site-specific modes and locations eight be 
substituted to make the question more relevant. The same 
principle applies to the recommended response categories; 
the above list is suggestive and needs to be adjusted to 
site-specific concerns such as measuring the number of 
auto passengers for evaluation of a carpool encouragement 
program.

*The use of "will you" or "did you" depends on whether 
the survey is filled out while the passenger is on board 
the vehicle or is completed later and returned by mail.

A-40


			   EXHIBIT A-l5
		RECOMMENDATIONS FOR QUESTIONS ON WHEN
		     PRESENT MODE WAS FIRST USED

QUESTION FORMAT

For User Surveys

	"When did you begin to use (specify services 
	regularly for the trip you are now taking?"

	. not applicable, or
	__________month_________year

For Non-user Surveys

	"When was the last time you regularly used (specify 
	service) for the trip you are now taking?"

	. not within the last 5 years, or
      	________month________year

For Non-user Surveys in Which Carpoolers and Those who 
Drove Alone are Given Separate Questionnaires

For Carpoolers:

	"When did you begin to regularly use this carpool 
	for the trip you are now taking"

	.  not applicable, or
     	________month_______year

For Those Who Drove Alone:

	1. "When was the last time you regularly used 
	(specify service) for the trip you are now taking?"

	. not within the last 5 years, or
      	_________month________year

	2. "when was the last time you regularly used a 
	carpool for the trip you are now taking?"
	
	. not within the last 5 years, or
     	_________month__________year

A-41


			EXHIBIT A-16
    RECOMMENDATIONS FOR QUESTIONS ON FORMER TRANSPORTATION MODE

QUESTION FORMAT

How did you make this trip before (specify 
service) was available?"

RESPONSE CATEGORIES

	U.S. Census				Recommended

	Private auto, driver			  Same (indicate
		                               total number of
		                               occupants)
	Private auto, passenger		          Same (indicate
		                               total number of
		                               occupants)
	Bus or streetcar			Same
	Subway, elevated train, railroad	Same
	Walked					Same
	Worked at home				Omit
	Taxi	                                  Same
	Bicycle, motorcycle 			Other
	Other

COMMENTS

The responses will have to be tailored to include 
particular local transportation alternatives.  For 
instance, it might be desirable to obtain information on 
former auto occupancy levels for ex-drivers/passengers.

A-42


			EXHIBIT A-117

   SET OF ATTITUDINAL QUESTIONS ON TRAVEL BY TRANSIT AND AUTO

YOUR OPTIONS OF TRAVEL BY DIFFERENT MODES

4. On the scales below, please indicate your general 
   opinion of car and bus travel for local travel.  Base 
   your opinion on what you have experienced or have heard 
   about local travel by each mode from the userís 
   viewpoint.  Even though you may not use the bus, you 
   probably have some perceptions of what this form of 
   travel is like, you donít need to have tried something in 
   order to be able to express some general opinions.

   To indicate your opinion, look at the descriptive scales 
   below, each of which allows for a range of opinions on a 
   particular characteristics, such as COMFORT.  Then, mark 
   what you consider to be the single most appropriate 
   description on each scale by circling the relevant 
   number.  For instance, on the COMFORT scale, if you 
   thought cars were a very comfortable forms of travel for 
   local travel, you would circle "1"  on the scale on the 
   line for cars; however, if you thought they were a 
   slightly uncomfortable form of travel, you would circle 
   "4" and so forth.


Click HERE for graphic.

A-43


		 	    EXHIBIT A-18

   SET OF QUESTIONS ON GENERAL ATTITUDES OF THE POPULATION

		YOUR OPINIONS ABOUT LOCAL GOVERNMENT


Everyone has different ideas about the kinds of things 
local government should be most concerned about.  Below 
is a list of different things the government might do.  
Please indicated your feeling about how much the 
government should do of each activity.


                                             About
                               Much  Slightly the  Slightly Much
Government Activities          more     more  same   less  less

I)    Reduced Crime              M       m      s      1      L
ii)   Reduced environmental 
      pollution                  M       m      s      1      L
iii)  Provide low-cost medical 
      care for  all              M       m      s      1      L
iv)   Control population growth
v)    Provide more housing for 
      low to medium income 
      families.                  M       m      s      1      L
vi)   Insure equal opportunity 
      for   women                M       m      s      1      L
vii)  Provide a consumer 
      protection                 M       m      s      1      L
viii) Add to and improve the 
      freeway    system          M       m      s      1      L
ix)   Increase direct aid to 
      the poor                   M       m      s      1      L
x)    Improve bus service and 
      other forms of  public 
      transportation             M       m      s      1      L
xi)   Have more parks and outdoor 
      recreation   areas         M       m      s      1      L
xii)  Improve the public schools
xiii) Reduce taxes               M       m      s      1      L

(a)   Which one of the activities do you feel is the most 
      important for the  to do?  Just give the letter.
				       Most important______

(b)    And which do you feel is the next most important?
			          Next most important______

A-44


Click HERE for graphic.

A-45


			EXHIBIT A-19
	RECOMMENDATIONS FOR QUESTIONS ON RESPONDENTS' SEX

QUESTION FORMAT

A. For Self-Administered Surveys
    
    "Are you --
     .   Male   .  Female 
or
   "Please indicate your sex"

     .  Male    .  Female

B. For Interview Surveys

   Respondentís sex is noted by the interviewer.

A-46


			EXHIBIT A-20
	RECOMMENDATIONS FOR QUESTIONS ON RESPONDENT'S AGE

QUESTION FORMAT

"To what age group do you belong?"

Click HERE for graphic.


Click HERE for graphic.
 
	
COMMENTS

The recommended response categories represent the minimum 
stratification of data to be collected about age.  Age 
responses can be further stratified according to the U.S. 
Census Categories, depending on the survey objectives and 
the expected age distribution of the respondent 
population.

It is important to use the phrase "age group" in all 
questions about age to minimize the respondentís 
resistance to this question.

A-47


			EXHIBIT A-21
	RECOMMENDATIONS FOR QUESTIONS ON RESPONDENT'S INCOME

QUESTION FORMAT


	"What is the combined annual income of all members of 
  	Your household?"

Click HERE for graphic.
 

RESPONSE CATEGORIES  (RECOMMENDED)

	Less than $10,000 
	$10,000 - $19,999 
	$20,000 - $29,999 
	$30,000 - $39,999 
	$40.000 - $49,999 
	$50,000 - $59.999 
	$60,000 - $69,999 
	$70,000 - $79,999 
	$80,000 - $89,999 
	$90.000 - $99,999 
	$100,000 and greater


COMMENTS

The recommended response categories represent the minimum 
stratification of income data.  Responses can be further 
stratified according to the U.S. Census categories, 
depending on the survey objectives and the expected 
income distribution of the respondent population.

For interview surveys, asking a respondent to point to 
one of the above categories on a card facilitates handing 
of this often sensitive question.

It is important to use the word "annual" or "yearly" in 
order to obtain responses on a consistent basis.  
Moreover, if deemed appropriate, the question can be 
phrased to refer to the most recently ended calendar 
year.

A-48


			     EXHIBIT A-22
	RECOMMENDATIONS FOR QUESTIONS ON AUTO AVAILABILITY
			(For User Surveys Only)

QUESTION FORMAT

	"Was a car available to you for this trip?"

RESPONSE CATEGORIES

"Was a car available....?"

	.  Yes, and without inconvenience to others.
	.  Yes, but with inconvenience to others.
	.  No

COMMENTS

Information on the availability of a car for a specific 
trip or time period is the most direct way of determining 
auto availability and its possible influence on mode 
used.


A-49


 			  EXHIBIT A-23
	RECOMMENDATIONS FOR QUESTIONS ON AUTO OWNERSHIP

QUESTION FORMAT

	"How many cars are owned or operated by members of 
	your household?"

RESPONSE CATEGORIES

		U.S. Census		Recommended
		0 cars	               . None, or_______auto (s)
		1 cars
		2 cars
		3 or more cars

			   EXHIBIT A-24
	RECOMMENDATIONS FOR QUESTIONS ON WHETHER RESPONDENT HAS
			DRIVER'S LICENSE

QUESTION FORMAT

"Are you a licensed driver?"

RESPONSE CATEGORIES

	"Are you ...."
    	.   Yes    .  No

A-50


			EXHIBIT A-25

    RECOMMENDATIONS FOR QUESTIONS ON RESPONDENT'S OCCUPATION

QUESTION FORMAT

	1. "Are you______"

	.  Employed
	.  Student
	.  House Spouse
	.  Retired
	.  Other

2. "If you are employed, describe briefly the kind of 
    work you do."
   
   ____________________________________________________
   ____________________________________________________


CODING CATEGORIES FOR QUESTION 2

U.S. Census		           Recommended

  Professional, technical ant		The survey form
       kindred workers		        should contain a
  Managers and administrators,		blank space for an
	  except farm		        open-ended description
  Salesworkers	         	        which can later be
  Clerical and kindred workers		codes using the
  Craftsmen and kindred workers		U.S. Census occupa-
  Operatives, except transport		tional categories.
  Transport equipment operatives
  Laborers, except farm
  Farmers and farm managers
  Farm laborers and farm foremen
  Service workers, except private
    household
  Private household workers

COMMENTS

Question 2 should be included in the survey only when 
there is a very specific reason for using employment 
data.  In order to perform the coding for question 2, it 
is necessary to obtain description of the type of work 
actually done as well as job title.

A-51


			   EXHIBIT A-26
	RECOMMENDATION'S FOR QUESTIONS ON RESPONDENT'S
			EDUCATIONAL LEVEL

QUESTION FORMAT

For All Surveys

	"What is the last grade (or year) of regular school 
	you (he/she) attended?"

 
(asked for each household member in dwelling unit survey)

RESPONSE CATEGORIES

	    U.S. Census			Recommended
    	None				No Formal Schooling
	1-4
	5-7				Grade School
        8
	High School: 1-3		Some High School
		     4			High School Degree
	College:  1-3			Some College
		  4 years or more 	College degree or higher


A-52


			EXHIBIT: A-27

RECOMMENDATIONS FOR QUESTIONS ON LENGTH OF RESIDENCE

QUESTION FORMAT

	"When did you (your household) move to your present 
	residence"

RESPONSE CATEGORIES

	U.S. Census	            Recommended
	1989-1990	    .  Not within the last 5 years, or
	1980-1984           _____________month___________year
	1970-1979
	1960-1969
	l959 or earlier

COMMENTS

The recommended response categories represent the minimum 
stratification of data to be collected about length of 
residence.  Responses can be further stratified (for 
greater than 5 years), depending on the survey objectives 
and the expected residency level distribution of the 
respondent population.

A-53


 			APPENDIX B

		STATISTICAL METHODOLOGY

This Appendix presents guidelines relevant to 
determining appropriate sample sizes for data collection 
as well as the subsequent analyses.

The determination of appropriate sample sizes and 
data analysis requirements is a crucial aspect of 
planning for data collection, since in general this phase 
involves scoping the level of activity related to 
collection of project-specific measures.  Just as failure 
to plan the basic evaluation approach will mean not 
having the proper framework in which to observe and 
evaluate the operational test, failure to plan or 
improper planning of sample size requirements and data 
analysis procedures will threaten the ultimate 
statistical validity and usefulness of project results.  
An insufficient quantity of data, whether due to no 
planning (i.e., haphazard data collection) or to an 
underestimate of needs, will be manifested in the loss of 
potentially valuable analyses and/or a loss in accuracy 
and validity of the analyses based on the data.  On the 
other hand, excessive quantities of data will mean the 
unnecessary expenditure of funds and possibly the 
sacrifice of other data items which could be useful but 
which are beyond a constrained budget.  The intent is to 
obtain an appropriate balance between analysis 
requirements and resource availability. It should be 
remembered that small samples, if they are well planned, 
can yield useful and interpretable data.

B.1 DEFINITIONS

To assure a complete understanding of the concepts 
presented in this Appendix, as well as those identified 
in the references thereto, the following terms are 
identified:

(1) OBSERVATIONAL ENTITY or ELEMENT - An individual item 
    in a set of items or responses, each of which is 
    identifiable by one or more measures. Examples of 
    observational entities are automobiles, vehicles, 
    persons, time periods.

(2) POPULATION or UNIVERSE - A finite or perhaps very 
    large collection of observational entities.  A 
    population is usually a group about which inferences 
    are desired.  Examples of populations would be all 
    those vehicles on a corridor leading to the central 
    business district during AM peak periods, all those 
    persons within 15 minutes access time of the transit 
    system, or all users of a service.

B-1


(3) SAMPLE - A finite subset of observational entities drawn 
    from a population.  Samples can be drawn by appropriate 
    procedures which will permit inferences to the 
    population from which the sample was drawn or they 
    may be obtained by non-controlled devices.  Examples 
    of samples would be some of the vehicles passing a 
    given screen-line during a specific time period, or a 
    subset of those individuals within a service area.

(4) OBSERVATION - One or more measures which describe the 
    observational entities included in the sample either 
    directly or derived from measurements, such as travel times 
    or passenger counts.

(5) POPULATION PARAMETER - A specific descriptive 
    characteristic of a population assumed to be constant at 
    any moment or period in time.

(6) SAMPLE STATISTIC - A summary value obtained from a sample 
    observation, usually descriptive of the sample but desired 
    for purposes of making inferences about the population or 
    changes in the population parameter.

B.2 DATA ANALYSIS DETERMINATION

	It should be evident that a major intent of using 
samples is to make inferences about changes in transit 
system characteristics or in the attitudinal/behavioral 
characteristics of the community being served.

	Before estimating sample size requirements, it is 
necessary to determine the appropriate types of analyses 
to be performed (i.e., What will be done with the data 
once they have been collected?).  Types of statistical 
analyses which can be performed are numerous.  As a 
general guideline, it is essential that the evaluations 
for APTS projects be confined to fairly fundamental types 
of analyses (i.e., involving the calculation of means, 
standard deviations or variances, proportions, ratios, 
and ranges). Suggested statistical techniques for 
performing these analyses are discussed later in this 
Appendix.

	More sophisticated statistical methods, such as 
multiple regression, factor analysis, and discriminant 
analysis may also be applicable in the current generation 
of APTS projects.  As more experience is gained with the 
data collected during these projects, it may be possible 
to institute some of the referenced multivariate 
techniques.

	The use of a simple analytical framework will have 
three main advantages: (1) the results will be expressed 
in numerical terms that have a direct relation to 
specific project

B-2


objectives; (2) the evaluation results will be meaningful 
to a wide audience; and (3) the results of a particular 
project can be more easily compared with those of other 
projects.

	The types of statistical analyses which can be 
performed and the appropriate equations and tables to be 
used in performing these analyses and determining sample 
sizes are presented in an organized, thorough manner in 
M.G. Natrella, Experimental Statistics, National Bureau 
of Standards Handbook 91, August, 1963.B-1   Included in 
this handbook are procedures for estimating average 
performance from a sample, estimating variability of 
performance for a sample, comparing two or more samples 
with respect to average performance or variability of 
performance, characterizing the functional relationship 
between two variances, and comparing samples with respect 
to discrete classifications such as income, mode of 
travel to work, etc.  Two other excellent references are 
given at the end of this Appendix. Since most of the 
specific equations to be employed in dealing with these 
situations are clearly presented in Natrella and other 
commonly used statistics reference books, the remainder 
of this section will be devoted primarily to a discussion 
of some of the statistical considerations by the 
contractor.

	Of the numerous cases presented in Natrella, the 
following basic set of underlying questions is considered 
applicable for APTS projects:

If estimates of population parameters only are required:

(1) What is an estimate for the average value (mean) of 
    the measure (Let X represent the measure)?

(2) What is an estimate for the variability (variance or 
    standard deviation) of the measure?

(3) What is an estimate of the proportion of units that 
    have a given   characteristic?

If comparisons between two groups (e.g., before vs. 
after, test vs. control) are involved:

(1) What is the difference between the average value of 
    the measure, X, for group A and the average value of the 
    measure, X, for group B?
_________________________
[B-1] The contractor is encouraged to obtain a copy of this 
      book, since it is referenced throughout this section of 
      the guidelines as a source for tables, equations and 
      other material.  It is available through the Government 
      printing office and was reprinted in 1983.

B-3


(2) Same question as (1) except applied to the variability of 
    the measure  in groups A and B.

(3) Same question as (1) except applied to proportions of 
    some discrete  measure in groups A and B.

	The same types of questions can be asked when there 
are more than two groups (time periods) involved in the 
comparisons.  Here, however, the methods for analysis 
become more complex, and greater care must be exercised 
in selecting and applying statistical techniques.

	In connection with addressing the question "What is 
the value...?" or "What is the difference...?", it is 
recommended that results be given in terms of confidence 
intervals rather than tests of significance.  By 
presenting a confidence interval (an interval which 
contains the true parameter, or difference between two 
parameters, with a known probability), the decision-maker 
can interpret the magnitude of this interval whether it 
be for an estimate of a population parameter or for the 
difference between two parameters.  On the other hand, if 
a test of significance is used, the interpretation of 
non-significance and significance becomes somewhat more 
difficult in terms of relating these inferences back to 
the project objectives.  In some instances where sample 
sizes are fairly large, differences that can be 
significant from a statistical viewpoint, may have little 
practical significance attached to them.  Statements on 
statistical significance may be made but the practical 
implications must be considered.

	It will generally be adequate for the contractor to 
report two-sided confidence intervals for a stated 
confidence level.

B.3 SAMPLE SIZE DETERMINATION

	As long as appropriate sampling methods are 
applied, the accuracy of a statistic computed from a 
sample will be greater with a larger sample size. 
However, this relationship can be one of diminishing 
returns for very large sample sizes.  Moreover, there is 
a cost, in time and money, which serves as a constraint 
on sample sizes in each APTS project.  The key aspect of 
sample size determination is finding the proper balance 
between desired accuracy and cost: on the one hand, the 
sample should not be so small that the results lack the 
required accuracy; conversely, the sample should not be 
wastefully large.

B-4


	In Section 3.3, variable stratification (the 
categorization of collected data by such factors as time 
of day) was discussed.  It was mentioned that the data 
collection activities should be planned with the finest 
level of stratification consistent with constraints of 
time, cost, and acceptable accuracy and confidence.  It 
is important that this determination of desired level of 
stratification be made as early as possible, since, from 
the statistical point of view, the sampling plans must 
include sufficient data in each category of interest for 
which cross-tabulations are to be performed.  The 
formulas for determining sample size must be applied with 
respect to each category, so that the appropriate 
quantity of data is collected for each one.  Clearly, an 
attempt at further stratification after the data has been 
collected would reduce the accuracy and/or confidence 
associated with these new sub-stratifications.

	The appropriate sample size formula depends on the 
type of statistical analysis to be performed.  Sample 
size formulas applicable for calculating means, 
variances, proportions, etc., are given in the references 
at the end of this Appendix, so the following discussion 
will be somewhat general.  The sample size calculation 
process should be viewed as providing input for the broad 
scoping and planning of the data collection effort.  The 
specific sample size values obtained from the formulas 
should be taken as rough indications of lower limits for 
data collection, rather than as precise targets or 
cut-off points.  Prudent expansion factors should be 
applied to the calculated sample size values so that the 
ultimate amount of usable data (i.e., the net sample size 
after the collection activities and editing) is 
sufficient to yield results with the desired level of 
precision and statistical accuracy, and allows for 
unforeseen stratification.  As data is collected, it 
should be possible to modify sample requirements for 
subsequent phases of a project.

	As has been mentioned earlier, it is desired to 
have results presented in the form of confidence 
intervals.  Determining the sample size for calculating a 
confidence interval requires three input factors:

(1) The desired confidence level,

(2) An estimate of the variability in the-population, and 
(3) The desired precision of the results.

B-5


	The confidence level of a statistical calculation 
(l-) can be defined as the proportion of samples of size 
n for which the calculated confidence interval may be 
expected to contain the true value of the population 
parameter being estimated.  For purposes of obtaining a 
conservative sample size estimate, it is recommended that 
the value  = .05 be used.

	An estimate for variability is usually taken as the 
standard deviation.  It is desirable initially for this 
value to be an overestimate to allow for a conservative 
determination of sample size.  White it is preferable to 
have some prior knowledge about the variability of those 
measures to be collected, Natrella (pages 2-8 to 2-10) 
gives an excellent approach for cases where the true 
standard deviation is unknown.

Determination of an acceptable level of precision 
is perhaps the most difficult input factor.  In the case 
of estimating means, variability measures, and 
proportions, the task is to determine the acceptable 
accuracy, say d, for each confidence interval.  The 
sample size calculated on the basis of a prescribed d and 
 = .05, reflects an acknowledged (permissible) risk that 
5 times in 100 the real precision will be worse than d.  
In the case of estimating the difference between means or 
between other statistics, the analogous task is to 
specify the absolute value of a minimum desired 
detectable average difference .   Here, too, if  = .05, 
then the sample size wilt reflect an acknowledged risk 
that 5 out of 100 times the true difference between the 
two groups being compared will exceed .

	In establishing values for d and   , consideration 
must be given to the problem of trading off the cost vs. 
benefits of increased precision.  The cost of increased 
accuracy can be seen as the marginal amount of time and 
money needed to collect an additional sample unit.  The 
benefits of increased accuracy can be viewed in terms of 
additional confidence in the results of a particular 
project and the consequent willingness of FTA to make 
policy and funding recommendations on the basis of these 
results. Clearly, FTA does not want to encourage cities 
to implement APTS innovations which have only a 
negligible impact on the quality or usage of transit 
service; this would argue in favor of setting relatively 
large values of d and   .   On the other hand, there is a 
desire to learn whatever possible about the effects of 
implementing new techniques; if the minimum detectable 
difference is set too large, the resultant sample size 
may be too small to detect the existence of minor, 
possibly unanticipated changes which might be of 
interest.

B-6


	Working with the cognizant FTA and Volpe Center 
professionals, the contractor should indicate the value 
of d or   selected for each measure to be collected, and 
should explain the rationale for choosing the particular 
value in terms of the cost-benefit considerations 
discussed above.  Issues concerning sample size 
determination and precision are discussed in Sampling 
Techniques, by W. G. Cochran.

B.4 DATA COLLECTION

	Once the minimum sample size for each 
stratification category of each sampled measure has been 
determined using the appropriate formula and the above 
three prescribed input factors, the data collection phase 
can be implemented.  As was mentioned above, the 
contractor should apply a prudent expansion factor to the 
minimum sample size to obtain a target sample size.

	Field observations should be scheduled for a 
sufficient number of days to collect the target quantity 
of sample units.  In most cases, the scheduling of data 
collection will present no particular problems: the 
required number of "representative" days can be 
designated, as well as alternate dates to be used in the 
event of unusual weather conditions or other atypical 
occurrences on the planned dates.  However, there may 
arise a situation where the day-to-day variability is 
known or suspected to be significant in relation to the 
variability within a day.  In this case, arbitrary 
spreading of the data collection phase over several 
consecutive days may adversely affect the inferences to 
be made.  Depending upon the project objectives, it may 
be more appropriate to schedule data collection for 
consecutive weeks on a particular day of the week (the 
most representative day).B-2

B.5 ANALYSIS METHODS

	Since numerous statistical methods are available, 
the balance of this Appendix discusses a family of 
statistical techniques which will be appropriate for 
project analyses.  The measures which will be collected 
and utilized to assess achievements of project objectives 
can be classified
_________________________
[B-2]   The preceding discussion deals with day-to-day 
	variability with a known pattern.  In the unusual 
	situation of day-to-day variability which exceeds 
	within-day variability and does not follow a 
	particular pattern, the target sample size must be 
	calculated according to different procedures, which 
	give a number of sample days  as well as number of 
	sample per day.

B-7


as discrete or continuous.  A discrete measure is one 
which can assume only a fixed and known set of values.  
Examples of such measures would be counts of numbers of 
vehicles and passengers, responses to qualitative 
questions and classifications of survey responses into 
categories such as yes/no. Continuous measures may assume 
(in theory) an infinite set of values.  The accuracy of 
these measures is con 
(ate.html)
Jump To Top