BTS Navigation Bar

NTL Menu


Advanced Public Transportation Systems: Evaluation Guidelines January 1994



			Advanced Public
			Transportation Systems:
			Evaluation Guidelines
			January 1994

Click HERE for graphic.




			Advanced Public
			Transportation
			Systems:
			Evaluation Guidelines

			Final Report
			January 1994


			Prepared by
			Robert F. Casey and John Collura
			John A. Volpe National
   			   Transportation Systems Center
			Kendall Square
			Cambridge, MA 02142

			Prepared for
			Office of Technical Assistance
			Federal Transit Administration
			400 Seventh Street SW
			Washington, DC 20590


			Distributed in Cooperation with
			Technology Sharing Program
			U.S. Department of Transportation
			Washington, D.C. 20590

			DOT-T-94-1 0


			PREFACE

	This document was prepared by the Office of 
Research and Analysis, Volpe National Transportation 
Systems Center, under the sponsorship of the Advanced 
Public Transportation Systems (APTS) Program, Federal 
Transit Administration (FTA) and with the guidance of Mr. 
Ronald Fisher, FTA's Director of Training, Research, and 
Rural Transportation.  The Volpe Center operates under 
the auspices of DOT's Research and Special Programs 
Administration (RSPA).  The major contributors were Mr. 
Robert Casey, RSPA/Volpe Center Operations Research 
Analyst, and Dr. John Collura, RSPA/Volpe Center Faculty     
Fellow and Professor of Civil Engineering at the 
University of Massachusetts, Amherst.  Technical 
assistance also was provided by Ms. Judith Schwenk and 
Mr. Lawrence Labell of the RSPA/Volpe Center and Dr. 
Thomas Horan of the Institute of Public Policy at George 
Mason University.  The summaries of the breakout sessions 
at the recent National Workshop on APTS Evaluations also 
were useful in the completion of the guidelines.  The 
summaries were prepared by Ms. Katherine Turnbull of the 
Texas Transportation Institute, Mr. John Mason of Science 
Applications International Corporation, Mr. Joel 
Markowitz of the Metropolitan Transportation Commission 
(San Francisco), and Mr. Philip Shucet of Michael Baker 
Jr., Inc.

	The preparation of this document was also 
facilitated by prior evaluation work by Mr. Mark 
Abkowitz, Ms. Carla Heaton, Mr. Chester McCall, Mr. 
Howard S. Slavin, and Mr. Robert Waksman as part of the 
Federal Transit Administration's Service and Methods 
Demonstration program.

	The document consists of evaluation guidelines for 
use by contractors responsible for evaluating APTS 
operational tests.  Although these guidelines are 
intended for the APTS Program, their potential 
applicability extends beyond the evaluation of 
FTA-sponsored operational tests to the evaluation of any 
innovative use of advanced technology in public 
transportation.

	It is anticipated that this document will be 
modified periodically to reflect additional experience 
gained in evaluating APTS operational tests.

iii

	
Click HERE for graphic.


TABLE OF CONTENTS

											
							PAGE

1. OVERVIEW OF EVALUATION GUIDELINES 			1

2. BACKGROUND 						3
	
   2.1 Overview of the Evaluation Process 		7
       2.1.1 Evaluation Frame of Reference 		7
       2.1.2 Evaluation Planning 			12
       2.1.3 Evaluation Implementation 			13
       2.1.4 Potential Evaluation Spin-offs. 		15
   2.2 Coordination of APTS Evaluations 		16

3. GUIDELINES FOR PLANNING EVALUATION ACTIVITIES 	19

   3.1 Determination of Site Data Requirements and 
      Sources 						21

  3.2 Determination of Measures and Collection/
      Derivation Techniques Required to Address 
      APTS; Program Objectives and Other Relevant 
      Project Objectives/Issues 			24

      3.2.1 Basic Set of Measures 			25
            3.2.1.1 APTS Costs and Functional 
                    Characteristics 			27
	    3.2.1.2 User Acceptance 			28
            3.2.1.3 System Efficiency and Effectiveness 28
	    3.2.1.4 Impacts 				29
            3.2.1.5 Relationship Between APTS Program 
                    Objectives and the Categories of
                    Measures				29
	    3.2.1.6 Other Objectives and Measures	34
      3.2.2 Data Collection/Derivation Techniques 	34

  3.3 Planning Considerations Relative to Data 
      Collection  and Analysis 				38
      3.3.1 Basic Data Collection/Analysis Design 	39
      3.3.2 Measure Stratification 			43
	    3.3.2.1 Categorization of a Measure Into 
                    Additive Components 		44
            3.3.2.2 Categorization of a Measure 
                    According to Target Market, 
                    Operational, Geographic, 
                    or Time Categories 			48
     3.3.3 Grouping of Raw Data Into Class Intervals    49
     3.3.4 Sampling Requirements			51
     3.3.5 Timing of Data Collection			52

v


4. GUIDELINES FOR PERFORMING EVALUATION ACTIVITIES	55

  4.1 Monitoring/Performance of Data Collection		55

  4.2 Data Reduction, Analysis, and Presentation	59

5. CONTENT AND ORGANIZATION OF REPORTS			75

  5.1 	Evaluation Plan					75

  5.2 Monthly Evaluation Progress Reports		77

  5.3 Interim Evaluation Reports			78

  5.4 Final Summary Evaluation Report			78

  5.5 Quarterly Project Progress Reports		79

APPENDIX A - SURVEY EXECUTION AND DESIGN		A-1

  A-1 Defining the Survey Universe			A-2
	
  A-2 Sampling the Survey Universe			A-3

  A-3 Techniques for Surveying the Samples Selected	A-7

  A-4 Survey Design Principles				A-10
      A.4.1 Organization				A-11
      A.4.2 Length					A-11
      A.4.3 Question Sequence and Wording		A-12
      A.4.4 Standardized Questions			A-14
            A.4.4.1 Behavioral Measures			A-14
            A.4.4.2 Attitudinal Measures		A-15
            A.4.4.3 Social and Demographic Measures	A-16

  A.5 Non-Response Bias					A-17

  A.6 Interviews With Transportation Agency Personnel	A-18

   A.7 References					A-19
	
vi


APPENDIX B - STATISTICAL METHODOLOGY			B-1

  B.1 Definitions					B-1

  B.2 Data Analysis Determination			B-2

  B.3 Sample Size Determination				B-4

  B.4 Data Collection					B-7

  B.5 Analysis Methods					B-7

  B.6 Methodology Documentation				B-10

  B.7 References					B-10

APPENDIX C-GLOSSARY					C-1

APPENDIX D-BIBLIOGRAPHY					D-1

APPENDIX GENERAL WORK TASKS OF EVALUATION CONTRACTOR	E-1

  E.1 Task 1: Evaluation and Task Administration Plans	E-1
			
  E.2 Task 2: Implementation and Analysis of Data	E-2

  E.3 Task 3: Report Preparation			E-3

vii
    

			LIST OF EXHIBITS

							PAGE

EXHIBIT 1: Selected Examples and Applications of APTS 	 4

EXHIBIT 2: Evaluation Relationships 			 6

EXHIBIT 3: Evaluation Process 				 8

EXHIBIT 4: APTS Operational Test Planning, 
           Implementation, and Evaluation Sequence 
           of and Responsibility for Activities 	17

EXHIBIT 5: Basic Site Data Requirements for APTS 
           Operational Tests 				20

EXHIBIT 6: Typical Sources for Site Data 		22

EXHIBIT 7: APTS Program Objectives and Examples of 
           Corresponding Measures 			31

EXHIBIT 8: Examples of Data Collection Techniques for 
           Selected Measures 				36

EXHIBIT	9: FTA/Section 15 Worksheet for Functional 
           Distribution of Expense Object Classes/
	   Level B 					47

EXHIBIT 10: Service Area for the Seattle Project 	63

EXHIBIT 11: Distribution of Park-and-Ride Users for the 
            Seattle Project 				64

EXHIBIT 12: Passenger Volume for the Seattle Project 	65

EXHIBIT 13: Bus Schedule Adherence for the Minneapolis 
            Urban Corridor Project 			66

EXHIBIT 14: Corridor Demographic Characteristics for 
            the Shirley Highway Express Bus-On-Freeway 
            Project 					67

EXHIBIT 15: Charge-A-Ride Usage by Card Type and Time 
            Period for the Merrimack Valley 
            Charge-A-Ride Program 			68

EXHIBIT 16: Comparison of Fare Payment Times Using 
            Different Methods 				69
EXHIBIT 17: Project Effectiveness Measures for the 
            Seattle  Project 				70

EXHIBIT 18: Highway Travel Time Distributions for the 
            Minneapolis Urban Corridor Project		71

viii


EXHIBIT 19: Results of Before and After Analyses for 
            Portland Self-Service Fare Collection	72

EXHIBIT 20: Benefit-Cost Analysis Results of Salt Lake 
            City Rider Information System 		73

EXHIBIT A-1: Summary of Survey Sampling Methods and 
             Applicable Survey Techniques 		A-20

EXHIBIT A-2: On-Board Bus Survey -- Katy Transitway 
             Transit  User Survey 			A-21

EXHIBIT A-3: Carpool/Vanpool Survey 			A-22

EXHIBIT A-4: Freeway Motorist Survey 			A-24

EXHIBIT A-5: Bus Riders Survey 				A-26

EXHIBIT A-6: Bus Riders Mail-Back Survey 		A-28

EXHIBIT A-7: Bellevue Smart Traveler Project Surveys 	A-30
 
EXHIBIT A-8: Washington, DC, Self-Administered Post 
             Card  Bus Survey 				A-34

EXHIBIT A-9: 1979 Downtown Crossing Bus Passenger 
             Survey 					A-35

EXHIBIT A-10: Recommendations for Questions on Boarding 
              and Alighting Points (for user surveys 
              only) 					A-36

EXHIBIT A-11: Recommendations for Questions on Trip 
              Origin 					A-37

EXHIBIT A-12: Recommendations for Questions on Trip 
              Destination 				A-38

EXHIBIT A-13: Recommendations for Questions on Trip
              Start and End Times 			A-39

EXHIBIT A-14: Recommendations for Questions on Access 
              Mode to Transit Vehicle 			A-40

EXHIBIT A-15: Recommendations for Questions on When 
              Present Mode was First Used 		A-41

EXHIBIT A-16: Recommendations for Questions on Former 
              Transportation Mode 			A-42

EXHIBIT A-17: Set of Attitudinal Questions on Travel by 
               Transit and Auto 			A-43

EXHIBIT A-18: Set of Questions on General Attitudes of 
              the Population 				A-44

ix

	
EXHIBIT A-19: Recommendations for Questions on 
              Respondents' Sex 				A-46   

EXHIBIT A-20: Recommendations for Questions on 
              Respondents' Age 				A-47

EXHIBIT A-21: Recommendations for Questions on 
              Respondents' income 			A-48

EXHIBIT A-22: Recommendations for Questions on Auto 
              Availability 				A-49

EXHIBIT A-23: Recommendations for Questions on Auto 
              Ownership 				A-50

EXHIBIT A-24: Recommendations for Questions on Whether 
              Respondent has a Driver's License 	A-50

EXHIBIT A-25: Recommendations for Questions on 
              Respondents' Occupation  			A-51

EXHIBIT A-26: Recommendations for Questions on 
              Respondents' Educational Level 		A-52

EXHIBIT A-27: Recommendations for Questions on Length of 
               Residence 				A-53
x


		1. OVERVIEW OF EVALUATION GUIDELINES

	This document presents guidelines for planning, 
implementing, and reporting the findings of the 
evaluation of Federal Transit Administration's (FTA) 
Advanced Public Transportation Systems (APTS) operational 
tests.  These evaluation guidelines are intended for use 
by organizations engaged by the Research and Special 
Programs Administration/Volpe National Transportation 
Systems Center (Volpe Center) to evaluate the APTS 
operational tests.  In addition, the guidelines will be 
useful to state and local organizations involved in the 
design and evaluation of Advanced Public Transportation 
Systems.

	An objective of these guidelines is to foster 
consistency of evaluation philosophy and techniques, and 
comparability and transferability of results to improve 
the quality and utility of information obtained from the 
APTS program.  The guidelines are designed to emphasize 
the assessment of the APTS Program's national objectives 
as well as the objectives of the local implementing 
agency.

	The various operational tests implemented under the 
APTS Program are meant to serve as learning tools and/or 
as models for other locales throughout the country.  In 
order for these tests to have maximum effectiveness in 
their respective operational capacities, a consistent, 
carefully structured approach to project evaluation is 
desirable.

	This document has been prepared to provide a common 
framework and methodology for developing and then 
executing the evaluation of individual operational tests.  
These evaluation guidelines are by no means 
comprehensive--that is, they do not offer a suggested or 
preferred course of action for every conceivable 
situation that might arise.  Nor are they to be rigidly 
or blindly followed, since each operational test and each 
site wilt be unique and wilt require somewhat tailor-made 
evaluation procedures.

	It is anticipated that these guidelines wilt be 
modified during the course of the APTS Program to reflect 
experience gained in implementing and monitoring the 
evaluations of individual tests.  Although it is not the 
desire to update these guidelines frequently, 
modifications resulting from field experience wilt be 
made where appropriate for enhancement of performance and 
evaluation of the various projects.

1


	In order to put these guidelines into a meaningful 
context, Chapter 2 provides background information on the 
FTA/APTS Program and the operational test evaluation 
process.  Chapters 3 and 4 present guidelines relative to 
planning and executing operational test evaluations.  
Finally, Chapter 5 presents the recommended content and 
organization for each type of report to be prepared in 
conjunction with the evaluation process.

2


			2. BACKGROUND

	The Federal Transit Administration has developed 
the Advanced Public Transportation Systems (APTS) Program 
which is an integral part of the overall U.S. DOT 
Intelligent Vehicle Highway Systems (IVHS) effort.  A 
major aim of the APTS Program is to promote research and 
development of innovative applications of advanced 
navigation, information, and communication technologies.  
These technologies would be designed and tested to 
achieve APTS Program goals directed toward enhancing the 
ability of public transportation systems to satisfy 
customer needs and contributing to the achievement of 
broader community goals and local objectives.  The APTS 
Program goals and objectives will be discussed further 
within the context of the evaluation frame of reference.

	The wide array of new technologies provides a 
unique opportunity to discover innovative and useful 
applications in public transportation.  These operational 
tests and evaluations will be the principal activities of 
the APTS Program.  Real world testing will be done in 
urban and rural areas using those technologies which 
appear to offer promise and represent useful 
applications.

	Major technologies include automated vehicle 
location systems, smart card systems, dynamic ridesharing 
systems, passenger information systems, high occupancy 
vehicle systems, and vehicle component monitoring 
systems. Exhibit 1 provides selected examples of these 
technologies and associated applications.  Tests will 
involve joint ventures with state and local governments, 
and, when appropriate, universities and private vendors.  
Tests may range from 3 - years: 1-2 years to develop 
implementation plans, 1 year to implement service, and 1 
year to evaluate the APTS application and associated 
impacts.

	In order for the APTS Program to encourage 
significant technological innovations by many urban and 
rural areas, the technologies tested and the results 
obtained must be evaluated, well documented, and widely 
distributed. It is important not only that the 
operational tests be structured and evaluated to 
facilitate transferability of results but also that 
evaluation results be disseminated so that prospective 
beneficiaries in other urban and rural areas are made 
aware of the potential of such technologies. Accordingly, 
a significant element of the APTS Program is the 
technology sharing function.

3


    EXHIBIT 1. SELECTED EXAMPLES AND APPLICATIONS OF APTS

APTS Examples				Applications

. Automated vehicle location (AVL)    . controlling and monitoring 
  system using satellite or ground-     the use of vehicles
  based technologies and compute-
  rized dispatching techniques	      . estimating vehicle 
                                        positions to assist 
				        dispatchers in improv-
					ing  on-street 
					schedule adherence

				      . obtaining boarding and
				        alighting information 
					in conjunction with
					automatic passenger 
					counters (APCs)

				      . assisting in the 
					development of more
					realistic schedules

			              . facilitating the 
					assignment of
					individuals to shared 
					ride, demand response 
					services 
				      . assisting in the 
					preparation of daily 
					driver logs

. Smart card systems using a contact  . facilitating the 
   or contactless plastic card with a	collection of fares, 
   microchip and storage and processing the verification of 
   capabilities	                        travel, and the 
					acquistion of 
					information about 
					passengers and vehicle 
					usage

				      . encouraging the 
					coordination of various
					modes including bus, 
					rail, auto, and 
					parking services

				      . aiding in the 
					establishment of a 
					postpayment fare 
					system and the 
					application of employer
					and human service 
					agency-based subsidy 
					programs

				      . assisting in the 
					design of a 
					comprehensive, 
					historical vehicle 
					maintenance and parts 
					inventory  database

4


			EXHIBIT 1 (continued)

APTS Examples

. Dynamic ridesharing systems using   . providing quick 
  real-time communication methods	and easy access to 
  with the aid of touch-tone telephone, up-to-date information
  television, radio, and videotex       to aid an individual 
  systems		                in arranging a carpool
					or vanpool the same 
					day or evening before 
					a trip

. Passenger information system	      . supplying passengers 
  using audio, visual, and/or hard      with real time infor-    
  copy methods such as digitized        mation on routes,       
  voice, interactive television,        schedules, cancella-       
  videotex, automated map displays,     tions, delays, rerout-    
  computer monitors and printers, and   ing, and other aspects
  other devices located in terminals,   of service to make
  stations, vehicles, places of         travel easier and to
  employment, and at home; also         facilitate intermodal
  could be provided in conjunction      transfers
  with a traffic management center
  (TMC).

. High occupancy vehicle systems      . providing traffic 
  (HOVs) including preferential	        control signal
  treatment methods and park and ride   preemption capabilities
  facilities

				      . monitoring vehicle
					occupancy remotely to 
					enforce HOV lane 
					restrictions

. Vehicle component monitoring	      . assisting in the early
  systems                               detection of problems 
					with vehicle components
					(e.g. engine, exhaust 
					system) to avoid 
					component failure while
			  		vehicle is in 


The exact number, general content, and location of 
the APTS operational tests are yet to be determined.  For 
each fiscal year program, a series of primary objectives 
will be selected, and a group of proposals corresponding 
to each objective, and in keeping with total budgetary 
constraints, will be developed.  Then, following an 
investigation, analysis, and negotiation

5


process involving FTA, the Volpe Center, and candidate 
sites, a final set of operational tests and respective 
sites will be agreed upon.  Once final negotiation and 
transfer of funds between FTA and the APTS local sponsor 
are completed, the operational test can be implemented 
and evaluated.

	As part of its responsibility to evaluate the 
operational tests implemented under the APTS Program, the 
Volpe Center shall engage contractor support to 
participate in all phases of the evaluation process.

	Exhibit 2 shows the interaction among FTA, the 
Volpe Center, the local sponsor, the evaluation 
contractor, and the APTS vendors involved in the 
operational test.

	FTA/APTS staff is responsible for overseeing and 
guiding all aspects of the operational test including 
planning, site selection, negotiations with the site, 
implementation, and evaluation.  The local sponsor is 
responsible for planning and implementing the actual 
conduct of the operational test as well as performing 
most of the data collection.  The Volpe Center assists 
FTA in the activities for which FTA is responsible, and 
directs and monitors the efforts of the evaluation 
contractor.  The Volpe Center, the evaluation contractor, 
and the vendors interface with the local sponsor (or the 
implementing agency, if different from the local 
sponsor). While being directly responsible to the Volpe 
Center for its activities, the evaluation contractor will

6


maintain an informal association and relationship with 
the local sponsor, the APTS vendors, and the cognizant 
FTA Project Manager.  The APTS vendors, as deemed 
appropriate by FTA and the Volpe Center, may participate 
in a review of the evaluation plan, data reduction and 
analysis, and the interim and final reports.  The APTS 
vendors may serve on the local evaluation
review team as discussed in Section 2.2.

2.1 OVERVIEW OF THE EVALUATION PROCESS

	The evaluation process can be thought of 
conceptua11y as a link between the operational tests and 
technology transfer portions of the APTS Program. That 
is, it serves as a bridge between the conduct of an 
operational test at a particular site and the 
understanding of its actual performance at that site as 
well as its potential effectiveness in other locales.  
The quality of the evaluation process directly influences 
the accuracy and perceptiveness of the operational test 
assessment and ultimately affects the applicability and 
transferability of tat findings.

	Exhibit 3 is a flow diagram representing the 
evaluation process for an APTS operational test.  The 
diagram is divided into four major sections: the 
evaluation frame of reference, evaluation planning, 
evaluation implementation, and potential evaluation 
spin-offs.  (The specific organizational responsibilities 
associated untie the various aspects of each APTS test 
are given later in this chapter.)  The first and fourth 
sections can be thought of, respectively, as input to and 
output from the active phases of the evaluation process, 
which are planning and implementation.  A discussion of 
each of the four sections follows.

2.1.1  Evaluation Frame of Reference

	The evaluation frame of reference consists of four 
elements: the operational test application(s); APTS 
Program objectives; external influences; and local 
issues, objectives and site characteristics.

	An APTS operational test will consist of one or 
more technological applications introduced individually 
or sequentially.  For example, a test might include the 
use of a smart card to facilitate automatic fare 
collection.  Another example could consist of an 
automated vehicle location (AVL) system to determine 
vehicle position, followed by the installation of an 
automated passenger counting (APC) system and a 
computerized dispatching and scheduling

7


		EXHIBIT 3: EVALUATION PROCESS*
8


system which work in conjunction with the AVL system.

     Each APTS operational test also is intended to meet the 
goals of the APTS Program which are: 1) to enhance the 
ability of public transportation to satisfy customer 
needs; and 2) to contribute to broader community goals by 
providing information on innovative applications of 
available IVHS technologies.  These goals can be 
translated into the following set of objectives:

Objective #1: Enhance the Quality of On-Street Service to 
              Customers
  . Improve the quality, timeliness, and availability of 
    customer information,
  . Increase the convenience of fare payments within and 
    between modes,
  . Improve safety and security,
  . Reduce passenger travel times, and
  . Enhance opportunities for customer feedback.
 
Objective #2: Improve System Productivity and Job 
              Satisfaction
  . Reduce transit system costs,
  . Improve schedule adherence and incident response,
  . Increase the timeliness and accuracy of operating data 
    for service planning and scheduling,
  . Enhance the response to vehicle and facility failures,
  . Provide integrated information management systems and 
    better management practices, and
  . Reduce worker stress and increase job satisfaction.

Objective #3: Enhance the Contribution of Public 
             Transportation Systems to Overall Community Goals
  . Facilitate the ability to provide discounted fares to 
    special user groups (e.g., disabled persons or 
    employees eligible for tax-free employer subsidies),
  . Improve communication with users having disabilities 
    (e.g., visual or hearing impairments),
  . Enhance the mobility of users with ambulatory 
    disabilities,
  . Increase the extent, scope, and effectiveness of 
    Transportation Demand Management programs,
9


  . Increase the utilization of high occupancy vehicles, 
    with an emphasis on reducing the use of single 
    occupant vehicle, and

  . Assist in achieving regional air quality goals and 
    mandates established in the Clean Air Act Amendments 
    of the Intermodal Surface Transportation Efficiency 
    Act (ISTEA).

Objective #4: Expand the Knowledge Base of Professionals 
              Concerned with APTS Innovations

  . Conduct thorough evaluations of operational tests,
  . Develop an effective information dissemination process,
  . Showcase successful APTS innovations in model 
     operational tests, and
  . Assist system design and integration.

      Objective #1 relates primarily to the riders and their 
desire for improved transit service.  Objective #2, on 
the other hand, deals in part with management aspects 
regarding system costs, service planning, scheduling, and 
operations.  Objective #3 concerns broader impacts in 
terms of the degree to which an APTS application 
contributes to local community goals and national issue 
pertaining to, for example, the special needs of disabled 
persons, congestion management activities, user-side 
subsidy initiatives, energy, air quality, and 
accessibility.  In section 3.2.1, measures are presented 
to examine the level to which these first three 
objectives are attained in each operational test.

         The fourth objective is directed at expanding the 
knowledge base of policy-makers, engineers, planners, 
researchers, and other individuals interested in the 
application of advanced technology to improve public 
transit.  Because this objective is a broader, 
overarching aim of the entire evaluation program, its 
level of achievement will not be assessed using measures 
such as those discussed in section 3.2.1.  Instead, an 
effort will be made to cull information from interim and 
final evaluation reports prepared as part of each 
operational test, and this information will be 
disseminated in publications such as FTA's APTS Briefs, 
IVHS America's Newsletter, and technical journals and 
conference proceedings of other organizations.  In 
addition, selected evaluation results will be summarized 
on electronic bulletin boards commonly available to 
transportation professionals, and results will be 
presented at national and

10


international meetings.  Finally, where appropriate, the 
findings and conclusions of the evaluations will be used 
as a basis for discussion in focus groups, meetings, and 
seminars.

	It should also be emphasized that for any given 
operational test, there may be objectives, over and above 
the APTS Program objectives, which are important 
evaluation considerations.  These might be state or local 
objectives which other participants (e.g. transit 
operator, state transportation agency, community group, 
or local government) are striving to attain (e.g., to 
encourage ridesharing into the downtown area for the 
purposes of reducing parking requirements or traffic 
congestion in the central business district, to preserve 
the stability, cohesion, and authenticity of 
neighborhoods).  The extent to which these state and 
local objectives relate to the APTS program objectives 
should be identified by the contractor.

	The operational test site can consist of anything 
from a corridor in a city to a group of cities or towns, 
and can be at any point along the population and density 
spectrum.  An understanding of the unique demographic, 
economic, geographic, and transportation characteristics 
of the site, as well as prevailing attitudes toward 
transportation, is a useful and necessary adjunct to 
knowledge about the APTS application and associated 
objectives.

	To the maximum extent possible, external influences 
on the project should also be identified and, if 
necessary, appropriate strategies should be designed to 
reduce the likelihood that such influences will have 
adverse effects on the operational test.  For example, if 
the APTS application has radio frequency (RF) spectrum 
requirements, such requirements should be analyzed, and 
political negotiation with authorized communication 
agencies should be initiated as early as possible.

	Information on the planned APTS innovations, 
project objectives, other issues and site 
characteristics, and external influences will generally 
be available from the application submitted to FTA by the 
site prior to approval of the project.  Depending on the 
timing of the evaluation contractor's initial involvement 
in the project, a more detailed description of the 
project may be available in the form of a Project 
Implementation Plan.  Further background on the 
operational test (e.g., genesis of the project concept, 
recent history of transit/para-transit developments at 
the site) can be obtained through discussions with the 
PTA Project Manager, the Volpe Center staff, and the 
local sponsor.

11


			2.1.2 Evaluation Planning

The evaluation planning phase of the evaluation 
process is the period during which the contractor 
interacts with FTA, the Volpe Center, and various 
agencies at the local level to transform the evaluation 
frame of reference into a detailed, structured program 
for conducting the evaluation. This phase sets the stage 
for the entire evaluation effort and, in addition, 
provides an opportunity to reassess and, if necessary, 
restructure the planned operational test.

The planning phase begins with the preparation of 
an Evaluation Strategy for the particular project, which 
describes:

(1) Pertinent information on the APTS application and 
    site (in particular, an indication of what features 
    of the operational test are unique and merit emphasis 
    in the evaluation).

(2) APTS Program objectives addressed by the operational 
    test.

(3) Relevant local, state and/or national objectives and 
    issues addressed (and the relative emphasis to be 
    placed on these objectives vs. APTS objectives).

(4) Key issues to be resolved.

(5) External influences to be addressed.

(6) Recommended scope and focus of the evaluation 
    including a discussion of the APTS costs and 
    functional characteristics and a review of the 
    potential efficiency, effectiveness and other impacts 
    anticipated.

The Evaluation Strategy may be prepared by the Volpe 
Center or the contractor.  The contents of each 
Evaluation Strategy will vary from test to test depending 
on the nature and timing of the project.

	The Evaluation Strategy becomes the basis for the 
more detailed Evaluation Plan1 which is developed by the 
contractor.  While the Volpe Center will provide a 
general evaluation strategy including suggestions 
regarding measures to be used, data to be collected, and 
analytical techniques to be employed, it is generally the 
contractor's responsibility to refine and elaborate on 
the Volpe Center's suggested strategy by developing 
specific procedures for collecting and analyzing data 
relative to project objectives, issues, and the site.

____________________
[1] Chapter 3 presents guidelines relative to the 
evaluation planning phase.  The recommended content and 
 organization of the Evaluation Plan presented in Chapter 5.

12


	In developing the Evaluation Plan, the contractor 
is encouraged to propose changes to the approach 
recommended by the Volpe Center, particularly if the 
proposed modifications have significant potential to 
improve the objectivity, accuracy, completeness, and/or 
efficiency of the project evaluation effort or to enhance 
the transferability of project findings.  In addition, 
total evaluation costs relative to potential findings 
must be borne in mind at all times.  Throughout the 
process of developing the Evaluation Plan, the contractor 
is urged to keep in close contact with the local sponsor 
or project team responsible for implementing and 
operating the test and performing data collection.  This 
continuing liaison with the local sponsor will ensure 
that the proposed methods of data collection are 
consistent with the resources available at the local 
level, with the operational implementation plan developed 
by the site, with important local objectives, and with 
reasonable costs for the evaluation contractor efforts.

	As is apparent from the preceding discussion, the 
evaluation planning phase entails substantial and 
continued interaction among all parties involved in the 
operational test.  Ideally, planning of the evaluation 
effort should be coordinated, and take place concurrently 
with the planning of the project itself.  This 
coordination between the implementation/operation and 
evaluation planning cycles permits optimum flexibility in 
the conduct of the overall test.  Where possible, 
operational aspects of the test will be planned to 
conform to requirements of the evaluation, rather than 
the evaluation having to be integrated into a 
pre-existing, rigid operational structure.  The 
concurrence of the two planning cycles ensures that the 
Evaluation Plan is completed prior to the implementation 
of the project.  Early development of the Plan, in turn, 
allows the necessary lead time for "before" data 
collection -- that is, observations of phenomena such as 
transit system performance prior to the introduction of 
the APTS application(s) as well as possible information 
on community awareness and attitudes prior to project 
implementation. Throughout this phase of the project, it 
is critical to recognize that the FTA Project Manager is 
the final authority in negotiating any operational test 
modifications with the local sponsor.

2.1.3 Evaluation Implementation

	The evaluation implementation phase is the period 
during which the approved Evaluation Plan is executed.  
Activities during this phase include collection/analysis 
of data relative to

13


project objectives and issues, collection/analysis of 
data on site characteristics, compilation of a chronology 
describing the implementation and operation of the test, 
and recording of external factors which might influence 
operational test findings and results.  Contractor 
functions during this phase include monitoring and in 
selected instances, supervising the data collection 
process (generally to be performed by the local sponsor), 
any data collection not performed by the local sponsor, 
data reduction and analysis, subjective analysis of 
information relative to project issues, and synthesis of 
project findings into one or more Interim Evaluation 
Reports and a Final Summary Evaluation Report.2

	This phase not only generates information on which 
the final assessment of the operational test is based but 
also provides feedback information relative to ongoing 
transit operations.  The ongoing evaluation activities, 
while adding to the cumulative body of quantitative and 
qualitative information regarding the project impacts, 
provide interim indications of costs and functions of 
APTS applications and the preliminary effects of these 
applications on transit system efficiency and 
effectiveness.  These interim findings serve as useful 
input to the local agency responsible for implementing 
and operating the test by suggesting the need for 
operational modifications.

	During this phase, modifications may be made to the 
evaluation procedures originally specified in the 
Evaluation Plan.  For instance, examination of interim 
findings may reveal certain gaps or redundancies in the 
originally planned data collection program.  Still other 
reasons for modifying the evaluation procedure might be 
changes in the operational test, unanticipated 
developments or institutional factors at the site, or 
discovery of an improved evaluation procedure.  
Procedural steps to accomplish this necessary update for 
the Evaluation Plan appear in Chapter 5.

	The culmination of the evaluation implementation 
phase is the Final Summary Evaluation Report, which 
presents the following types of findings:
_________________
[2] Chapter 4 presents guidelines relative to the evaluation 
implementation phase.   Chapter 5 gives the 
recommended content and organization of the various 
contractor reports prepared during this phase, 
including the Monthly Evaluation Progress Report, 
the Annual Project Status Summary, the Interim 
Evaluation Report, and Final Summary Evaluation 
Report.  In addition, Chapter 5 describes the 
content of local sponsor’s quarterly Project 
Progress Report to FTA, which can serve as useful 
input to the contractor’s work.

14


(1) Evaluation of the project in terms of its attainment 
    of relevant APTS Program objectives and other (local 
    and/or national) project objectives.
   
(2) Insight into project issues associated with 
    operational feasibility and  characteristics of the 
    applications.

(3) Assessment of the influence of site-specific 
    characteristics and external factors on the outcome 
    of the operational test.

(4) Lessons learned, based on practical experience, 
    relative to the implementation and operation of the 
    APTS applications (possibly to include 
    recommendations for project modifications in the test 
    site or for future applications in other locales).

(5) Appraisal of the evaluation procedures employed in 
    terms of effectiveness, cost, accuracy, etc.

In essence, this report presents an assessment of the 
impact of the APTS applications at the site and provides 
guidance for the transferability of results to other 
locales.

	The body of the Final Summary Evaluation Report 
includes both narrative and graphic exposition, while 
detailed quantitative data and documentation of 
procedures are provided in technical appendices.  Since 
the report is intended for a variety of audiences -- 
including transportation planners; transit operators; 
federal, state, and local officials; and private industry 
- - it contains an Executive Summary which highlights the 
salient project findings.

2.1.4 Potential Evaluation Spin-Offs

	It is anticipated that each operational test will 
give rise to potential implementation and analytical 
spin-offs.  The Final Summary Evaluation Report, while 
essentially documenting the history and effects of a 
single project, also serves the broader function of 
increasing the understanding of and stimulating the 
application of the demonstrated APTS technologies in 
other localities.  Information presented in the report 
provides a versatile basis for comparing the effects of a 
particular APTS application with those of other similar 
projects, suggesting modifications to the applications 
for future use, and predicting the effectiveness and 
utility of the APTS applications in other cities.  
Moreover, the report's assessment of project evaluation 
procedures can serve as a stimulus for improving the 
state-of-art of evaluation techniques.  Since these 
broader functions of the Final Summary Evaluation Report 
generally materialize after

15


the test period and are not within the purview of the 
evaluation contractor assigned to a particular project, 
they are shown in Exhibit 3 as potential evaluation 
spin-offs.

2.2 COORDINATION OF APTS EVALUATIONS

      Exhibit 4 summarizes the various activities involved in 
planning, implementing, and evaluating an APTS 
operational test and indicates the allocation of 
responsibility for these activities.  The sequence of 
activities ranges from overall APTS Program definition, 
to the operation and evaluation of an individual test, to 
the spin-off uses of the project.  It can be seen that 
the entire stream of activities, especially those 
comprising the evaluation process, involves extensive 
interaction among FTA, the local sponsor, the Volpe 
Center, the evaluation contractor, and the APTS vendors.  
Moreover, it should be noted that the activities shown do 
not always occur in a fixed sequence.  Time constraints 
may require that some of the steps be performed in 
parallel, and there will ideally be considerable 
interaction and feedback between the project planning and 
evaluation planning phases.  The review functions of the 
Volpe Center, the local sponsor, and the APTS vendors 
associated with the data analysis provide a mechanism to 
identify, on a continuing basis, major problems (if any) 
so that APTS operational changes can be made (if 
necessary) during the course of the test. Evaluation 
spin-offs, while arising out of individual tests, will 
result in activities which extend beyond the FTA, Volpe 
Center, local sponsor, and evaluation contractor.

	The diversity of activities and generally long 
(three to four years) time frame for an individual test 
necessitate close and continual coordination among the 
groups involved.  To facilitate communication among local 
test participants and the contractor concerning the 
evaluations, FTA will encourage the establishment of a 
local evaluation review team consisting of 
representatives of transit providers, metropolitan 
planning agencies, human service organizations, 
environmental groups, APTS vendors, and the general 
public.  It may also be appropriate to include faculty 
from local colleges and universities on the evaluation 
review team.  The contractor will meet with- the local 
evaluation review team to discuss the project objectives 
and the emphasis to be placed on each objective in the 
evaluation; to determine the roles and responsibilities 
of all parties involved in the anticipated data 
collection activities; to review problems encountered (if 
any) during the conduct of major data collection 
activities and overall

16


EXHIBIT 4: APTS OPERATIONAL TEST PLANNING, IMPLEMENTATION, AND
EVALUATION: SEQUENCE OF AND RESPONSIBILITY FOR ACTIVITIES

Click HERE for graphic.



KEY:

P = Primary role   M = Monitoring role   R = Review function

a  	Includes local evaluation review team.
b	Local evaluation review team will be established as 
        part of negotiations.
c	Primary role may also be assigned to the 
	contractor.  It may be necessary to have the 
	contractor on-site to monitor the conduct of some 
	data collection efforts such as an on-board survey 
	to ensure that such efforts are carried out properly 
	and that appropriate personnel are available to 
	address unanticipated problems and questions.
d	FTA will disseminate information from these reports, 
	where appropriate.  Such information will appear in 
	FTA’s  APTS Briefs, IVHS America’s Newsletter, 
	professional conference papers, and electronic 
	bulletin boards.  The final evaluation reports 
	themselves will also be published.

17


operational test implementation; to present preliminary 
findings and results of the data analyses; and to seek 
the team's input.  However, equally as important as 
coordination within a particular project is coordination 
across test sites, so as to maximize the effectiveness of 
the APTS Program in encouraging the application of new 
innovations.  This coordination across sites is 
essentially important with respect to the evaluation 
process.  Given the multiplicity of sites, operational 
tests, and participating organizations within the APTS 
Program, there is a strong need for coordination of the 
evaluation process so as to achieve consistency in the 
planning, implementation, and output of individual 
project evaluations.

	With respect to the conduct of the evaluations, 
such coordination will ensure that: (1) the scope of each 
evaluation effort is consistent with the importance of 
that particular APTS test relative to other APTS tests; 
(2) the technical approaches used to evaluate tests are 
consistent with the current state-of-the-art of 
evaluation techniques; (3) common data and definitions 
are employed; and (4) statistical reliability is 
maintained.

	With respect to evaluation output, such 
coordination will ensure that the Final Summary 
Evaluation Reports associated with individual projects 
are consistent in terms of content, format, perspective, 
and level of detail. This consistency in output will, in 
addition, enhance the spin-off potential of the 
evaluations.  The achievement of a basic data set of 
uniform quality across operational tests will make 
possible inter-project comparisons in terms of rider 
characteristics, site characteristics, user acceptance, 
and system efficiency and effectiveness and associated 
criteria.  These types of comparisons will be especially 
significant in the case of multiple applications of a 
particular APTS technology in several locations, or in 
the case of operational tests involving alternative APTS 
technologies directed towards a particular APTS Program 
objective.

	The coordination of the individual evaluation 
efforts will be achieved through the Volpe Center's 
active and continual participation in the program, with 
functions ranging from initial planning of each project 
evaluation effort, to monitoring of the contractor team, 
and finally to the synthesis of individual operational 
tests, evaluation reports and results. This document 
constitutes the first stage of the Volpe Center's 
evaluation coordination function, in that it describes 
general procedures to be followed by each contractor in 
performing the various evaluation tasks speeded in the 
contract.

18


	3. GUIDELINES FOR PLANNING EVALUATION ACTIVITIES

	This chapter presents guidelines for planning the 
evaluation activities associated with a particular APTS 
operational test.  As was mentioned in Chapter 2, the 
evaluation planning phase of the evaluation process is 
that period during which the contractor prepares a 
detailed Evaluation Plan based on the Volpe Center's 
Evaluation Strategy.  The Evaluation Plan contains, among 
other things, a listing of relevant quantitative and 
qualitative measures related to various APTS, local, and 
national objectives and relevant issues, associated data 
collection and analysis procedures, and site specific 
data requirements and sources (both one-time and 
recurring).  As such, the Evaluation Plan constitutes a 
structured, time-phased program for subsequently 
conducting the evaluation.

	The chapter is organized into three sections, 
corresponding to the basic decision-making elements 
shown in Exhibit 3:

  . determination of site data requirements and sources,
  . determination of measures and collection/derivation 
    techniques required to address APTS Program objectives 
    and other relevant objectives/issues, and
  . planning considerations relative to data collection 
    and analysis.

The organization of the chapter is not meant to imply a 
highly ordered time-sequencing of activities, since the 
evaluation planning phase is in fact highly iterative and 
dynamic.  Moreover, it is important to realize that the 
guidelines comprise a basic set of ground rules for 
planning evaluations.  The evaluation contractor will, in 
all probability, need to depart from these guidelines 
during the actual planning phase, so as to conform to the 
unique conditions surrounding a given operational test.

	The contractor should recognize his responsibility 
in working with the local sponsor and the Volpe Center to 
assure that an objective assessment of the project is 
achieved.  One or more site visits during the evaluation 
planning phase is desirable to establish working 
relationships and channels of communication among the 
involved organizations and to uncover any constraints 
which may have a significant bearing on the development 
of the Evaluation Plan.  During this planning effort, 
clarification must be made regarding responsibilities for 
performing and/or

19


EXHIBIT 5: BASIC SITE DATA REQUIREMENTS FOR APTS OPERATIONAL 
           TESTS
 1. Population

 2. Square miles

 3. Population density, persons per square mile

 4. Number of persons in the labor force

 5. Number of households, by type

 6. Age, sex, education, occupation, income distributions

 7. Household auto ownership

 8. Number of persons with no drivers license

 9. Modal split, by trip purpose or time of day if available

10. Existing (Pre-operational test) transit service 
    characteristics

  . Organizational arrangements
  . Route miles (fixed route systems)
  . Tour area (non-faced route systems)
  . In-service vehicles per square mile of service 
    area (non-faced route system)
  . In-service vehicles per hour within service area
  . Time of service operation throughout day
  . Days of service operation throughout year
  . Service frequency (fixed route systems)
  . Fare schedule

11. Description of para-ransit service characteristics

  . Data on taxi operations
  . Information on carpool promotion/matching programs

12. Map of the site showing:

  . The APTS project service area - note that this 
    might be a contiguous area served throughout by 
    the APTS transit system, or it might be two or 
    more non-contiguous areas linked by the APTS 
    service through a travel corridor
 
  . The existing transportation network - major 
    highways, transit lines, commuter rail lines
  . Air quality attainment and non-attainment areas
  . Major topographical features such as rivers
  . The central business district
  . Any other important activity centers

13. Description of relevant site features such as: 
  . Weather conditions
  . Seasonal population variations
  . Institutional/political climate
  . Economic conditions
  . Cost indices (e.g., cost of living index, prevailing 
    transit wage rates)
  . Population/employment growth rate, land use 
    development patterns
  . Residential mobility
  . Air quality conditions concerning ozone, lead, 
    carbon monoxide, PM10, and other environmental concerns

20


overseeing various activities.  The Evaluation Plan 
should indicate the finally agreed upon allocation of 
responsibility between the contractor and local 
evaluation review teams.


3.1 DETERMINATION OF SITE DATA REQUIREMENTS AND SOURCES

	The purpose of the site data is to provide an in-depth   
understanding of those characteristics of the site which 
might in some way influence the outcome of the project or 
the interpretation of project results.  Obviously, the 
APTS operational test will not be implemented in a static 
environment, but rather it will affect the surrounding 
area.  Thus, an examination of certain site 
characteristics is necessary in order to assess fully and 
accurately the impacts of the APTS application.

	An additional function of site data is to enhance 
the comparability and transferability of APTS project 
findings.  Specifically, if conclusions drawn from one 
project are to be compared with findings of other similar 
projects or "transferred" to other potential sites, there 
must exist an objective approach for such a comparison or 
transfer.  This requires the identification of a set of 
site-specific measures which permit one to classify sites 
in terms of meaningful similarities or to identify 
significant areas in which sites differ.  Such measures 
might employ data pertaining to demographic and land use 
attributes, transportation facilities, and vehicle travel 
characteristics, both intra and inter-urban. In addition, 
information on the political/institutional climate of the 
area and prevailing attitudes toward 
transportation-related issues might be helpful in 
anticipating or understanding any problems regarding 
implementation and evaluation of the project.

	A review of past transit project evaluations 
indicates an inconsistency in both the amounts of and 
details concerning reported site-specific data.  To some 
extent this inconsistency reflects a lack of standardized 
site data requirements, but more significantly it 
reflects deficiencies in knowledge regarding the 
interplay between site characteristics and test results.  
In an attempt to shed further light on the subject, a 
basic set of data requirements has been developed for use 
in APTS operational test projects (see Exhibit 5).

	Contractors are encouraged to propose additions or 
deletions to this list, in the context of particular 
projects, if it is felt that the nature and scope of the 
project call for a wider or narrower set of site 
descriptors.  Contractors are also encouraged to propose 
permanent

21


EXHIBIT 6: TYPICAL SOURCES FOR SITE DATA

DATA NEEDED				TYPICAL SOURCES

Demographic			U.S. Bureau of the Census
				City of County Clerk
				State Department of Labor
				State Department of Internal 
				  Revenue
				City or County Planning 
Board

Air Quality			Environmental Protection 
Agency

Land Use Characteristics	City Directories
				Local, Regional and State 
				   Planning Agencies
				Tax Assessor's Records
				Planning Studies

Motor Vehicle Travel		State Highway department 
				   (or State DOT)
				U.S. Census (Journey-to-work)
				Local Traffic Department
				Earlier Travel Surveys
				State Registration Records
				Gasoline Tax Collection Records

Public Transportation Travel	Private Transit-Paratransit 
				   Companies
				Transit Authorities
				State Highway Department  
				   (or State DOT)		
				Local Planning Agency
				U.S. Census (Journey-to-work)
				Earlier Travel Surveys

Travel by Intercity Modes	Federal Agencies such as:
(air, rail, bus)		Federal Aviation Administration
				  Interstate Railroad 
				     Administration
				  Federal Railroad 
				    Administration
				  Department of Commerce
				State Regulatory Agencies
				Earlier Travel Surveys
				Private Carriers

22


additions, deletions, or changes to this minimum list 
based on their cumulative experience in conducting APTS 
evaluations.

	Aside from the site data requirements in Exhibit 5, 
it may be desirable in certain instances to collect a 
standardized set of attitudinal measures to obtain a 
profile of the community.  Examples would be general 
opinions regarding the role of government, environmental 
issues, adequacy of transportation facilities, and 
desirability of travel by alternative modes. Since the 
value of this type of data for evaluation and 
transferability purposes has not yet been fully explored, 
community profile data will be collected only in selected 
operational tests (to be identified by the Volpe Center).  
Appendix A contains sample questionnaires which might be 
used to obtain such data.  As experience is gained in 
this area, a standardized approach to developing an 
attitudinal profile of the test site may be formally 
incorporated into these guidelines.

	It is anticipated that the data set and descriptive 
information shown in Exhibit 5 will be available from 
secondary sources or from the local sponsor and will not 
involve specialized data collection activities (an 
exception being attitudinal profile data, which will 
entail surveys). Exhibit 6 indicates typical sources for 
various categories of site-specific data.3

	Once the contractor has determined the type of site 
data required and the appropriate sources, two decisions 
remain: (1) the geographic scope of the area, and (2) the 
time period (s).

	Regarding the geographic scope, it was indicated 
above that a basic data set should be assembled for the 
APTS service area.4   In some cases, data conforming 
exactly to the service area boundary may be unavailable 
or may be obtained only by aggregation of fine-grained 
data (e.g., Census tract).  If data is available for an 
area approximating the service area, the contractor may 
choose to use this pre-existing data base rather than 
deriving a special data base, provided that such a 
substitution will not be misleading and bias the 
evaluation. O n the other
_______________________________
[3] Adapted from Heaton, Carla; McCall, Chester; and 
    Waksman, Robert;  "Evaluation Guidelines for Service and 
    Methods Demonstration Projects",  USDOT/UMTA-SMD; 
    Washington, DC, 1976.

[4] A definition of the APTS service area may not be 
    available at the outset of the project, but rather will 
    need to be developed during the evaluation implementation 
    phase on the basis of user surveys.

23


hand, the use of fine-grained data may be appropriate if 
the service area is large and heterogeneous and thus 
should be divided into zones.

	The time period(s) for which data is to be assembled 
depends on the time period of the operational test and 
the rate at which conditions at the site are changing.  
If the project spans a fairly long period it may be 
desirable to gather site data for periods before, during, 
and after the project.  In the case of a rapidly changing 
area or a staged project, data for even more points in 
time may be necessary.  Moreover, if an historical 
perspective on the site is deemed relevant to the 
evaluation, it may be desirable to obtain 1980 as well as 
1990 Census figures or recent trend data for key 
variables such as population, employment, and modal 
split.  Since original data collection by the contractor 
is not anticipated, the number and exact timing of site 
data periods will be constrained by the collection cycles 
of existing sources.

3.2 DETERMINATION OF MEASURES AND COLLECTION/DERIVATION 
    TECHNIQUES REQUIRED TO ADDRESS APTS PROGRAM OBJECTIVES AND 
    OTHER RELEVANT PROJECT  OBJECTIVES/ISSUES 

	It was pointed out in Chapter 2 that the Evaluation 
Strategy will set forth a recommended set of APTS Program 
objectives, relevant project objectives (of local and 
national significance), and project issues to be 
examined.  The contractor, in developing the Evaluation 
Plan, is  responsible for reviewing this recommended set 
in the context of the local sponsor's Project 
implementation Plan and the various national and local 
perspectives, and then proposing appropriate 
modifications to the list of objectives and issues.

	Once the set of project objectives and issues has 
been finalized (which involves obtaining concurrence from 
the Volpe Center), the contractor must associate with 
these items a set of germane measures and identify 
suitable techniques to derive each measure and to collect 
necessary data.  It is important to note that certain 
issues may not lend themselves to the use of quantitative 
measures but may rather involve qualitative analysis of 
pertinent information.

	The material presented below is intended to guide the 
contractor in developing appropriate measures and 
associated collection/derivation techniques.  It is 
important to recognize that this material will 
undoubtedly be modified as information is gained through 
the consistent application and analysis of evaluation 
techniques on the operational tests. Therefore, because 
revisions to data program requirements in terms of basic 
data sets, collection and

24


analysis procedures, and presentation techniques can be 
expected, the fundamental value of this section of the 
guidelines lies in the manner in which it structures the 
approach to the selection of measures and the selection 
of techniques for collecting/deriving them.

	In preparing this material, considerable documentation 
was reviewed (see Bibliography).  In addition, direct 
observance and participation in many previous and ongoing 
Federally-funded projects has permitted those preparing 
this document to identify not only a logical structure 
for project evaluation but also to highlight problem 
areas of which all potential project evaluators should be 
aware.  The specific projects which contributed the 
greatest amount of insight were the evaluation plan 
development for the APTS/AVL operational tests and the 
Service and Methods demonstration projects.

3.2.1 Basic Set of Measures
	To assist the evaluation contractor and the local 
evaluation team in the selection of measures to assess 
operational test objectives, six categories of measures 
are suggested:
 
  . APTS costs,
  . APTS functional characteristics,
  . user acceptance,
  . transit system efficiency,
  . transit system effectiveness, and
  . impacts.

The first three categories of measures relate directly to 
the costs, functional aspects, and utility of the APTS 
application and associated equipment.  The next two 
categories pertain to transit system performance in terms 
of actual delivery and usage of the transit services 
provided.  The final category of measures addresses 
project impacts related to critical transportation issues 
and societal goals and concerns.

	While many operational tests will be designed to 
achieve the same (or similar) objectives, some tests 
might be particularly unique in their ability to address 
certain objectives.  Consequently, "priority objectives" 
should be identified in these unique tests, and a 
corresponding set of measures should be formulated so 
that these "priority objectives" are given

25


proper attention, emphasis and evaluation resources.  
Furthermore, the type of measure and the method of 
measurement should be considered as discussed below.

  . Type of measure

    Quantitative -- a measure which is expressed in 
    terms of counts, measurements, dollars, or other physical 
    units

    Qualitative -- a measure which is expressed in terms 
    of people's attitudes, perceptions, or observations

  . Method of obtaining measure

    Collected -- obtained by measurement (vehicle travel time),
    counting number of passengers), surveying (perceived 
    reliability), or from records (daily revenue)

    Derived -- calculated from collected measures either by 
    simple arithmetic procedures (passenger miles per seat 
    mile) or through use of analytic models (reduction in air 
    pollution or fuel consumption)

In reviewing the basic set of measures, it is important 
to note that some of these measures would be more 
meaningful if stratified by time of day (beak versus 
off-peak), location (corridor versus arterial), person 
time segments (waiting, access, transfer, in-vehicle), 
route type (fixed route versus demand responsive), and 
vehicle tour segments (in-service, non-service). Because 
such a classification of measures would have needlessly 
extended the list, the subject of stratification, or 
categorization, with respect to specific data collection 
plans is discussed separately in Section 3.3.2 of this 
chapter.

	The above categories of measures are not to be 
construed as a minimum requirement for every APTS 
project, since an evaluation need only encompass measures 
corresponding to the APTS Program objectives and other 
project objectives/issues addressed by the particular

26


operational test.  Rather, the categories of measures 
should be used by the contractor as a checklist from 
which the most germane measures can be selected and to 
which other relevant measures can be added as 
appropriate.

	It will be noted that for each of the APTS Program 
objectives, it is possible to measure attainment of some 
objectives from two vantage points: the actual and the 
perceived attributes of the transit system (as 
represented by quantitative and qualitative measures, 
respectively).  In the case of transit travel time, it 
might be appropriate to measure actual changes in travel 
time and then to compare with perceived travel time. 
Similarly, in the case of APTS equipment reliability and 
user acceptance measures, comparisons with user 
perceptions and attitudes might also be appropriate.

	Until more is learned about the interrelationship 
between actual measurements and attitudinal data, it is 
not possible to set forth hard and fast rules for when to 
supplement quantitative measures with qualitative 
measures.  Clearly, it may be prohibitively expensive to 
employ this two-pronged procedure for each area of 
interest; on the other hand, mere reliance on 
quantitative measures may result in overlooking what is 
in fact the major behavioral determinant -- people's 
perceptions of the system.  For the time being, the 
contractor must exercise sound judgment in deciding which 
situations are unique and instructive enough to warrant a 
two-pronged data collection effort.  In no case should an 
attitudinal measure ever be used in place of a 
quantitative measure, where both are available.

	The rationale underlying each category of measures 
and their association with operational test objectives is 
discussed in Sections 3.2.1.1 to 3.2.1.6.  Further 
discussion of data collection/derivation techniques 
appears in Section 3.2.2.

3.2.1.1 APTS Costs and Functional Characteristics

	Central to an operational test evaluation is the 
performance of the APTS system and its individual 
components.  Questions surrounding the costs and 
functional characteristics (including reliability, 
usefulness, maintainability, adherence to specifications) 
should be addressed, and the relationship between these 
APTS characteristics and overall operational test 
objectives should be examined.  Examples of such 
questions are:

27


  . What are the life cycle costs (including fixed and 
    recurring expenses) of the APTS system and its 
    individual components? Which are "start-up" costs 
    associated with the newness of the system and might be 
    avoided in future applications?

  . Is the automated vehicle location system easy to use 
    and are vehicle positions determined quickly and 
    accurately so that on-time scheduling can be carried 
    out and that passengers are provided with timely 
    information?

  . Is the smart card system reliable, and does the system 
    meet the required design specifications?

	To the extent possible, the objective (or 
objectives) related to a particular APTS component should 
be clearly articulated and the specific component costs 
and associated functionality should be determined.  This 
will facilitate the comparison of APTS costs and 
associated benefits.  It is recognized, however, that 
individual component costs may be difficult to determine 
if the procurement process allows lump sum bids.

3.2.1.2 User Acceptance

	The extent to which various APTS applications are 
actually utilized will be an extremely important 
dimension of performance in each operational test.  The 
percentages and numbers of riders using a smart card for 
fare payments are just examples of quantitative 
measurements in this category.  In addition, qualitative 
measures of user acceptance (or utility) would be 
employed, examples of which include the attitudes of 
riders regarding the usefulness of AVL-based pre-trip 
information and the perceptions of dispatchers concerning 
the benefits of component monitoring equipment.

3.2.1.3 System Efficiency and  Effectiveness

	Transit system performance is typically viewed in 
terms of efficiency and effectiveness, both of which may 
be influenced by the use of the APTS application and 
other technology.  Efficiency is related to the extent to 
which system inputs such as vehicles, personnel, fuel, 
and funds are employed to produce outputs; examples of 
outputs include the actual number of vehicle miles or 
vehicle hours of service.  For example, reductions in 
unit operating costs would be

28


examined in part with the use of efficiency measures such 
as the operating cost per vehicle mile or operating cost 
per vehicle hour.

	Effectiveness concerns the users and actual demand for 
service and relates to financial aspects such as revenue 
and cost effectiveness service utilization, quality, 
convenience, safety,	security, and reliability.  In 
addition, non-financial aspects of effectiveness include 
service utilization, safety, security, and service 
reliability.

3.2.1.4 Impacts

	To examine the extent to which the operational test 
responds to critical transportation issues and national 
mandates such as the Americans with Disabilities Act, the 
Clean Air Act, and other Federal legislative efforts, 
both quantitative and qualitative impact measures are 
required. Such impacts may be anticipated or 
unanticipated and positive or negative. These impacts 
relate to, for example, the transit agency and its 
internal activities and administrative procedures; 
aspects of human factors; privacy; and matters dealing 
with equity, social, energy, traffic congestion, air 
quality, special mobility needs, institutional and 
political concerns. For example, the use of a smart card 
might facilitate the implementation of a more equitable 
and efficient fare policy as may have been anticipated, 
but it unexpectedly required a reorganization of the 
transit system's finance department and the existing fare 
collection and accounting activities and procedures.  
Another example concerns the use of an automated vehicle 
location (AVL) system which, as intended, may improve 
on-time scheduling; however, such scheduling improvements 
will only be realized after the transit dispatching staff 
has been properly trained and has learned to use the AVL 
system for the purpose of communicating with the bus 
operators.

3.2.1.5 Relationship Between APTS Program Objectives and 
        the Categories of Measures

	While the six categories of measures discussed 
above are not meant to be exhaustive, they do provide 
structure and guidance in the selection of measures to 
evaluate the APTS program objectives #1$ #2$ and #3 to 
the extent that they are associated with each operational 
test.

29


	The first APTS program objective, as stated in 
Section 2.1.1, focuses on enhancing the quality of 
on-street service to riders in terms of safety, security, 
convenience, ease of travel, and travel time.  These 
concerns fall largely under the categories of APTS 
functional characteristics, transit service efficiency 
and effectiveness, user acceptance, and impacts as 
discussed above.  Examples of corresponding measures 
appear in Exhibit 7.

	The second APTS program objective is to improve 
system productivity and job satisfaction.  Anticipated 
system productivity improvements might result from 
reductions in system costs; better schedule adherence; 
quick and effective responses to incidents and vehicle 
and facility failures; and information management systems 
to provide reliable and accurate operating data in a 
timely manner.  Job satisfaction pertains directly to 
another group of potential APTS beneficiaries; that is, 
the employees, such as drivers, dispatchers, and data 
analysts.  An APTS application may lead to a change in 
the day-to-day activities of such employees and may, in 
turn, lead to reductions in worker stress and increases 
in job satisfaction.  Examples of measures to evaluate 
the association of each test with this objective are 
given in Exhibit 7.

	The third APTS program objective centers around the 
contribution of public transportation to larger societal 
issues and community goals.  These issues and goals 
relate to such elements as special mobility needs, 
traffic congestion, air quality, energy, privacy, equity, 
and other concerns. Appropriate measures to assess this 
APTS objective are mainly included in Exhibit 7 under the 
categories of user acceptance, effectiveness, and 
impacts.

	As discussed in Section 2.1.1., the fourth APTS 
program objective is a somewhat broader objective than 
the other three and consequently, the above measures will 
not be used to measure its level of achievement in each 
test. However, as mentioned in Section 2.1.1., to expand 
the knowledge base, results of tests will be disseminated 
in journals, conference proceedings, electronic bulletin 
boards, technical meetings, and seminars.

	Each category of measures includes criteria 
associated with various aspects of APTS applications 
ranging from their costs and functional characteristics 
to their association with overall transit system 
efficiency and effectiveness and other broader societal 
issues, such as air quality, energy, and special mobility 
needs.  The results of each evaluation will be widely 
disseminated as discussed in Chapter 2, so that 
professionals have access to the knowledge they need 
regarding the actual performance of APTS technologies and 
the use of the analytical

30

Click HERE for graphic.

31


Click HERE for graphic.

32


Click HERE for graphic.


33


techniques employed in the analyses.  The availability of 
such knowledge will lead to the design of improved APTS 
applications, in the conduct of more thorough 
evaluations, and the utilization of enhanced evaluation 
analysis tools.

3.2.1.6 Other Objectives and Measures

	The six categories of measures in Exhibit 7 are 
also useful in the selection of measures for other 
operational test objectives.  As pointed out in Section 
2.1.2, there will likely be state or local objectives in 
addition to the APTS program objectives.  For example, a 
state objective might be to reduce the amount of 
financial operating assistance needed.  This would imply 
that either operating costs must decrease or operating 
revenues (e.g., fares) must increase.  Measures 
associated with this objective relate to system 
efficiency and effectiveness.  Another example might be a 
desire to revitalize the central business district.  
Measures for this objective would fall under the area of 
economic concerns in the impacts category.

3.2.2 Data Collection/Derivation Techniques

Once the relevant measures for project evaluation 
have been determined, it is necessary to identify 
appropriate collection or derivation techniques. 
Collected measures can be obtained through the following 
four basic methods:

(1) By measurements, using various instruments, such as 
    stopwatches, odometers, speedometers, and lap-top 
    computers.  The accuracy of the recorded data is a 
    function of the accuracy of the measuring instrument 
    itself. Typical measurements include travel times and 
    vehicle velocities.

(2) By counts or observations involving tallies either 
    from discrete digitized recording equipment, lap-top 
    computers, or manual counts. Typical counts would be 
    numbers of passengers in vehicles.

(3) By surveys or interviews which provide information 
    relative to the individual being questioned, said 
    information to include such items as origin, 
    destination, income level, previous travel modes, 
    observations of how the service is functioning, and 
    attitudes towards transit amenities.

(4) By searching records such as those available through 
    the transit system, local sponsor, and other local 
    planning agencies and Census records.

34


Derived measures can be calculated either through the use 
of simple arithmetic processes or special analytic 
models.  This form of measures builds upon basic data 
collected through some of the above means.  An 
illustration of a simple derived measure might be 
dividing passengers per day by vehicle miles per day to 
obtain passengers per vehicle mile.  Examples of the 
latter type of derived measures resulting from analytic 
models might be the use of a time-delay curve to estimate 
vehicle speeds or the calculation of reductions in fuel 
consumption and air pollution based on a model using 
changes in traffic volumes as input.

	In view of the large number and variety of measures 
in Exhibit 7 and the even larger number which are likely 
to arise during the course of the APTS Program, it would 
be very difficult to specify in these guidelines a 
preferred method of data collection for each measure.  
Moreover, it would be inappropriate to attempt to choose 
a set of "best" methods from among the techniques already 
tried; rather, it is desirable to encourage the continual 
development and implementation of novel techniques with 
potential for increasing the efficiency or accuracy of 
evaluations.  Finally, there is really no requirement for 
uniformity among data collection techniques, but rather 
there is a need for consistency and comparability of the 
data obtained by these collection techniques.  The 
techniques can differ from project to project, as long as 
they are comparable in terms of accuracy and yield data 
in a form suitable for analysis both within the project 
and among projects.

	For the above reasons, it is not the intent here to 
prescribe a standardized approach to data collection.  
However, it is appropriate to discuss the potential 
applicability of some of the specific techniques, drawing 
where possible from previous experience.

	Exhibit 8 illustrates the range of techniques 
employed for selected measures in past transportation 
projects.5   Specific comments on these techniques and 
general recommendations applicable to collecting the 
measures follow:

(1) Travel time, speed, and vehicle volume data collection 
    techniques can range from manual to automatic.  In general,
    automatic techniques are effective only where the 
    magnitude of data requirements or some -other special 
    circumstances warrant their use.  Some of the more 
    sophisticated automatic procedures are subject to 
    reliability problems.  Failure of

____________________

[5] For further details on collecting transit date, see 
    "Review of Data Collection Techniques," prepared by 
    Booz-Allen & Hamilton, Inc. for  FTA, March 1985

35


EXHIBIT 8. EXAMPLES OF DATA COLLECTION TECHNIQUES
FOR SELECTED MEASURES

Travel times for transit vehicles:
  . On-board checkers or on-street checker with stop watches or
    lap-top computers
  . Time referenced equipment connected to bus

Speeds for transit vehicles and autos:
  . On-street checkers with radar units or other equipment
  . Test vehicle with use of odometer, clock, and other 
    equipment
  . Real-time surveillance system with image processing 
    capabilities

Counting auto occupants:
  . On-street counts recorded on paper, counters, or lap-top 
    computers

Counting transit vehicle passengers:
  . On-board checkers or on-street counts recorded on 
    paper, counters, or lap-top computers
  . Bus drivers recording passenger load
  . Automatic Passenger Counters

Travel times for autos:
  . On-street checkers at selected locations recording license 
    plates and times; calculation of elapsed time by matching 
    plates; possibly in conjunction with video camera and image 
    processing technology 
  . Time lapse aerial photographs or video
  . Floating car with observers to record travel time and 
    stopped time delay using stop watches or other equipment

Counting of transit vehicles and autos:
  . Permanent or temporary tube counters or loop detector in 
    lanes or zones of interest
  . Visual counts recorded by persons
  . Time lapse aerial photographs or video
  . Real-time surveillance system with image processing 
    capabilities
  . Electronic detectors

Demographic/behavioral/attitudinal data on users/non-users/
operators:
  . Post cards distributed to auto drivers at exit ramps, to 
    boarding and on-board passengers, and at park-n-ride 
    facilities
  . Forms, usually no longer than one page, distributed and 
    returned by mail or collected on buses
  . Sampling of autos by noting license plates and subsequent 
    identification through Department of Motor Vehicles files; 
    possibly with video camera and image processing technology
  . Interview conducted either at home, work, or within the 
    transit system itself (on board, at stations, etc.) or with 
    transit or local officials

36


    these devices can cause loss of vital data, which 
    will in turn delay the evaluation, and considerably 
    increase costs.  In addition, the measurement 
    accuracy of automatic or semiautomatic devices may 
    be questionable, particularly if they have not been 
    used extensively before.  In cases where definitive 
    information on devise accuracy is not available, it 
    is essential to confirm the accuracy of 
    automatically collected data by periodic use of 
    manual devices.

    Simple manual devices can be deployed so as to 
    maximize utilization of roadside personnel.  For 
    example, in one project, the use of special 
    counters by each observer permitted keeping track 
    of the auto occupancy of each vehicle counted, with 
    the result that two measures were obtained at once.  
    In other projects, special manual devices were used 
    to obtain vehicle counts and occupancy data 
    simultaneously.

(2) Past experience has shown that there is a lack of 
    consistency between passenger counts recorded by 
    transit personnel and counts by on-board or roadside 
    observers.  For instance, in one project, it was 
    found that bus drivers tend to overestimate the 
    passenger load and that on-board and on-street 
    counters tend, on the average, to be consistent with 
    the other.  If transit personnel are to record such 
    data, it is essential that verifications be made 
    during the project to detect any potential bias or 
    unusual variability in this data.

(3) In utilizing transit system records and service area 
    records, such as census data, it is critical to 
    ascertain accuracy of these data. Usually, 
    discussions with personnel who initially record these 
    data will provide an assessment of accuracy.  
    Further, where special data are collected for the 
    project by a local organization, monitoring 
    procedures will be established to assure that no 
    modifications in procedures or notations have 
    occurred which might have an impact on the evaluation 
    process.

(4) Demographic, behavioral, and attitudinal data on 
    users and non-users of the services provided as part 
    of the operational test, as well as attitudinal 
    information from transit operators, can be collected 
    through a wide variety of survey and interview 
    techniques, with varying degrees of respondent 
    cooperation, accuracy, and cost.  In view of the 
    large amount of documented survey experience relating 
    to both transportation and general market research 
    contexts, and in view of the large anticipated role 
    of surveys in APTS evaluations, Appendix A has been 
    devoted to a discussion of survey design and 
    execution.

	In evaluating the array of existing and potentially 
innovative collection techniques relative to a particular 
measure, some of which are included in Exhibit 8 as 
examples, the contractor should consider factors such as 
the cost and accuracy of each method, the availability of 
local resources to implement each method, the ease of 
implementation, and the ultimate data analysis 
requirements.

37


	With respect to cost, the contractor should apply 
sound judgment in determining whether the anticipated 
cost of using a particular technique is justifiable in 
terms of the contribution to the overall project 
evaluation of the specific measure being collected.  
Clearly, the total project expenditure for data 
collection should be allocated among individual measures, 
taking into account each measure's contribution to the 
project evaluation.  The contractor should make special 
note of any data item which is relevant to the evaluation 
but whose collection cost appears to be 
disproportionately high in relation to other items.

	The contractor should determine whether the 
accuracy of a particular technique is consistent with the 
accuracy requirement for the measure, which in turn is 
dependent on the relative importance of the measure.  A 
very accurate technique is probably not warranted for a 
relatively insignificant measure, especially if that 
technique would be expensive to implement.  In addition, 
a high degree of accuracy for some measures may be 
inconsistent with a lesser degree of accuracy for others.  
The contractor should also evaluate alternative 
techniques in light of the available local resources--
labor resources as well as equipment.  An attempt should 
be made to utilize existing equipment or rental equipment 
arrangements wherever feasible, rather than opting for 
techniques which require the purchase of new equipment 
(which might not be needed by the locality after the APTS 
evaluation).

	The contractor's Evaluation Plan should contain 
justification for selecting the particular technique 
applicable to each measure in terms of these 
considerations.  In the case of a novel technique, it is 
required that the contractor demonstrate acceptable 
accuracy before it can be used as the sole source for 
data collection.  It is further required that the 
evaluation contractor document his experience with those 
data collection methods employed in an evaluation, as 
explained below in Chapter IV.  As this further 
experience develops, the Volpe Center will make this 
information available via updates to this Guidelines 
document.

3.3 PLANNING CONSIDERATIONS RELATIVE TO DATA COLLECTION 
    AND ANALYSIS

	The preceding section contained-guidelines relative 
to specifying appropriate measures and 
collection/derivation techniques for addressing APTS 
Program objectives and other project objectives and 
issues.  This section completes the discussion of 
evaluation planning activities with general guidelines 
for data collection and analysis procedures.  The 
material in this section,

38


while intended to be applied to individual measures 
selected for inclusion in the evaluation, is presented in 
a general context.  The following topics are included: 
basic data collection/analysis design, measure 
stratification, sampling requirements, and the timing of 
data collection.

3.3.1 Basic Data Collection/Analysis Design

	A significant aspect of the evaluation process for 
APTS operational tests is determining the basic data 
collection and analysis design to be employed relative to 
specific project objectives.  There are a great variety 
of potential design approaches, ranging from an 
"after-only" design (a one-shot case study approach 
involving a single set of measurements taken after the 
project is operational) to a "before-after with control 
group" design (involving a comparison of multiple 
measurements).  A General Accounting Office (1991) Report 
entitled, "Design Evaluations," presents guidelines with 
the use of a "decision tree" to assist in the selection 
of an evaluation design including case studies, 
cross-section or panel surveys, comparative group 
analyses, or a before and after study.  A comprehensive 
discussion of the specific utility and the relative pros 
and cons of the various design approaches can be found in 
Donald T. Campbell and Julian C. Stanley, Experimental 
and Quasi-Experimental Designs for Research, 1968, and L. 
Mohr, Impact Analysis for Program Evaluation, 1988.  The 
information which follows is intended to discuss the 
relative advantages of various approaches in the context 
of the APTS program and to highlight the major 
considerations involved in selecting the appropriate 
design for each APTS evaluation, or for individual 
measures included in the evaluations.

	In general, a single set of measurements (for 
example, taken while the test is in operation) will be 
insufficient for assessing the impact of the test, since 
it will not provide any yardstick with which to interpret 
the measurements.  It is recommended, therefore, that 
every data  collection/analysis program be structured 
around some form of comparison.  If such an approach is 
for some reason infeasible, the contractor must indicate 
the reason(s) in the Evaluation Plan.

	Given that the basic data collection/analysis 
design will generally be in the form of a comparison of 
multiple measurements, the next question to be considered 
is what types of comparison are appropriate.  The two 
main forms of comparison are before vs. after and test

39


vs. control.  In a before-after comparison, a given 
measure is collected on a given system element before the 
experimental or exemplary operational test technique is 
instituted and then again while the technique is 
operational.6 In a test-control comparison, a given 
measure is collected on a system element which has been 
affected by the introduction of a technique (test unit) 
and also on an equivalent system element which has not 
been similarly treated (control unit).  Each type of 
comparison is somewhat limited: the before-after 
comparison fails to show what portion of the change in 
the measure is due to external factors; the test-control 
comparison shows the difference between "after" measures 
and hence accounts for external factors, but fails to 
indicate the degree of change from the before state to 
the after state.  Accordingly, it is desirable, where 
feasible, to conduct a before-after comparison in 
conjunction with a test-control comparison.  In other 
words, the data design should, if possible, involve 
collection/analysis observation of both a control and 
test unit before and after the institution of the APTS 
application.

	To make the foregoing discussion more concrete, 
consider a large area with many bus routes and suppose 
that a certain fraction of them are treated in some 
manner (i.e., an APTS application is implemented which 
can be expected to reduce bus travel time).  If 
pre-application and post-application measures of travel 
time are made only on the treated routes and a reduction 
in time is indicated, there is no way of knowing the 
extent to which the improvement is attributable to 
external factors (for instance, a decrease in auto 
traffic on the streets where the buses operate).  In 
order to account for, in a quantitative fashion, these 
known or unknown factors which have arisen during the 
interval between the before and after measurements, it is 
necessary to make before and after measurements of bus 
travel time on routes which are comparable to the test 
routes and therefore susceptible to the same set of 
external factors.  The difference between the travel time 
reduction on the test vs. control routes can then be 
taken as the true change due to the application.  To make 
these statements, it is necessary to be fairly

___________________
 
[6]  As is discussed below, a before-after comparison does not 
     necessarily imply a single measurement before the 
     operational test is implemented and another measurement 
     while it is operation.  Rather, this type of comparison 
     can take the form of a series of measurements prior 
     to, during, and after the operational phase of the 
     operational test.  If the project is implemented is 
     stages, there will be a series of measurements 
     corresponding to each stage.
40


confident that conditions affecting both control and 
experimental units are reasonably similar a requirement 
which is sometimes difficult, if not impossible, to 
assure.

	To reiterate, the proper use of the combined 
before-after/test-control approach guarantees to the 
greatest extent that any observed improvement is indeed 
due to an operational test application.  Thus, the 
contractor should employ both types of comparisons 
wherever appropriate and feasible.  The determination of 
appropriateness of the combined approach involves a
consideration of the time span of the operational test.

	Regarding the scope of the project, the larger the 
geographic area encompassed by or affected by the 
project, the greater the possibility that no control 
units can be identified (i.e., the entire population is 
composed of test units).

	Regarding the time span of the project, no 
generalizations can be made since tests will vary in 
length depending on a variety of factors.  As a general 
rule-of-thumb, the desirability of the combined 
before-after/test-control approach increases with the 
time span of the project, since this approach reveals 
internal as well as external changes occurring over the 
project's duration.  The determination of feasibility of 
the combined approach involves questions of data 
availability and project timing.  If there is a known 
deficiency in either type of comparison, then only the 
valid comparison should be employed; it is generally 
better to do without a before observation or a control 
observation than to settle for unsuitable before or 
control data.  In the event that only one type of 
comparison is feasible, there are alternative techniques 
and precautionary measures available to the contractor to 
compensate for the absence of the other type of 
comparison.

	If no control group exists (e.g., if the 
operational test affects the entire population of 
observation units, making each one a test-unit) or if no 
suitable group can be found (each test unit is unique), 
then the contractor should be especially observant 
throughout the evaluation period of possible external 
factors which might influence the interpretation of 
project results.  Any statistics regarding the before vs. 
after change due to the applied technique should be 
examined very carefully in the context of these observed 
external factors, and any conclusions based on such 
statistics should be qualified accordingly.

	If, due to project timing, there is no opportunity to 
perform before measurements, or if it is known beforehand 
that the units to be observed will undergo considerable 
change between

4l


the before and after periods, the contractor should 
attempt to obtain surrogate data for the before period.  
Possible sources of surrogate data would include: (1) 
surveys conducted after the test is operational which 
question people about conditions or their behavior prior 
to the implementation of the technique; and (2) 
demographic and travel data collected by the local 
highway department, planning agency, or transit operator 
some time prior to the operational test.  The surrogate 
data can be used to provide some indication of the 
magnitude of the before-after change experienced by the 
test and control groups.

	In using the before-after and/or test-control 
approach, one of the key steps is identifying comparable 
units.  To as great an extent as possible, the units 
observed for the before case must be equivalent to the 
units observed for the after units.  Returning to the 
previous example of bus routes, before-after 
comparability is not a difficult problem, since the same 
routes can be observed for both time periods.  The only 
note of caution is that the routes should be unchanged 
(with respect to length, number and location of stops, 
etc.)  from one measurement period to the next.

	Test-control comparability, on the other hand, 
raises some interesting problems.  Theoretically, the 
test and control units should be as nearly alike as 
possible to rule out any chance of the observed change 
being a result of something other than the operational 
test application.  Test and control units should be 
chosen which are similar in terms of variables assumed to 
be related to the particular measure.  Again, using the 
example of bus routes and the measure travel time, 
matching of test and control routes could be done on the 
basis of such descriptors as route length, total trips 
along the route, peak headway, and average speed.  The 
Volpe Center's Evaluation Strategy will generally suggest 
the basic data  collection/analysis design to be employed 
for each project as a whole or for particular measures 
(e.g., before-after comparison, test-control comparison, 
both types of comparison, or a single set of 
measurements.  The contractor should determine the 
feasibility of such suggestions in terms of the data 
availability and time frame of the particular project and 
site.  The contractor's Evaluation Plan should then 
elaborate on the approach finally selected for each 
measure, indicating information such as the specific 
units chosen for the control and test groups.
42


3.3.2 Measure Stratification

	Measure stratification refers to the categorization 
of individual measures for collection/derivation and/or 
analysis purposes.  Examples of measure stratification 
are:

(1) peak versus off-peak time periods,
(2) day of the week,
(3) revenue (in-service) versus non-revenue service,
(4) waiting, access, transfer and in-vehicle travel times, and 
(5) fixed route versus demand responsive.

Measure stratification improves the quality of the 
evaluation by allowing an assessment of how changes in 
measures relate to the stratification categories, hence 
facilitating the formulation of more specific findings 
and conclusions.

	Whereas collection of an unstratified measure 
provides only a single, average reference point, the use 
of a stratified measure provides a series of reference 
points, each of which may be significant to the analysis 
and interpretation of results.  Knowledge of 
inter-category differences in results enhances 
transferability; for instance, if a particular 
operational test proves to be especially beneficial in 
congested areas but of limited value in sparsely traveled 
areas, then other sites considering implementation of the 
service will know to focus their efforts in congested 
areas.

	Stratification can take the following forms:

(1) categorization of a measure into additive components 
    (e.g., measuring person trip time in terms of trip 
    components such as access time, line-haul time);

(2) categorization of a measure, and possibly its components, 
    according to target market, operational, geographic, or 
    time categories (e.g., measuring trip time for peak and 
    off-peak periods); and

(3) grouping of raw values of a measure into class intervals, 
    with class intervals determined either before or after data
    collection (e.g., determining the distribution of early, 
    late, and on-time arrivals).

	It is not possible apriori to present a 
standardized approach to be used for each measure.  
Clearly, the appropriate type and level of stratification 
depend on the particular measure and on the 
characteristics of the site and project.  However, in 
order to provide the contractor with

43


some guidance in this area, examples of possible types 
and levels of stratification are presented below.

3.3.2.1 Categorization of a Measure Into Additive Components

	This form of stratification involves collecting and 
reporting data separately for specific components, or 
sub-breakdowns, of a measure.  The purpose of 
categorizing in this manner is to single out the effect 
of an APTS application on these specific components.  
Examples of this form of stratification are available for 
measures relating to travel time, reliability, and 
productivity.

Person transit trip time for fixed route systems 
can be broken into segments as depicted in the following 
diagram:

Origin                                        Destination



where:

  Segment A = Access time
  Segment W = Waiting time for first vehicle or for subsequent 
                      transfer
  Segment T = In-vehicle transit time
  Segment E = Egress time
  ti        = Time for ith trip segment

	If further amplification is desired, access time 
and egress time can be subdivided into walking, riding, 
and other portions; or in-vehicle transit time can be 
subdivided into collection, line-haul, and distribution 
phases.

	In the case of demand-responsive systems, some of 
the trip time components might take	on a different 
definition: for example, access time would be zero, and 
waiting time would refer to the difference between the 
caller's requested time of pick up and the arrival time 
of the vehicle at the origin. In cases where the caller 
is told that pick up can only be made later than

44


the requested time,7 wait time can be further divided 
into the time between the requested pick-up time and the 
promised pick-up time, and the time between the promised 
pick-up time and the arrival time of the vehicle at the 
origin.  This latter travel time component, is, in 
itself, a basic transit system reliability measure in the 
category of effectiveness measures summarized in Exhibit 
7.  In-vehicle transit time, if desired, can be divided 
into the direct routing travel time (the time between the 
person's origin and destination if no other pick-ups or 
drop-offs are made) and the detour travel time (the time 
spent detouring to make other pick-ups and drop-offs).

Transit vehicle time is always to be broken into 
in-service time and non-service time.  However, if 
desired, these two prime categories can be further 
divided as indicated below.

For fixed route systems:

	In-service
		In motion
		Loading
		Non-productive -- waiting for lights, metering, 
		                  or other obstacles to motion

	Non-service
		Garage to first service point
		Last service point to garage
		Dead turnaround time
		Deadhead time
		Other

	For demand responsive systems:

	In-service
		In motion with one or more passengers onboard
	   	In motion with no passengers onboard and in the 
		   act of picking up one or   more passengers
  		Loading

	Non-service
   		Garage to first pick-up point
   		Last drop-off point to garage
   		Between first pick-up point and last drop-off 
		  point with no passengers onboard and not in 
		  the act of picking up one or more passengers
________________________
[7]  Due to the potential ambiguity associated with requests for
     immediate service, the contractor should note how the 
     particular transit operator maintains data on requested 
     and promised pick-up times.

45


These time segments are depicted in the following 
diagram:


where:
Point A = Garage
      B = First pick-up point
      C = Drop off point -- no passengers on vehicle but 
	                    driver is instructed to proceed 
                            immediately to pick up a passenger

      D = Pick-up point
      E = Drop-off point -- no passengers on vehicle and 
			    there are no requests for 
			    immediate pick-up; driver is 
                            instructed to proceed to 
			    central waiting point

     F = Point enroute to central waiting point-- driver is
                                                  instructed 
						  to proceed 
						  immediately 
						  to  pick 
						  up a passenger

    G = Pick-up point
    H = Last drop-off point of day
    I = Garage

Note that in segments BC and GH pick-ups and drop-offs 
are being made and at least one passenger is always 
onboard.  Also, all pick-up and drop-off points include 
time spent waiting for riders to board and deboard 
vehicles.


For operating costs of APTS operational tests, it 
has been decided that the aggregation of cost items 
should be consistent with FTA Section 15 expense 
categories.  Exhibit 9 is a matrix showing the 
distribution of expense object classes into functional 
areas under Section 15.

46


Click HERE for graphic.

47


	Because of possible differences in current internal 
accounting practices, it is essential that any techniques 
for disaggregation and allocation of costs be described 
in the Evaluation Plan.  In addition, because of 
different funding mechanisms, it is important to review 
in depth individual transit authority practices.  It is 
also recognized that the reporting of operating costs 
should be carried out using a consistent time frame for 
reporting periods.

3.3.2.2 Categorization of a Measure According to Target Market, 
        Operational, Geographic, or Time Categories

       The primary purpose of this form of stratification is to 
evaluate the effect of APTS applications in different 
contexts.  As in the case of categorization into additive 
components, this form of stratification involves 
collecting and reporting measures separately for each 
category.	Examples are as follows:

	Target Market:
		Trip purpose -- work/non-work
		User group -- commuters/non-commuters
		Mode -- auto/transit/other

	Operational:
		Type of transit service -- express/local; fixed 
		  route/demand responsive
		Direction of traffic flow - inbound/outbound
		Type of thoroughfare -- freeway/arterial

	Geographic:
		Within/outside central business district
		Zones with different demographic characteristics

	Time:
		peak/off-peak
		weekday/week-end

	Finer stratification in the above examples is also 
possible.  For instance, within the target market 
category, the trip purpose "non-work" can be divided into 
medical, social, recreational, etc.; non-commuter can be 
stratified into elderly, disabled (ambulatory and 
non-ambulatory) unemployed, etc.; and mode can be divided 
into solo driver auto, carpool auto, chauffeured auto, 
and specific local transit service options.  Types of bus 
service can be divided into local feeder, local 
line-haul, and express line-haul, and further divided 
into individual routes, and

48


beyond that into route segments.  Time of day can be 
refined into the four Section 15 categories (A.M. peak, 
midday, P.M. peak, night) or even further into hour, 
half-hour, or 15-minute segments within certain 
categories.

	In general, in some instances it will be desirable to 
partition collected data into various target market 
categories, since most operational tests will probably 
consist of specific innovations aimed at particular user 
groups.  The decision as to whether to stratify collected 
data by operational and geographic categories depends on 
the nature of the project and thus will have to be made 
on a case-by-case basis.  However, it is recommended that 
serious consideration be given to using a minimum time of 
day stratification (peak/off-peak) for every measure, 
since many transit system operating characteristics as 
well as general traffic conditions vary widely between 
peak and off-peak periods.  The decision as to 
stratification of data collection within the peak period 
(i.e., morning vs. evening peak) and within the off-peak 
period (i.e., midday vs. nighttime) should be made in 
accordance with the time of APTS service operation 
throughout the day and the variability of travel 
conditions and other relevant factors between the 
different categories.  It is important to note that the 
peak period may be a changing period depending upon 
distance from the CBD and type of transit system. Other 
issues regarding data stratification and analysis are 
discussed in Section 4.2.

3.3.3 Grouping of Raw Data Into Class Intervals

	Measure stratification can also refer to the 
grouping of raw data into intervals, with intervals 
determined before or after data collection. Whereas the 
first two forms of stratification involve collecting and 
reporting a measure separately for each category (e.g., 
change in travel time during peak periods, off-peak 
periods), this type of grouping produces a frequency 
distribution for the particular measure.

	Survey data on traveler behavior, characteristics 
and attitudes is a good example of pre-collection 
determination of intervals.  For instance, comparisons of 
users and non-users of an APTS test can be made using 
distributions of-such measures as age, income, auto 
availability, and attitudes toward transit, with the 
particular response categories of each measure having 
been determined beforehand.  Appendix A contains 
recommended response categories for selected

49


demographic and travel behavior measures, as well as 
sample questions and response categories for selected 
attitudinal data.

	Reliability measures provide examples of intervals 
that can be determined after data collection.  The 
difference between scheduled and actual arrival time at 
an access point would be collected in its raw form (i.e., 
each vehicle's time difference in minutes), but would be 
reported as a frequency distribution. A suggested minimum 
stratification of this measure is:

	% early
	% on time (vehicles arriving within +x or -y 
	  minutes of scheduled time) 
	% late

The contractor should be aware of differences in transit 
company standards with respect to schedule adherence, and 
the potential impact on data collection and analysis 
procedures.

	Vehicle delays due to breakdowns can be grouped 
according to the following minimum stratification:

	% No delay (delay of 2 minutes or less) 
	% Delayed
	% 96 Total disruption of service

	If further detail is desired, the late category under 
schedule adherence and the delayed category under vehicle 
reliability can be divided into categories such as: 1-5 
minutes delay, 6-10 minutes delay, over 10 minutes delay.

	The basic intent of grouping is to summarize the 
raw data without masking the real form of the 
distribution for a given measure.  In addition, the 
extent of grouping may also depend upon the specific 
analyses which are planned.

	Interval grouping can be used in conjunction with 
either of the two forms of stratification previously 
discussed.  For instance, person trip time can be 
stratified into components (access time, etc.), and time 
period (peak vs. off-peak) can be grouped into 5 or 10 
minute intervals to obtain a frequency distribution.

	As was stated above, it is not possible in these 
guidelines to present a standardized approach to 
stratification for each measure.  The contractor will 
therefore have to rely on judgment and past experience to 
determine which types of variable stratification are most 
likely to enhance understanding of specific areas of 
project effectiveness and potential application.

50


The contractor should plan data collection activities 
with the finest stratification which can be justified as 
appropriate for the APTS objectives.  Since the ultimate 
sample size will be directly related to the number of 
categories employed, the contractor should make sure that 
the available sample units are sufficient to support the 
level of stratification deemed desirable.  The Evaluation 
Plan developed by the contractor should contain 
justification for the type(s) and level of stratification 
selected, as well as evidence that such stratifications 
are feasible from the standpoint of data and sample size 
availability.

3.3.4 Sampling Requirements

	Once the contractor has determined the basic data 
collection/analysis design for the project evaluation and 
the type(s) and level of stratification for each measure, 
the final question to be addressed is sampling 
requirements.

	In general, data required from records maintained 
by the transit operator or other organizations should be 
available on a continual basis over the entire lifetime 
of the experimental test and such data should not require 
sampling.  On the other hand, data obtained from 
measurements, counts, and surveys will generally not be 
available on a continual basis but will have to be 
collected in the form of samples.  There may also be 
situations where measurements or counts yield continual 
data, but sampling is desired in order to reduce data 
processing expenses.

	When collection of a particular measure involves 
sampling, an estimate of the minimum sample size must be 
made prior to the initiation of the data collection 
effort.  In estimating sample size requirements, the 
objective is to have a large enough sample to be able to 
draw valid inferences about the population from which the 
sample is drawn.  As might be expected, the determination 
of appropriate sample sizes involves trade-offs between 
the desired level of precision and the cost of data 
collection.  These trade-off decisions in turn require a 
determination, during the evaluation planning phase, of 
the appropriate types of analyses to be performed 
(e.g.,-estimates of population parameters, 
comparisons-between two or more groups of sampled data).

	Appendix B presents specific guidelines relevant to 
estimating required sample sizes.  Included in the 
discussion are: (1) references to statistics books 
containing sample size

51


equations, (2) recommendations regarding values for the 
three input factors in the sample size equation, and (3) 
suggestions regarding implementation of the field data 
collection effort based on the calculated sample size 
values. Appendix B also contains a section on the basic 
types of possible statistical analyses, appropriate 
confidence levels, and desirable reporting formats.

	The contractor should follow the guidelines in 
Appendix B to develop appropriate sample sizes for each 
measure.  The Evaluation Plan should contain the sample 
size values, along with an explanation of any assumptions 
or special procedures underlying these values (e.g., 
equations, input factor values used).

	3.3.5 Timing of Data Collection

	For measures based on sampling, another issue to be 
addressed by the contractor is the timing of data 
collection.  The exact periods during which measures are 
collected have a significant effect on the validity and 
representativeness of evaluation results, since the 
operation and effectiveness of a transportation system 
are sensitive to various factors associated with time.

Four basic questions arise concerning the timing of data 
collection:

(1) The appropriate season(s) of the year and day(s) of the 
    week to  include in the sample,
(2) The appropriate duration of each data collection period,
(3) The proper time to initiate data collection, and
(4) The appropriateness of "one- shot" vs. periodic monitoring

	The particular season(s) and day(s) depend largely 
on the assumed sensitivity of the APTS application to 
each time unit.  If it is deemed appropriate to assess 
the impact of the APTS application under reasonably 
normal conditions, data collection should be performed 
during the fall and spring, when weather conditions are 
not severe, schools are in session, and few people are on 
vacation.  To the extent that the experimental test 
evaluation involves measures related to travel patterns 
and transit usage, the contractor should attempt to 
schedule data collection activities during those two 
seasons which are most representative of normal 
conditions.  On the other hand, if severe weather 
conditions or other atypical conditions are an inherent 
feature of the site and it is desirable to examine the 
experimental test under a full range

52


of possible conditions, the contractor should schedule 
data collection throughout the year so the sample 
observations include extreme as well as normal 
conditions.

	If a particular transit service operates seven days 
a week, then the sample of days should include both 
weekdays and week-end days (in fact, the data should be 
stratified by weekday vs. week-end day to highlight the 
differences during these two periods).  Regarding which 
day(s) to include in the weekday sample, similar logic 
applies as in the case of seasons.  If the aim is to 
observe the project under typical weekday conditions, 
then any day(s) with abnormal traffic patterns should be 
avoided.  In some cities, there is a difference between 
Monday and/or Friday conditions vs. 
Tuesday/Wednesday/Thursday conditions; if this is known 
to be the case for a particular test site, then data 
collection should be scheduled for the three "typical" 
days rather than either of the typical days.  The 
contractor should consider the special characteristics of 
the operational test and the site in deciding which days 
are appropriate.  If a large number of days is going to 
be involved, and there is no particularly significant 
distinction among days of the week, then a randomly 
selected sample of days would be preferable.

	The duration of each data collection period should 
be determined based on the degree of day-to-day 
variability and on the required sample size.  If the 
particular item being measured is suspected to vary in 
behavior from one day to the next, then the data 
collection period should include several days; if it has 
been determined that only Tuesdays, Wednesdays, and 
Thursdays can be used, then several weeks may be 
necessary to achieve the required sample of days.  
Moreover, if the sample size required for a particular 
variable is large, then several days of data collection 
may be appropriate to obtain the minimum sample of 
observations.

	The choice of initiation time for each data 
collection period is dependent on a number of 
considerations, the chief one being that the "after" data 
collection not begin until the use of the APTS 
application is fully operational and its performance has 
stabilized.  In general, it will probably take at least a 
few months for an APTS application to become fully 
operational, with all the "bugs" worked out, and possible 
behavioral influences associated with the application are 
eliminated.  The desire is to achieve a "steady state" 
for the system after the application has been 
implemented.  The time to achieve this "steady state" 
undoubtedly will vary from project to project.  Thus, 
data collection related to the test should not commence 
until these adjustments and modifications are completed.  
Other factors determining the initiation date for data 
collection

53


are the desire to avoid summer and winter months 
and the overall schedule of the operational test.

	In most instances, data collection will be 
performed for discrete phases of the operational 
test (i.e., before the project is implemented, 
while the project is operational, and possibly 
after the project is terminated).  
Post-operational test data collection would only 
be performed if there was a desire to see whether 
operation of the APTS experiment for a limited 
period had led to permanent changes in people's 
travel patterns or attitudes.  However, if 
operational test elements are by nature changing 
continually or if it is expected that the APTS 
application will cause gradual but continual 
changes in transit performance measures, then a 
periodic process of data collection would be more 
appropriate than merely "before," "during," and 
"after"  data collection.  The multitude of data 
points obtained from a periodic monitoring 
process will make possible the examination of 
functional relationships either among measures of 
interest or in a time series.   Moreover, 
monitoring of certain measures during the early 
months following introduction of the 
application(s) may be useful in determining when 
the effects have stabilized enough to initiate 
full-scale data collection.  It should be noted 
that if periodic data collection is appropriate, 
then a sequential analysis procedure (similar to 
control charts) may be useful to permit 
reductions in sampling requirements.

The contractor's Evaluation Plan should 
indicate the exact timing of data collection for 
each measure involving sampling.  This 
information should be presented in a schedule 
which also shows the projected implementation 
dates for the various elements of the project.

54


4. GUIDELINES FOR PERFORMING EVALUATION ACTIVITIES

	This chapter presents suggestions relative to 
implementing the evaluation of an APTS operational test.  
During the evaluation implementation phase of the 
evaluation process, data collection/analysis relating to 
site characteristics, quantitative measures, and 
qualitative measures is undertaken according to the plans 
and procedures laid out in the Evaluation Plan.  In 
addition, information is gathered relative to the 
project's operational history and external events which 
may have some bearing on the project outcome.  This 
information is eventually incorporated into the analysis 
and interpretation of project results.

	Contractor functions during the evaluation 
implementation phase include monitoring and/or 
performance of data collection activities, data reduction 
and analysis, subjective analysis of information relative 
to project issues, and synthesis of project findings into 
a Final Summary Evaluation Report.  In accordance with 
these contractor functions, this chapter of the 
guidelines is organized into two sections: (1) 
monitoring/performance of data collection and (2) data 
reduction, analysis, and presentation.  The recommended 
content and organization of the various contractor 
reports prepared during this phase are presented in 
Chapter 5.

	During this phase, the contractor must maintain a 
sensitivity to the relationships among the organizations 
involved in the project -- in particular the local 
sponsor or project team, FTA, and the Volpe Center (see 
Chapter 2).  The contractor must work closely with these 
groups at the appropriate times, while maintaining the 
role and perspective of an external, objective 
organization assessing the impact of the operational 
test.

4.1 MONITORING/PERFORMANCE OF DATA COLLECTION

	Since much of the data required for evaluations 
will be unavailable from pre-existing data bases and 
secondary sources, each operational test will undoubtedly 
involve significant data collection efforts.  Given the 
considerable amount of time and money which will be spent 
on data collection; careful management and oversight of 
the data collection process are essential.  Where 
possible and appropriate, data collection may involve the 
use of students from local colleges and universities.

55


	The contractor is responsible for ensuring that 
data collection is performed according to the Volpe 
Center/FTA-approved Evaluation Plan.  There are three 
potential alternatives associated with data collection.  
One of these occurs when the local sponsor or operator 
collects all data (under FTA/APTS and/or local funding), 
and the contractor acts in a monitoring role to assure 
the quality and timeliness of data collected, as well as 
adherence to procedures laid out in the Evaluation Plan.  
A second alternative occurs when the contractor collects 
the data, and coordinates the timing and performance of 
these activities through the local sponsor or operator.  
The third possibility is one in which both collect 
various elements of the data.

	In order to monitor and/or perform the data 
collection activities called for in a given evaluation, 
the contractor will need to maintain open channels of 
communication with the site, in the form of visits, 
telephone and written correspondence with the appropriate 
local agencies as well as subscriptions to local 
newspapers.  In the rare instance where day-to-day 
contact with the site is necessary, the contractor should 
arrange to base a member of the firm at the site.

	Whether data collection is being performed by the 
contractor or by the local sponsor, the contractor must 
stay closely involved in all phases to make sure the 
procedures specified in the Evaluation Plan are followed. 
In cases where the local sponsor or other local agency is 
collecting data, the contractor should meet frequently 
with the agency to discuss progress and problems, work 
out solutions to the problems, and observe key phases of 
field data collection.  In addition, the contractor 
should occasionally perform independent spot checks, 
especially in the case of measures for which the local 
agency has limited experience in data collection.

	The contractor is expected to inform the Volpe 
Center of the status of data collection in its Monthly 
Evaluation Progress Reports (see Chapter 5 for the 
recommended content and organization of this type of 
report).  Should there be an unacceptable degradation of 
quality or timeliness of data collected by the local 
sponsor, the contractor should notify the Volpe Center in 
writing.  The Volpe Center will in turn take steps 
through the FTA Project Manager to rectify the situation.

	Over and above monitoring data collection 
activities, the contractor should keep abreast of the 
status of the operational test. This awareness of project 
operational status is important so that: (1) data 
collection activities can be smoothly coordinated with 
ongoing project activities (causing minimum disruption of 
day-to-day operations), and (2) evaluation results can be

56


interpreted in the context of project history.  The local 
sponsor's quarterly project progress reports to FTA/Volpe 
Center (see Chapter 5 for recommended content and 
organization) will be a useful source of information on 
the project's operational evaluation.  However, the 
contractor is encouraged to obtain a more detailed 
account of progress/problems relative to implementing and 
operating the APTS test by talking with the local sponsor 
at the site.

	In addition to keeping abreast of project 
operations, the contractor should be continually watching 
at the site for unexpected (external) events which might 
affect the validity of project results.  In any 
implemented operational test, no matter how well 
controlled or planned, the possibility remains for 
unexpected events to occur that may have an impact on 
measures of the project