Skip navigation to main content
United States Department of Transportation - Federal Highway Administration FHWA HomeFeedback

APPENDIX C - OHIO DEPARTMENT OF TRANSPORTATION CASE STUDY

Introduction

This case study describes procedures for calculating the data quality measures in a specific setting: statewide traffic data collection and dissemination by a traffic monitoring group. Calculation of quality measures for the traffic monitoring program is different from the ITS-based traffic operations. This case study is based on data provided in part by the Ohio Department of Transportation (ODOT) and partly on hypothetical assumptions.

Traffic Data Flows: Identifying the Data Consumers

Figure C.1 illustrates the data flows involved in traffic data collection, dissemination, and archiving showing details related to the specific context of traffic monitoring perspective. The following are the main sources of data whose quality should be represented in the data quality measures.

The following sections present the calculation of the six data quality measures for the three data sources described above.

In diagram, Ohio traffic data collected by continuous and short-term methods flow to traffic monitoring groups and planning office. These data are validated and archived. Archived data flow to the FHWA, MPOs, and state DOTs.

Figure C.1. Data Flows and Consumers – Ohio Case Study

Accuracy

Accuracy is defined as "the measure or degree of agreement between a data value or set of values and a source assumed to be correct." As its definition indicates, accuracy requires "…a source (of data) assumed to be correct." This correct source of data is typically referred to as ground truth, reference, or baseline measurements. Ground truth data can be collected in several different ways for each type of traffic data. Calculation of accuracy in a traditional traffic monitoring context is a very difficult task. Often times, the only version of the ground-truth or the reference values are from manual counts, which are expensive to perform and are also not error-free.

Original Source Data – Continuous Counts

ATRs and AVCs

In this example, four hours of data from manual counts and vehicle classification is compared to the data reported by the traffic counter as shown in Table C.1. The manual count is assumed to represent the ground truth or baseline. The accuracy of ATR total volume counts compared very well with the total volume from manual counts. The RMSE for the total volume counts is about 4 vehicles with MAPE of only 0.89%. However, vehicle classification accuracy varies depending on the class of vehicle and quite significant for some vehicle classes. This could be due to the programmed class tables or the classification bins or a pointer to the need for recalibration of equipment.

This example is based on data from on tests performed on new equipment. The performance of the equipment in the field might be significantly different. It is recommended that the accuracy measure is calculated at random or periodic intervals on equipment installed in the field. While this is desirable, the cost implications should be taken into account. Accuracy tests can be performed during routine or periodic maintenance or calibration visits.

Table C.1. Comparison of Manual and ATR Counts for Vehicle Classes and Volumes
Vehicle Class Hour 1
Manual Count
Hour 1
Detector Count
Hour 2
Manual Count
Hour 2
Detector Count
Hour 3
Manual Count
Hour 3
Detector Count
Hour 4
Manual Count
Hour 4
Detector Count
Accuracy Measures
RMSE
Accuracy Measures
MAPE
Class 1000000000 
Class 215314613912712912316614712.17.33%
Class 347525361445754618.817.06%
Class 41504080148.5100.00%
Class 51614181526131878.735.07%
Class 655141554961.715.12%
Class 7000300001.5 
Class 81519913151714235.437.18%
Class 91871861901911971961911910.90.39%
Class 103364772218.33%
Class 119101719161715151.27.28%
Class 123322111100.00%
Class 130000000000.00%
TOTAL4394434484544404434704674.20.89%

WIM Data

This example illustrates the calculation of accuracy measure using data collected during WIM station calibration. Table C.2 describes three runs over a WIM station with a known truck configuration (i.e., baseline). As shown in Table C.2, the WIM data corresponds well with actual configurations with vehicle length being the parameter with highest error. For the three runs during this calibration, the accuracy measures for length and weight measurements at the WIM station can be calculated as follows.

Accuracy of vehicle length measurement

Accuracy of gross vehicle weight measurement

Table C.2. WIM Calibration Report from DOT
Measurements Baseline Run 1
Value
Run 1
% Error
Run 2
Value
Run 2
% Error
Run 3
Value
Run 3
% Error
Dimensions (ft)
Axle1-2
12.612.60.00%12.50.79%12.7-0.79%
Dimensions (ft)
Axle 2-3
4.54.6-2.22%4.50.00%4.50.00%
Dimensions (ft)
Total Length
20.221.2-4.95%23.3-15.35%19.91.49%
Weight (kips)
Steering axle
13.4613.12.67%15.3-13.67%12.94.16%
Weight (kips)
Drive tandem axle
28.56275.46%2512.46%27.63.36%
Weight (kips)
GVW
42.0240.14.57%40.24.33%40.53.62%
Speed (mph)55, 53, 55550530550

Note: Percent errors are calculated relative to the baseline value.
Negative error indicates that the baseline value is lower.
Allowable tolerances (source, ODOT)

The MAPE indicates the mean absolute percent error of total vehicle length over the three runs for the WIM station. The percent error was 7.26 percent. The RMSE for vehicle length was about 1.89 feet which is over the tolerance of 12 inches. Similarly, for GVW, the detector was in error by 4.17 percent (and RMSE of 1.76 kips) which is less than the 15 percent allowable tolerance for the truck used in the test.

Original Source Data - Short-Term Counts

For short-term volume and classification data collection using portable equipment - the accuracy measure is calculated at the time of procuring the traffic counters. A manual count is performed and compared with data reported from the counter in each of the vehicle classes. The manual count is assumed to be the ground truth. Since this is a single count, the MAPE and RMSE cannot be calculated for the particular detector. A more useful measure of accuracy in this case is the percent or the actual error in each of the 13 vehicle classes and in the total volume. (Figure C.2). The figure indicates that there was a positive error (undercount) of 2 vehicles for Class 2 and Class 4, whereas a negative error (over count) of 4 vehicles was reported for class 3. All other vehicle classes were correctly classified and counted with no errors. Also, it is noted that while there are small classification errors, the total volume is accurate as the errors cancel each other out.

Chart shows differences between manual or ground truth counts and detector counts for Vehicle Classes 2, 3, and 4 in accuracy test of DOT Detector #3 for Ohio. Classes 2 and 4 show an undercount of two vehicles, Class 3 an overcount of four vehicles.

Figure C.2. Accuracy Test of Detector #3
(Difference between manual and detector counts)

Table C.3 above shows the manual counts compared to five detectors from the same vendor. In this case MAPE and RMSE can be calculated across detectors for each vehicle class and total volumes. These measures represent the detection capability of the set of detectors. The accuracy measures for two classes vehicles with the highest errors compared to total volume are shown below.

Table C.3. Accuracy Tests of Five Detectors from a Vendor
Vehicle Class Baseline Detector 1
Count
Detector 1
Diff.
Detector 2
Count
Detector 2
Diff.
Detector 3
Count
Detector 3
Diff.
Detector 4
Count
Detector 4
Diff.
Detector 5
Count
Detector 5
Diff.
111010101010
2399399039903972401-2403-4
31381380140-2142-413801371
444040224040
539390390390390390
635350350350350350
722020202020
837370370370370370
964764346443647064436425
1055050505050
1136360360360360360
1211010101010
1311010101010
Total13451341413441134501344113432

Class three volume
– MAPE – 1.01%,
– RMSE – 2 vehicles

Class nine volume
– MAPE – 0.46%
– RMSE – 3.43 vehicles

Total volume
– MAPE – 0.11%,
– RMSE – 2 vehicles

These accuracy measures indicate that the volumes for class three vehicles measured by the detectors have an error of 1.01 percent or 2 vehicles associated with them. However the total volume has a much lower error of about 0.11 percent and a root mean square error of 2 vehicles.

As with continuous counts, these accuracy numbers can be misleading because the tests may not necessarily represent long term field conditions. It may be useful, however informally, to collect at least 5 minutes (or 100 vehicles) worth of data manually at the beginning of each short term count. The data from the manual counts and the data collected by the traffic counter can then be compared to establish count and classification accuracy in a more realistic manner.

Archived Data

Accuracy for archived data is a function of the processes used to generate some of the outputs of traffic data including AADTs, classification and WIM data. AADT especially from short-counts is adjusted for weekly and seasonal variations. While these processes may be estimates, it is often difficult to determine the reduction in accuracy due to these processes for the lack of reference values. Nonetheless, it is possible to generate some accuracy estimates. For example, by using historical data, forecasted AADTs, and current estimates, it is possible to determine if the data follow existing trends. A problem with trend analyses for such data is that while it identifies anomalies, there is no way of determining if the anomaly is an error or an unusual but true value (for example, spikes due to incidents, construction etc.).

In addition, it is possible to have an idea about the quality of AADT estimates if the accuracy measures of the ATRs and the short-count equipment are available. Since AADT = ADT (from short counts) * Adjustment factors (from ATRs), data derived from high-quality ATRs and accurate short-counts is expected to result in better AADT estimates as shown.

Completeness

Original Source Data - Continuous Counts

For continuous counts, the completeness measure used by the DOT is the number of complete days of data in a month. Complete data is characterized by hourly records for each day of the month containing volume, and classification data for each lane being monitored.

ODOT has a total of 220 ATRs statewide. Data is aggregated in 60 minute intervals. Completeness can be calculated for traffic volume data in two ways. In the first approach, the calculation of completeness assumes a perfect data collection situation where all ATRs record data in all the days of a given month (30 days average). Hypothetically, assume only 140 ATRs have no missing records. In that case, completeness will be calculated as follows:

Total Expected Records – 24 (hours)*30 (days)*220 = 158,400
Records with no data missing – 24*30 (days)* 140 = 100,800
Completeness = (100,800/158,400)*100 = 63.6 %

The second case, which is more realistic, the DOT uses data from a particular ATR if there are at least 14 days of useable data in a given month. On an average, there are 190 sites with sufficient data to generate a monthly ADT (i.e., at least 14 days of 24 hour worth of data). In this case, completeness for traffic data volume can be calculated as shown below:

Minimum Expected Records – 24 (hours)* 14 (days) * 220 = 73,920
Available Records – 24 (hours) * 14 (days) * 190 = 63,840
Completeness = (63,840/73,920) * 100 = 86.4%

It is important to note that the completeness measure calculated above is a good indicator of completeness for monthly ADT calculations only. These measures will need to be recalculated if the agency requires more than 14 days of data as a minimum.

Original Source Data - Short-Term Counts

For short-term counts, usually for 24-48 hours aggregated in 60 minute intervals, the completeness measure is slightly different because the agency has the option of resetting the count and collecting data again. The DOT has a goal of 4,200 short-term counts annually. Incomplete counts are not used. If the count is not complete, they are reset in the field. There is no available statistics on this number. All counts are 24-48 hours in duration and are distributed as follows:

The completeness measure for short-term data collection, as defined, is 100%.

Archived Data

The completeness measure for data users is determined by their applications. As a hypothetical example, FHWA could define completeness of DOT data based on HPMS submittals. The State DOT submits data for 3,900 segments annually of which 3,600 records are deemed complete based on FHWA review. The completeness measure then is (3600/3900) * 100 = 92%

The completeness measures are summarized in Table C.4.

Table C.4. Summary of Completeness Measures
Categories Completeness Measure
Original Source Data
Continuous Count
86.4% for ADT generation
Original Source Data
Short-Term Count
100%
Processed Data
Archived Data
92% based on HPMS submittals (hypothetical)

Validity

Original Source Data – Short-Term Counts

The DOT uses a mainframe based program to analyze data from continuous count stations. Data is downloaded daily and processed through the software. Questionable data records are flagged for review by a manual operator. At this point, the manual operator makes a decision on whether to accept the data or to delete it.

According data provided for this example, 190 of the 220 ATRs (86.4%) on average have complete data as presented above. Out of the 190 ATRs, only 180, on average, record valid data. Valid data is defined as data that is verified using high/low range checks and historical ADT trends data. Thus the validity measure, as calculated as

Expected records – 24 (hours)* 14 (days) * 220
Complete records for validity criteria – 24 (hours)*14 (days) * 190
Valid records – 24 (hours) * 14 (days) * 180
Validity = (180/190) * 100 = 94.7%

Original Source Data - Short-Term Counts

The validity of short count data is intrinsically related to completeness. Since it is possible for the DOT to reset the counter, the key indicator remains how many of these counts are usable for ADT data and how many have to be reset. Historical data indicates that approximately 5 percent of the 4,200 (i.e., 210 counters) annual short term counts need to be reset. Resets are based upon reviews of hourly data, high/low ADT and historical AADT values.

Thus, the validity of short-term counts is (100 – 5) = 95%. Note that invalid counts are reset ensuring that all the short-counts are valid.

Archived Data

Validity as perceived by archived data users is dependent on the application. Frequently, the users assume the data provided to them as valid. The data reported by the DOT is considered "official". Several users might have their own validity criteria which are applied to the data from the DOT. For example, HPMS administrators at FHWA might check the validity of the dataset submitted by ODOT for sample size adequacy, inventory errors, pavement information errors etc.

As a hypothetical example, FHWA could define validity of DOT data based on HPMS submittals. The state DOT submits data for 3,900 segments annually and 3,600 records are complete. Assume that only 3,200 records pass the validity checks. Thus the validity measure can be calculated as (3200/3600) * 100 = 88%

Table C.5. Summarizes the Validity Measures
Categories Validity Measure
Original Source Data
Continuous Count
94.7% for ADT generation
Original Source Data
Short-Term Counts
95.0%
Processed Data
Archived Data
88% based on HPMS submittals (hypothetical)

Timeliness

Original Source Data – Continuous Counts

Due to the archival nature of traffic monitoring, timeliness is not as critical measure as in traffic operations and management. Data from about 180 sites with telemetry are polled every weekday by the DOT with the remaining 40 sites manually polled monthly. All data is received within time period required to process and submit to FHWA before the 20th of the month deadline.

Timeliness of continuous data collection is not as critical as for ITS applications like traveler information. However, the time elapsed between the download of the data to the review and approval of the data is important. Long review times can result in delayed identification of detector problems and consequently loss of data. According to information provided by the state DOT, the average review time (time from download to approval) for ATR data is about a week.

Original Source Data – Short-Term Counts

Data is collected by district crews. It is sent to central office for processing every 1-2 weeks. Data is typically sent by email from the count crews, however some of the data is relayed through central office personnel via Take Away Memory (TAM) cards.

Timeliness of data collection is important to short-term counts in a similar manner to continuous counts. Once the data is collected, the time to review, approve and upload to the database is critical. In this case, the average time for data to be sent to the central office from the time it was collected is 2 weeks. The time to review and approve data is another 2 weeks.

Totally, 4 weeks are required for short-count data to be collected, processed, reviewed and uploaded.

Archived Data

The timeliness measure for data users indicates the availability of data when they require it. For example, FHWA drives the 20th of every month deadline for submittal of permanent count data for the DOT. As a hypothetical example, let us assume the DOT is able to provide data to FHWA on the 20th of every month for 8 months in a year and by the 30th for the four remaining months. The timeliness measure for FHWA can be based on the number of timely submittals from the DOT (say 8 months of the 12 = 75%) and an average delay of 10 days.

% timely data equals 8 months-ontime submittal over 12 months required equals 75%

Coverage

Coverage for the DOT traffic monitoring program is driven by federal requirements and guidelines. The DOT bases the program on the Traffic Monitoring Guide 2001 and HPMS requirements. Table C.6 below compares the coverage requires with actual coverage in the state for continuous counts and Table C.7 shows the comparisons for short term counts.

Table C.6. Coverage for Continuous Counts
Recommended by TMG and HPMS Actual Coverage in State
TMG – Traffic Volume – minimum of 5 Factor Groups with 5 to 8 ATRs per group.

Vehicle Classification – Determine appropriate number of factor groups and assign at 6 continuous counters within each group.
TMG – 8 factor groupings based on functional classification. More than 6 permanent count stations per factor group. Higher functionally classified groupings have the greatest number of sites.
HPMS – At least one continuous counter on each major PAS/NHS highway route. HPMS – there is at least one continuous count station on each "major" PAS/NHS route.

Table C.7. Coverage for Short-Counts
Recommended by TMG and HPMS Actual Coverage in State
TMG – Traffic Volume – roadway segment-specific data traffic count information on a cyclical basis.

Vehicle Classification – Count all arterial and major collector roadways.

HPMS – All HPMS universe, standard sample, and donut area sample sections.
TMG – Over 12,000 counts spotted statewide. One vehicle classification count spotted between each interchange of the entire Interstate system. At least one count spotted between all State and US routes statewide.

HPMS – Standard samples are collected by state DOT using 24-hour vehicle classification counts.

Based on the two tables, it can be concluded that the DOT has adequate coverage with respect to the coverage requirements of the Traffic Monitoring Guide and HPMS. Thus, coverage is quantified as 100%.

Accessibility

Continuous data is combined and made available to public in an annual Traffic Survey Report. Internally, permanent data made available in a GIS/Web application on the intranet. Short term count hourly and AADTs made available to the public via the Traffic Survey Report and PKZIP ASCII files by individual count. AADT data is compiled and used in DOT's Traffic Survey report.

Qualitative information on the times required for data consumers to perform specified tasks is not available.

Interpretation of Data Quality Statistics

The data quality statistics for the Ohio case study are summarized in Table C.8.

Table C.8. Data Quality Summary
Quality Measure Original Source Data
Continuous Count
Original Source Data
Short-Term Counts
Archived Data
AccuracyATR
Volume
  • MAPE - 0.89%
  • RMSE - 4 vehicles
WIM
Vehicle Length
  • MAPE - 7.26%,
  • RMSE - 1.89 feet
Gross Vehicle Weight
  • MAPE - 4.17%
  • RMSE - 1,760 pounds
Volume and classification
  • MAPE -0.11%,
  • RMSE - 2 vehicles
Estimates of AADTs are derived from ATR adjustment factors and ADT data from short-term counts. Accuracy of AADT estimates depends on the underlying accuracy of ATRs and short-term counts.
Completeness86.3%100% (Only complete counts are accepted)92% based on HPMS submittals
Validity94.7%95%88.8% based on HPMS submittals
TimelinessAverage review time for ATR data is one week.Average time from data collection to final approval is 4 weeks.20th of every month deadline met 75% of the time
Average delay when deadline was not met - 10 days.
Coverage100% of TMG and HPMS requirements
AccessibilityData is readily accessible

Appendix B | Table of Contents



FHWA Home | Feedback
FHWA