ITS - Intelligent Transportation Systems Report ITS Home Page

3. Evaluation Results

3.1 Preface

The evaluation schedule was delayed from what was originally planned. This delay was not at all related to the development progress of the ADMS Virginia system, which was slightly ahead of its deployment schedule. Rather, the delay was prompted by two factors: 1) problems with the data quality and availability from the HRSTC and 2) lack of use on the part of stakeholders (primarily due to the data quality issue). As a result, the decision was made to delay the evaluation, with the hope that additional time would rectify these problems. As it turns out, however, these problems were to plague the evaluation even with the delay, as discussed below.

3.2 Interviews of System Users

3.2.1 Initial Interviews, July 2004

Interviews were held with personnel in the Hampton Roads area in July 2004. The purpose was to uncover basic usage facts about the system and to help structure the remaining evaluation.

3.2.1.1 HRPDC

Two staff planners were interviewed. One was involved in the rapid prototyping of the system and was highly familiar with the functions and interface; he had accessed the system about 25 times prior to the interview. The second planner had accessed the system only a few times. The three primary uses of the system were to 1) obtain volume data for use in a variety of planning functions, especially near the tunnels, 2) tabulate incident characteristics for use with the Congestion Management System, and 3) determine speeds on a limited number of segments. However, all the uses were done in "test mode" - widespread use of the data in planning applications had not been made. (But it was hoped that in the near future, practical use of the data could be made.) The traffic data from the sensors is more likely to be bad (missing) than good. The main problem was the lack of data altogether due to problems with the field data. This requires the user to do a lot of checking on data availability before accessing the data. The coverage (only a limited number of freeway miles) is still not complete enough to be useful on a regional basis. However, the planners liked the concept of having the data directly available to them. They commented that this system is much better than the previous methods of accessing data; in the past HRPDC had to request data from VDOT and wait a couple of weeks to receive it.

The planners were very pleased with the functionality of the system. ("We can obtain data with three clicks.") Obtaining the same data used to take at least a day, and probably more due to several iterations of data requests. The planner with experience with the system commented that the interface was not intuitive for the first-time or casual user. Menu items are categorized by organizational data source (e.g., STC) not by function (e.g., freeway traffic volumes). The uninformed user needs to search around to find the data source. He suggests a glossary of terms be readily available since he had to ask several times what a table or data set name meant. However, once just a small amount of experience is gained, the system is easy to navigate. This may suggest that a short but formal training course be provided to potential users.

The traffic volume information from ADMS Virginia could be used now on a limited basis, but this can't be done until the data is reliable and the coverage is expanded to region wide. Until then, it is easier to use VDOT's periodic (3 year) short counts. HRPDC's travel demand forecasting model is a daily model (it predicts total daily traffic rather than peak hour or peak period traffic), so hourly volumes and speeds are not yet as useful. However, when reliable data are available, it will make the migration to a peak hour model easier. (Peak hour models are considered a more sophisticated form of travel forecasting and HRPDC intends to convert their model to it at some point in the future.)

HRPDC currently collects travel time data on major highways using the "floating car method." Assuming the data was reliable and region wide, some travel time runs performed by HRPDC could be eliminated. Also, VDOT would no longer need to conduct periodic short counts on freeways. In the short-term, redundant data collection is needed to provide a consistent data source region wide and provide validation for the ADMS data, because the planners were suspicious of the data quality. When the ADMS data could be used, the savings would probably be used to collect data on additional roadways.

Additional data could be used in developing models more sensitive to varying incident and weather conditions for conducting operations modeling (e.g., HRPDC attempted to conduct a previous analysis on the air quality benefits of the incident management system. The analysis was not successful due to the lack of data representing incident conditions). They have considered developing an evacuation model, which will be easier with reliable ADMS data.

Until data is reliably available on a region wide basis, little opportunity exists to use data for regional reporting, such as in the Congestion Management System (CMS). Data may be used for special studies in the meantime. ADMS is also used to fulfill some internal and external requests for traffic data. When reliable, ADMS speed data may be used in congestion reporting and the CMS. Until then, speed data is based on HCM method using periodic traffic volumes.

Advantages of the ADMS to support planning functions include decreased time required to fulfill requests for data and data for special studies. A constant stream of data would provide a more accurate picture of what's currently happening on the system, especially in assessing day-to-day variability. When data quality and coverage improves, the planners intend to access the ADMS several times per month to obtain data for planning applications.

The planners offered their opinion about the value of the ADMS. Planning efforts based on ADMS data would provide for implementation of better solutions and more cost-effective planning activities, but would probably bring about only marginal improvements for the traveling public. However, use of the ADMS's data to evaluate the effectiveness of the TMC and to alter operations strategies should lead to tangible benefits, the planners felt. Evaluation of other special projects would also be enabled.

The planners would like to have data on signalized highways, even it is only continuous volumes. Much of their planning effort is focused on these types of highways. Also, any data that could be used to supply origin/destination patterns would be of tremendous help. This is a key piece of information for the travel demand forecasting model and the only way to get it now is conduct special and expensive household travel surveys.

3.2.1.2 HRSTC

The Lead Project Engineer, Maintenance Supervisor, and IT Department Manager were interviewed. All three individuals were familiar with the system and had been involved in testing and commenting on prototype versions of the system. The system had been used in an exploratory way but had as yet not been used in day-to-day operations or to effect operating decisions. This use included: reviewing speed and volume data as compared with ground counts and reviewing incident data when developing incident response plans.

The same problem with the data quality and availability noted by the planners was also noted by the operators. The lack of data obtained through the sensor system currently inhibits the usefulness of the data. Because the data is unreliable, a second source of data must often be obtained for confirmation. Incident data was useful, and the operators were able to see patterns at selected locations.

Many of the field detection systems throughout the region are unreliable. Loop detectors, which have been used extensively throughout the region, provide the most accurate data when they are working, but are often inadvertently damaged. The equipment inventory system ties all equipment location to a filed cabinet, not necessarily its actual physical location, often resulting in loop detectors being damaged during repair work. Therefore, the Department has recently replaced loop detectors with number of less intrusive technologies including side looking radar and acoustic sensors, so the reliability of the data from existing sensor locations should increase, simultaneously with the expansion of the overall coverage of the system. As the reliability of the detection system increases, the usefulness of the data should increase.

It was noted that VDOT has historically been in the business of collecting the freeway performance data, but hasn't been a significant user of the data. Other entities (e.g., HRPDC, STL, other researchers) have more often been the "customers" of the data and would have better perspective of the data usefulness. The unreliable nature of the sensor data does limit the usefulness of the data. In short, performance measurement was not seen (at this time) as a priority with the HRSTC, although it was recognized that performance measurement was both a useful activity for HRSTC and may eventually be required by VDOT management as part of a department-wide performance measurement effort.2

The operators commented positively on the operation of the ADMS as an information system; system is well designed and intuitive to use. The system itself works well, but the poor data quality limits its usefulness.

Incident management is probably the most important operations strategy performed by HRSTC. The procedure for incident response is as follows. Once incidents are detected and verified, the response is largely coordinated by the controllers in the TMC. Several Incident Response Plans have been developed that specify appropriated responses for various types and locations of incidents. The decision to implement one of these response plans or to implement a customized response is left to the discretion of the controllers and shift supervisor. This decision is based on their knowledge of local conditions, available information on the incident, and experience. The operators are trained on the Standard Operating Procedures (SOP) Manual and refer to it for many decisions. The primary sources of information on the actual incident conditions include CCTV surveillance and from direct communication with responders on the ground. A more formal standard operating procedure defines all the decisions regarding operation of the reversible lanes. Regardless of the response, a log is maintained for all implemented strategies during the full duration of time that the incident is actively being managed by the STC. It is hoped that the ADMS can help in the development of the Incident Response Plans (Figure 3). It is not expected that the ADMS would typically be accessed in real-time by controllers during an actual incident. The rapid speed at which decisions are made during these conditions does not allow the controllers the time to access an additional data source to analyze what conditions were during similar incidents in the past. Decisions need to be made quickly relying on the experience of the controllers. It is possible that the shift supervisor might have the opportunity to perform some of this type of real-time analysis on occasion, but he currently has no plans to implement ADMS use as a part of typical incident response procedures. Any sort of decision support in managing incidents in real-time would have to have immediate turnaround time and be extremely easy to access through the current TMC software.

Flow chart outlines the role of ADMS in the incident response process. It shows that once incidents are detected and verified, the response is coordinated by the controllers in the TMC, who log the incident into the IRM. At this point, incident information is made available through ADMS. Operators can initiate a query in ADMS, and ADMS will generate a report with historic incident details. Based on this information, the operators review the IRM response, approve it, and initiate it.
Figure 3. Potential Role of ADMS in Incident Management Process at HRSTC with Incident Response Module

With regard to evacuation planning, fairly comprehensive evacuation plans/procedures already exist. Once implemented, there aren't that many variables that the TMC has control over. It is possible that the ADMS data might be used in these situations, but it is not clear exactly how they would be used real-time. The data could be used as historical data for event debriefings and evacuation plan refinement, however. (What happened, what worked, what failed, etc.)

In the future, using the ADMS data and/or the traffic forecasting function could be useful to the HRSTC. Communicating expected delay information to travelers is seen as function that should eventually be performed.3 The ADMS has a much greater potential to be used as an evaluation/planning tool than to be used in the day-to-day functioning of the TMC (STC). Potential applications include use as an "after-the-fact" tool for evaluating the effectiveness of implemented responses. The tool could also be used in the evaluation and modification of the Incident Response Plans or in the development of new operating procedures. More data would be available to supervisors in gauging the effectiveness of the implemented response strategies. Many of the current Incident Response Plans are not greatly sensitive to the level of congestion. The ability to examine incident response in different incident conditions could provide the ability to modify the plans to increase their sensitivity to congestion and other conditions (e.g., weather).

Operators felt there would be little noticeable impact for travelers; however, if the system provided the ability to evaluate and improve the incident response plans, there could be some travel time savings. Also, if the data could be used to provide estimates of expected delay, it is possible that this information could be passed onto travelers through the DMS or through a website.

The ADMS could also be used in training exercises (e.g., providing the ability to examine similar incidents with different responses and compare the resulting impacts). VDOT has discussed with FHWA the use of archived data to estimate and simulate delay and queue length (the DynaMIT simulation model). This led to observation by one of the operators that HRPDC has more uses for the data than the TMC (STC) staff.

Making the data available on-line may reduce the amount of staff time required to fulfill data requests from other agencies/organizations. The data is also available to all the ISP's which currently have access to STC cameras and incident information. Currently there are no known users who are performing additional analysis on the data and making any system performance data available to the public, but it remains a possibility.

3.2.1.3 Summary

As mentioned in the Preface, the first round of interviews revealed that the ADMS had not been used much except in an exploratory way. This was primarily due to issues concerning data quality, but also was not clear exactly how or if the ADMS would be integrated into the HRSTC operations. On the other hand, it did appear that HRPDC was planning to use the ADMS to supply data for a variety of planning needs. At this point, it was decided to give the system more time to work out data quality problems and to give personnel a chance to access the system more fully. Also, the transporting of the ADMS to the NoVA District of VDOT offered additional opportunities for the evaluation.

3.2.2 Second Interviews, February 2005

3.2.2.1 NoVA District

Attending this meeting were operations personnel; planning personnel were invited but did not attend. From the NoVA perspective, the evaluation of the system is occurring too soon because personnel had not had adequate time to use the system yet.

A function that NoVA personnel thought would be useful was the ability to monitor HOV lane utilization. At that point, the ADMS did not have a specific report to present data collected from HOV lanes, but ad hoc reporting capabilities did exist. (The HOV monitoring function was being developed at this time and is now operational in ADMS Virginia for NoVA.)

The main deterrent to using the ADMS was the speed of the login and the queries - access time was very slow, measured in many minutes.4 Sometimes, it was not possible to logon successfully. Even when queries are small, users still need to wait a long time for the results. First time users mentioned that the tool looked cumbersome, but with a little experience, became easy to use and navigate. A short amount of training - perhaps a videotape demo - was suggested as a way to overcome the initial learning curve.

One type of analysis that the ADMS currently does not do is a "timeline" analysis of incidents. That is, how long the various components of total incident duration are: detection, verification, response, on-scene time, etc. Such an analysis, however, requires that the data be accurately collected by operations personnel before the ADMS can store and summarize it. NoVA TMC is not collecting all the data that would be needed for rebuilding incidents, however, they are making improvements towards getting to that point. They hope the ADMS data structure and functions could be modified in the future to accommodate this.

The issue was raised about the effectiveness of the ADMS to support real-time operations. The time involved in retrieving and accessing archived data was seen as a hindrance to real-time use. Operational planning, especially for HOV, was seen as maybe the best application set for the ADMS. But the data has to be easy to access and use. Maps of interstates and arterials with seasonal trends and shifts would be valuable for short-range planning and evaluations. There may be value for offline analysis, especially for transportation planners and operators. However, the value of the data as it is used for real time operations was seen as questionable.

One of the strong points of archived operations data is the fact that since it is continuous, fluctuations in demand and congestion can be seen. However, other than providing data for offline analysis (which the planners and operators would have to develop themselves), the ADMS functions are not that useful. ("There needs to be a better way of looking at the data directly.") Having the data in GIS format or in some customized graphical views would be helpful, and in fact, several ADMS services are available as graphical output. (The lack of usage ny NoVA personnel at the time of the evaluation interview may be influencing this observation.)

These suggested improvements were relatively new in the minds of NoVA personnel they had not been discussed during the Build 4 user requirements process. This was largely because NoVA personnel did not have a good idea of what was possible until they started to use the system.

The heaviest users of the ADMS were foreseen to be planners and engineers involved in planning activities. VDOT planners were asked to experiment with the ADMS, but most of them have not provided any feedback. People in operations on the TMC floor would be light users of the system for the immediate future - their workload is already heavy. Training and internal marketing of the tool could be improved. Other potential uses that were identified for the ADMS would be performance monitoring reports, especially for incident management. The data could also be used in the data-intensive traffic simulation models. Reformatting the data into the specific input structure for simulation and travel demand forecasting models would be another future enhancement that would avoid post-processing the data by planners. The ADMS would be helpful to determine the timing of lane closures for work zones.

It was noted that VDOT is launching a department-wide performance measurement initiative. As these requirements are pushed down to operations, the value of the ADMS becomes apparent. Having continuous data on system conditions from the ADMS will be very useful. It will be easy to meet internal reporting requirements and there is the potential to slice the data in several ways. One way would be to expand VDOT's Dashboard (which currently only reports construction progress) to include congestion statistics. Again, either the ADMS would have to be modified to provide this function directly, or an offline application fed with data from the ADMS would have to be developed.

The main problem with the ADMS for NoVA was reiterated - the accessibility (logon ability) of the system and the speed of downloads and queries. Solutions suggested by the interviewees were to increase bandwidth or download the data overnight, although ADMS Virginia developers have been working to improve the speed.

3.2.2.2 HRPDC and HRSTC

These two organizations are reported together for ease of discussion. In summary, the evaluators found that little had changed from the time the initial interviews were done. HRSTC had made no operational use of the ADMS. HRPDC still expressed interest in using the ADMS to supply data for planning applications, but with one notable exception, had still only used the system in an exploratory way. The notable exception is that HRPDC is currently using the ADMS to help produce their reports on the incident response program. The details of the discussions with these two agencies follow.

Performance Measurement and Decision-Making

Recent changes within VDOT that reflect the increased importance of improving operational performance of the roadway system have not yet been reflected in active requests for and use of performance statistics for Hampton Roads. Consequently, relatively little use of the ADMS has taken place in these organizations as of the project interviews.

As with many roadway agencies, resources at both HRPDC and HRSTC are already stretched very thin. Because there has been no formal reporting requirement for performance information on the Hampton Roads freeway network, and until recently, no clear audience for such reports, few resources have been allocated to the use of the ADMS. For example, reporting on facility performance was not a contract task for the contractor staffing the HRSTC. The lack of facility performance reporting is likely to change with the new organizational structure adopted by VDOT and the corresponding increase in visibility of operations within the Department. The new organizational structure, combined with VDOT's increased reporting of system performance statistics (i.e., trends in congestion level and supporting attributes like incident characteristics) is expected to increase use of the ADMS in the near future. VDOT's Chief of Systems Operations will become the "primary client" within the Department for this information, thus increasing interest in, and likely resources for, performance measures that can be most effectively produced by the ADMS. HRSTC is a likely candidate for producing these reports, but whether they or some other group (such as HRPDC) are tasked with these efforts is still being determined by VDOT.

The VDOT organizational changes may also result in minor modifications to the ADMS software. While VDOT's new organizational structure will result in the publication of operational performance measures, no specific measures have been determined. Once these measures are selected, the ADMS may need to be changed to quickly and easily produce those statistics. There has been talk of VDOT requesting a quarterly performance report from each District, but the content of that report is currently unknown and the availability of additional data collection resources to support that reporting effort is also unknown

Organizational changes similar to those being implemented by VDOT that increase the profile of operations have not yet occurred within the transportation agencies in the region. Each of the groups we interviewed noted that in Virginia, funding was passed directly to the seven independent municipalities in the region, and those municipalities select their own projects to receive that funding. The result is that there is relatively little incentive (or funding) to address regional problems, unless the local impacts from poorly performing regional facilities result in problems for specific municipalities. The result is that there is relatively little call at the regional level for a more numerically based, data intensive, regional view of facility performance. Such a reporting system simply doesn't match the political decision-making style of the region. (Perhaps this attitude is best explained by paraphrasing one interviewee's view of the regional project selection process, "If we don't have money for a construction project, we won't study it, and if we have money for the project, we don't need to study it.")

Effect of Policy Decisions

As with many states, there appears to have been relatively little historical interest within Virginia to measure the effectiveness of many transportation policy decisions, or the specific implementations of systems intended to meet policy objectives. To the credit of both VDOT and HRPDC, efforts are now underway to provide more information describing on the effects of many statewide and regional policies on transportation systems performance.

An example of this transition in attitude towards performance measures involves the Hampton Roads incident response patrols. At one point, VDOT cut back funding for incident response patrols as a result of the loss of general transportation funding in the state. The money was restored as a result of an outcry from the public, not an analysis that said roadway performance was suffering from the reduction in response vehicles. However, HRPDC is now producing ongoing reports on the performance of the VDOT incident response program so that the performance of this program can now be quantified.

A lesson learned from our interviews is that agencies that do not actively seek to use or provide information on facility performance are less likely to need, use or benefit from an ADMS. However, the national trend is towards greater levels of accountability in government. We expect this to result in increased need for the types of performance reporting that can be most effectively provided by an ADMS.

Places Where the ADMS Would Be Beneficial

There are several periodic studies done by VDOT/HRPDC that would benefit strongly from the ADMS, if the ADMS contained valid data, and if the staff accessed it. The most obvious of these efforts would be to report on the use and performance of the Hampton Roads HOV system and on the congestion experienced at the two tunnels.

Currently VDOT performs additional, special purpose, data collection (volumes and travel times) on the HOV system and its parallel general purpose lanes in order to examine the performance of the HOV system. If the data in the ADMS were reliable, these tasks could easily be completed without additional data collection (other than vehicle occupancy counting.)

The ADMS could be a key tool in studying the potential for use of the HOV lanes as a HOT facility. It would also be essential for measuring performance changes on the facility if VDOT, for example, were to implement HOT lanes in Hampton Roads at some point in the future. Analysis of the performance changes would be key to effectively managing the operation of the HOT lanes, especially if the HOT lanes involve some aspect of congestion pricing in order to maintain free flow operations during peak periods. The TMC (STC) operations staff we spoke to gave little thought to performing these analyses, and the current level of data quality most likely prevents their currently being done with the ADMS. The HRPDC staff had considered the use of the ADMS to examine general tolling options for the region (mostly in conjunction with a proposal to build a third tunnel), but lack the resources to perform such a study (not to mention the problems with data quality.)

The second obvious place where the ADMS would be of significant use is for reporting and managing the delays found at the two tunnel crossings. Such a report would be of significant assistance in obtaining legislative assistance for changing operational policy in the region. Currently, significant delays are caused by the need to shut down both directions of traffic approaching either tunnel whenever an over-height vehicle attempts to use that tunnel. The dual stoppage of traffic creates considerable congestion, but the trucks causing the congestion are not fined or punished. The region would like the state legislature to adopt a fine/fee/deterrent in order to reduce the number of these occurrences. A simple performance report listing the number of times over-height trucks result in temporary roadway closures, and the delays caused by those closures would most likely provide the ammunition needed to pass this change in the Virginia legal code. Analysis of delays caused by over-height truck movements combined with historical traffic volume patterns would also allow TMC personnel to more effectively select appropriate times to actually close the roadway as well as improve the estimates of delays given to motorists when closures are taking place.

Even though tunnel delays are a major public information task of the TMC, no analysis of their size, frequency, or cause has been performed. Again, the lack of accurate sensor coverage seems to prohibit, at this time, this type of routine analysis and reporting. (Note that this is partly a function of the current detector placement, and partly a result of the poor performance of those sensors which do exist.) VMS messages describing expected tunnel delays are currently based on an operator's empirical knowledge of expected delays relative to the length of traffic queues as measured by visual inspection off of the CCTV system. VMS messages are NOT stored, so it is not even possible to get a historical estimate of tunnel delays based on posted VMS messages. Storing VMS messages in the ADMS might allow such a performance report to be produced.

Another place where the ADMS could be used, but where to date no request has been made to use it, is for reviewing traffic management practices that maintain traffic flow into and out of the beaches on weekends. The Virginia Beach area is a key tourist destination, and the reversible roadway operation is used to help bring tourists into and out of the beach area on weekends. No analysis has been performed to determine if the timing of changes in the reversible roadway's direction of operation could be improved, or if historical roadway performance information could be used to provide advanced traveler information to further improve roadway performance.5

The ADMS might also be used to help develop the HRPDC Congestion Monitoring report done every three years. Currently that report (based on volume counts and a single floating car run done on each major corridor) is primarily compiled using special data collection efforts. With some forethought and a modest extension of the automated data collection effort, the ADMS might be able to provide much more detailed and accurate congestion estimates for this report, as well as provide for a more routine update of those measurements.

It should be noted that Hampton Roads has a bit of a Catch-22 dilemma when it comes to increased data collection. The region has limited funds for detector expansion, operation, and maintenance, because it can not show the benefit of the use of the added surveillance expenditures. Unfortunately, it can not show those benefits without the additional data that would be provided by the extra detection. HRPDC is currently using the ADMS to help produce their reports on the incident response program. The current HRPDC report basically lists the number of incidents responded to by VDOT, but does not quantify the congestion associated with those incidents or the benefits from having the incident response program in place.

HRPDC also has begun to produce an annual report on the "state of the region's roads." This report would be a logical place to publish ADMS based roadway performance statistics. The only real issue is whether the ADMS data quality problems will be corrected enough to allow the use of the data in this fashion, and whether the HRPDC can afford to produce the report with the greater level of detail provided by the ADMS.

Potential ADMS Improvements Suggested by HRPDC and HRSTC

For those staff that have used the ADMS to date, data quality and availability issues dominated their view of the usefulness of the system. When pressed for additional information, it became clear that there are two basic groups of users, the casual user, and the routine user.

The casual (or infrequent) user generally found the system to be somewhat slow and intimidating. HRPDC staff that fell within this category indicated that until you were familiar with the system, even getting simple statistics such as AADT values from the ADMS was difficult. A conclusion that may be drawn from this is that making access to finished data products easier is key to encouraging these individuals to use the system more often. This means that more "canned" reports (tailored to specific reporting needs) probably would need to be created, and the user interface may need to be changed slightly in order to make it obvious how to obtain those outputs.

On the other hand, routine users of the system did not find the system confusing. Training and familiarity with the system resulted in the user's ability to easily obtain the data they were interested in. These users were more interested in obtaining fairly raw statistics and data from the database, but they too expressed interest in speeding and simplifying the interface process for getting key roadway statistics. The operators, in particular, could foresee using the ADMS much more if they had more "easy to obtain" reports. However until these reports became both easier to obtain and more relevant to how their job performance was judged it is unlikely that the operators will access the ADMS routinely.

HRPDC staff also noted that the Phase 2 detectors are now coming on-line, but that the ADMS referencing system has not been kept up to date with those data updates. It is therefore not possible for the "average user" to determine where these new detectors are located. The ADMS may need some additional configuration management support in order to address this problem in the future.

The HRPDC staff was also interested in having more congestion performance related workshops taught in Hampton Roads. And they were also interested in having more training done for staff at the local level.

Data Quality, Validation, and Availability

As noted elsewhere, the lack of data quality is a significant roadblock towards more active use of the ADMS. Most of the Phase 1 detectors do not currently work, and for much of the time when they did work, data quality was highly suspect. The operators interviewed indicated that the data currently being collected has never been validated. None of the groups we interviewed felt it was really their job to perform that validation.

The upcoming performance-based TMC contract may bring some of those duties to the TMC, but there still appears to be disconnects between different divisions within VDOT on who is responsible for (and must fund) the maintenance of surveillance equipment. For example, the Phase 3 detector contract appears to be purchasing acoustic detectors that have traditionally had problems performing accurately at speeds below 30 mph. Consequently, TMC staff is concerned about the performance of the detectors purchased under the Phase 3 expansion, but appears to have no ability to do much about this concern. Just as importantly, there is little budget available for repairing or replacing failed detectors after it becomes clear that a detector has failed.

The TMC currently has little detection on many of the roads that ought to be included in a "regional freeway performance report." Cost-effective extension of data collection to those roadways appears to require both additional sensors, and a change in the nature of the sensors used. (For example, detection might be primarily oriented towards collecting travel time data on key roadway segments supported by limited traffic volume data, rather looking to expand the existing point detection system.) Such a change in sensor deployment would require changes to the ADMS.

Other Issues

HRPDC indicated that it would be possible for VDOT to request funding for studies that used the ADMS through the regular HRPDC process. There was considerable speculation about whether CMAQ funds could be used to support this type of analysis. However, funding from this source is unlikely to be available for detector maintenance or repair.

The HRPDC has an ITS Committee that is currently charged with determining where Operations Planning will take place. (Should it be a function of the agency that operates the roadway, or should it be a regionally controlled system.) They are getting good support for this function from VDOT, but the decisions have not been made at this time.

Summary Lesson Learned

Our discussions in Hampton Roads highlighted a key lesson learned. Use of an ADMS and the quality of data included in that ADMS are directly correlated to how actively agencies are using (or are interested in using) performance measures that describe facility operations for decision making.

For an ADMS to be useful, the performance measures such a system produces must be actively used by an agency. If no one actively uses performance measures to make decisions, the quality of the data in the ADMS tends to degrade. The result is that when the ADMS is accessed, data quality issues tend to limit the usefulness of the archive. On the other hand, if the data are routinely used, data quality issues are identified (and fixed) when they occur, because decision makers are relying on, and value, those data.

Historically, facility performance measures have not been actively used in Hampton Roads. Neither HRPDC nor HRSTC had seen reporting on roadway operational performance as "being their job." The result is that data quality in the ADMS has suffered, further degrading the use of the ADMS for other purposes. Both VDOT and HRPDC are in the process of trying to increase their use of facility performance measures. For example, it has been suggested by some parties that VDOT use such measures as one measure of the job performance of the next contractor chosen to operate the HRSTC. If the reliability of roadway operation is used as a factor for judging (and paying) the TMC contractor, then both the contractor and the agency hiring the contractor have a direct interest in accurately measuring and understanding the performance of those facilities. Such an interest will cause the quality of the data to improve, as well as dramatically increase the use of the ADMS as a tool for improving roadway performance.

3.2.3 Final Interviews, May 2005

3.2.3.1 VDOT, Smart Travel Laboratory (STL), University of Virginia

The STL was the ADMS Virginia project coordinator. They performed the functional design of the system and supplied many of the algorithms for data processing such as quality control procedures, imputation, and the travel forecasting procedure.

A formal user requirement process was followed whereby STL and their software contractor (Open Roads) interviewed potential ADMS users. STL felt that the process they followed led to effective system design and would not do anything differently. Initially when they went to the stakeholders with user requirements and design specifications, they received no feedback. It was felt that potential users did not have enough detail to react to, so mockups of the interim builds were provided at subsequent sessions. Giving those examples to stakeholders was essential to getting feedback from them.

The HRSTC is contractor-operated with VDOT supervision. Initially, this was thought to be a potential problem, but this has never become an issue. The HRSTC team there is very well integrated, primarily because URS has had the contract for a long time and work well with VDOT personnel.

STL personnel reflected on the primary users of the ADMS - it is not designed primarily for the TMC operator for use in real-time; the tool is more for planning applications, both operations planning and traditional planning applications. It was observed that in order to be effective any software application for "on the floor" use at a TMC must be integrated with existing TMC software. That is, it must be part of the same console that operators use - anything else is simply not "ergonomic" or time-efficient to use. Also, query interfaces have to be relatively automatic "point and click" - operators are simply too busy to structure custom queries.

User feedback has been positive. They like the potential that the ADMS affords and are asking for more data, more coverage, and better quality data. The speed of the queries (noted in the February interviews) had been relayed as a problem. This is being addressed in a software update, Build 4.1. The slowness in the speed of queries was related to several factors. Some queries were not structured properly in SQL. Also, the shp files from GIS added to the time needed to run the queries. These map layers in GIS will be available for future applications. There were no complaints about the interface. They were thrilled about getting to the data so easily.

There is a growing interest in archiving data throughout the state. Virginia Beach wants to archive their signal data, but this is only in the early discussion stages. VDOT is talking about a statewide incident log. They would be able to add these sources to the ADMS, should funding be made available. It was observed that ADMS Virginia has demonstrated the value of archived data as a resource for many applications. It has made transportation personnel aware of the vastness of the data available, and it also made them aware that quality matters. There has been more attention on data now than ever before in the history of VDOT - the new performance measure initiative is proof of that. It is likely that the first aspect of performance that will be considered is related to incidents - much data already exist on incidents and there is not same issue of quality and coverage as for detector data.

Build 4.1 also incorporates another feature reported by the NoVA personnel the February interviews. It will allow analysis of congestion patterns on the I-95 and I 395 HOV lanes. VDOT needs to quantify the demand usage on these reversible lanes. Such an application is an example of how the ADMS may be used to support operational planning.

Many data issues had to be addressed by ADMS Virginia that the TMCs would have had to address eventually anyway. Foremost of these was the coordinates of the field detectors, and the naming conventions for the detectors (configuration management). ADMS developers verified locations of the detectors. This rectification will have benefit for the TMCs as well as for ADMS Virginia users. The base maps are from the VDOT GIS division, most of which are in ArcGIS.

Data security is also an issue - in the case of Hampton Roads, the data is put behind a firewall, and the ADMS accesses the data there. NoVA transmits the data via ftp the data to STL every minute, a mutual decision between the NOVASTC and the ADMS team based on aggregation parameters.

The addition of RTMS to the detector inventory should not pose a problem. The RTMS specification says that it has to look like a loop. The structure of the data from RTMS and loop detectors look the same. The inventory file describes what type of detector exists, but the data structure is indistinguishable.

From the outset of the project, portability from the initial deployment (Hampton Roads) was assumed. To enhance portability, the data schema was developed to be consistent with national and VDOT standards (primarily the Traffic Management Data Dictionary). When porting to NoVA, data translation code had to be written to put data into the standard schema, but it was relatively easy to do so. Having this commonality in the underlying data structure (schema) helps to keep performance measures consistent.

The issue of data quality has been the dominant issue in the usage of the ADMS. The QC procedures and quality reports are two ways the ADMS can exert influence on the quality issue, but ultimately it ties back to the status of field detectors: proper installation, calibration, and maintenance of the detectors are the only actions that will improve data quality. These are outside of the control of the ADMS - this function resides at the TMCs. Feedback to the TMCs on data quality status has been initiated in Hampton Roads, but correcting the problems with field equipment and communications that lead to data quality problems has not yet been undertaken (although it is viewed as important).

VDOT has set aside $300,000 annually to maintain and upgrade ADMS Virginia. At STL, there is one full-time database administrator, one part-time database administrator, one part-time documentation person, and one full-time software person. If the ADMS maintenance takes up 1/3 of the time, then maintaining the ADMS may take about one to one and a half persons.

STL personnel gave their insights on lessons learned that can be beneficial to other ADMS developers:

3.2.3.2 NoVA District of VDOT

The Evaluation Team contacted the NoVA district to schedule a final interview. However, we learned that usage of the ADMS had not increased from February. Logon to the system was still a problem. NoVA personnel recommended that we not interview them as the lack of usage would not reveal anything beyond what was learned at the February interviews.

3.2.3.3 Open Roads Consulting, Inc (ORCI)

ORCI provided the software engineering for ADMS Virginia. The interview was structured into several major topics:

Project Roles and Responsibilities

STL developed the Concept of Operations, the detailed system requirements, and also supervised stakeholder input. ORCI's role was to develop the system but they participated actively in all of the up-front activities and provided feedback during the process. The participation of ORCI in this process was important in assuring that the product addressed the goals, objectives, and requirements of the project.

The original contract called for a single system deployment but the team decided instead to implement the project in three sets of builds. A fourth build was later added for Northern Virginia. ORCI saw the benefits of the multiple build approach from the beginning but had concerns about making such a significant change to the original plan. While this change involved some risk, the phased build approach turned out to be beneficial. The developers received feedback after each build and were able to improve the product as a result. A robust requirements definition process was included as part of each build. The three key members of the team, VDOT, STL, and ORCI knew each other well and worked together closely. Regular meetings fostered cooperation and helped keep the project on track. Each build had its own set of requirements and was conducted as a separate project, but added incrementally to the work of the previous builds.

The other major change from the original plan was use of a real-time GIS web-based interface. This was implemented in Build 2, to replace the original image map. This has been beneficial to the project and has been well-received. Real time features include current conditions, active incidents and detector quality reports. Data quality reports have been useful to both maintenance personnel. Planners and traffic engineers also have made use of these reports in order to determine which data sources to use.

All of the planned capabilities have been implemented. Other than the two items mentioned above there were only minor modifications in the original requirements. As a result of the multiple builds, there were less overall changes in the capabilities vs. the requirements. The multiple builds allowed for the incremental redefining of requirements to meet the stakeholders' needs STL serves as the primary interface for users. The system does provide an easy way for users to provide feedback or ask questions. These go to STL but may be passed along to ORCI. Planners and traffic engineers have been primary users. There have been more requests recently from consultants doing traffic congestion studies and researchers from out of state. This appears to be a unique resource. The flexibility of the system is perhaps its greatest benefit. The data are available to anyone who wants it and through the query system users can find the information that is useful to them.

STL was responsible for compiling data from the field detectors and developing the standardized database. ORCI was able to develop the interface tools based on this standardized format. In this arrangement, STL took the brunt of the responsibilities related to data quality and shielded ORCI from dealing with these issues (i.e., customizing the system to address these data problems). The availability of this central database allowed ORCI to develop their tools on an agreed data format and promoted the greater portability of the tools.

ADMS Implementation/Operation

Close coordination between the parties was a major factor in the success of the project. The project team had weekly meetings to review progress and these were important in making sure everything stayed on track. The meetings were especially helpful in handling the shift from a single system deployment to build phases. VDOT, as the ultimate client, was proactive in making needed decisions. STL had unique expertise that contributed to the success of the project. They had very strong database expertise that they used to develop data conversion procedures and clean up the data. This was combined with strong domain expertise in ITS and transportation. Their ability to address data gaps was important in making sure the system met the requirements and needs of the users. Rapid prototyping was used to obtain feedback from both stakeholders and the internal team. Some early versions of the software were issued only to the internal team for testing. Some very good input was obtained from both VDOT and STL by doing this. After these reviews the system was tested by a wider group of stakeholders. During the development process, pages would be posted to a website as they were developed to allow team members to review and make comments while the functionality was still being designed. This provided immediate feedback on the appropriateness of the design.

The data quality issues were raised early on during Build 1. The visibility of these issues highlighted the need to address data gaps in the software development process. It also raised the visibility of the detector maintenance issue for VDOT. ORCI made much greater use of third-party tools than originally anticipated. These tools worked effectively, saved development time and helped to produce a better product. The implementation through build phases made it easier to incorporate these tools and other new technology as the project progressed. The tradeoff in using these off the shelf components was that ORCI had less control and ability to customize the application. The development team worked closely to weigh the benefits of rapid implementation and lessened resource demands against the loss of flexibility/customized capabilities.

The strong points of the system are its flexibility in responding to different types of queries. The graphics and mapping capabilities are also strong points of the system.

ADMS Improvements

The speed of the system has been a concern for some users. ORCI is looking for ways to speed up the system but there are many variables impacting access time that are difficult to calculate - most related to external issues of data transfer over the internet. The system currently provides an estimate of query time; however, the time estimates are approximate (e.g., less than 5 minutes) since it is difficult to gauge the impact of these external factors that impact transfer time. ORCI noted that the software was developed using a smaller, fixed database. It would have been helpful to grow the database concurrently with the development effort. Although the fields were identical to the production database and allowed the various analysis capabilities to be successfully developed, the eventual size of the database contained many more records than anticipated. Having a more accurate representation of the final database would have allowed for greater optimization of the algorithms and provided a better indication of query time as the database grew.

ORCI noted that the speed could be improved for some users by developing a series of standard reports. For those users who want the same set of data on a regular basis, standard reports would be faster. Focused, packaged reports may help to attract more usage from operational users, since they need information quickly for decision making. The ADMS is developed using open source, providing the opportunity for these standard queries to be made on a system-to-system basis.

Use of the system for operational purposes was an important goal. ORCI noted that Hampton Roads has been interested in using the data for work zone planning. ORCI also noted that in their installation on the I-81 corridor there is interest in having the TMC central system draw data directly from the ADMS. Automating this capability would provide quick access to data for management decisions, particularly in addressing unplanned events such as incidents and bad weather.

Portability

Portability was the major issues addressed in Build 4, with the installation in Northern Virginia. This has been successfully accomplished due to several factors:

One key to successful portability is controlling customization. Ideally all systems would receive the same data so the platform can remain the same. ORCI noted, for example, that Northern Virginia did not initially have weather data so some customization was needed to remove the displays.6 The team carefully weighed the tradeoff: Is it better to have a standard template for all areas and risk showing blank fields when data is unavailable, or is it preferable to customize the outputs to the specific data availability and limited the reusability/portability of the system. In general, customization was limited in Build 4.

3.2.3.4 Summary of Major Lessons Learned

The standardization of archived data on a statewide basis was seen as having numerous benefits. It permits the query and display systems to be easily ported and provides a common basis for statewide system performance measurement. This also enabled ORCI, as the software developer, to concentrate on their task using a clear set of requirements. They were able to interface with users through a central point, STL, rather than dealing with multiple parties. The high level of technical expertise at STL was critical in accomplishing this. Knowledge of databases and ITS/traffic was important. A lesson for other States is that before funding individual systems to develop stand-alone ADMS, they should consider using University resources to standardize the systems across the State.

The phased incremental approach to software development was very successful for the Virginia ADMS. This approach enabled both internal developers and stakeholders to test the system before its completion and provide feedback. It also enabled the software developers to incorporate new tools and technologies as they became available, without causing delays in the project. Rapid prototyping was also a helpful mechanism in obtaining client feedback.

Weekly meetings of the project team were an important element in success of the project, particularly in the beginning. Small issues were worked out and did not become large problems later on. Frequent contact among the team members meant that minor adjustments could be made with little difficulty.

The flexibility of the query system makes the system useful to a wider range of users. However, some users become impatient with the time it takes the system to execute large queries. Some users may benefit from the development of standard reports.

It is helpful to test the system on the full database, as well as a limited sample of the database. This will provide a better understanding of system response time.

Reports on data quality have been very helpful to the end users. In addition to maintenance personnel, planners and traffic engineers have found that these reports provide a better understanding of the data and help them to focus their queries on higher quality data. This issue should be raised and addressed early in the software development process.

3.3 Evaluation of Hypotheses

For subjective information, this section relies heavily on the results of the interviews documented in the previous section. In these cases, the evaluation refers back to the interviews, but does not repeat the full text.

3.3.1 TMC Operations Planning

3.3.1.1 Hypothesis #1: Archived data tools enable STC staff to perform more effective Operations Planning

Goal: Improved TMC operations

Discussion: As shown in the interviews, STC staff did not use the ADMS for operations planning during the evaluation period. Operations planning was cited as the most practical use of the ADMS by STC personnel, but they currently do little in this regard. STC's focus is on real-time management of the system, mainly through coordinated incident management and posting traveler information. A big part of this issue is related to how the STC is staffed - most of the personnel are contractors. VDOT managers expressed positive comments about how this relationship is working and by all accounts day-to-day operations functions are handled very well. However, the contractors are not required under the terms of the existing contract to do any type of operations planning. VDOT has very few staff at the STC, and their duties are consumed by managing and overseeing the contractors. Finally, there was a feeling among STC staff that HRPDC is the group that "studies things" and they would be the proper unit to conduct operational planning in the short-term. However, the relationship between STC and HRPDC with regard to operations planning is still being worked out, so during the evaluation period, neither STC nor HRPDC had used the ADMS for operations planning.

Another major impediment to ADMS use - for operations planning or anything else - was the severe data quality and availability problem. This was cited by the STC as the main reason why the ADMS was not explored ("the data are so incomplete why bother with any analysis for now"). However, given the above discussion, it is not clear if the ADMS would have been used for operations planning even if data quality was high.

3.3.1.2 Hypothesis #2: Use of the ADMS Improves System Wide Travel Conditions

Goal: Less total delay and increased reliability

Discussion: The low overall quality of the data makes it difficult to draw any conclusions from data analysis. When coupled with the fact that the ADMS was not used to effect operations, it is clear that any changes in congestion or reliability levels can not be attributed to ADMS Virginia.

Nonetheless, an analysis of the data was undertaken. The data used came directly from the HRSTC for years 2000-2003. The data for 2004 came via ADMS Virginia. Data were subjected to the quality control procedures used in FHWA's Mobility Monitoring Program.7 These procedures encompass those used by ADMS Virginia plus several others. Tables 3, 4, and 5 illustrate the history of the data quality problem in Hampton Roads in dramatic fashion:

Table 3. Trends in the Travel Time Index on Hampton Roads Freeways
Corridor
Length
Travel Time Index
2000
2001
2002
2003
2004
I-64, EB (I-564 to I-264)
7.90
No data
1.03
1.13
2.16
1.42
I-64, EB (I-264 to Ches. City Line)
3.95
No data
1.01
1.12
1.42
1.41
I-64, WB (Ches. City Line to I-264)
3.90
No data
1.02
No data
1.45
1.28
I-64, WB (I-264 to I-564)
7.95
No data
1.03
1.00
1.38
1.28
I-64, HOV (I-564 to I-64)
9.20
1.03
1.01
No data
1.31
1.23
I-264, EB (I-64 to Va. Beach)
7.55
No data
1.00
1.02
No data
1.21
I-264, WB (Va. Beach to I-64)
7.55
No data
1.00
1.02
No data
1.39
2.40
1.00
1.01
No data
No data
1.61
I-564, WB (I-64 to Naval Station)
2.90
1.18
1.02
No data
No data
1.06
I-64 EB: 8th View St. to I-564
3.69
ND
ND
ND
ND
1.07
I-64 EB: I-664 to S. Willard Ave
3.88
ND
ND
ND
ND
1.31
I-64 WB:I-564 to 8th View St
3.94
ND
ND
ND
ND
1.37
I-64 WB: S. Willard Ave to I-664
3.89
ND
ND
ND
ND
1.06
I-264 EB: Va. Beach to Birdneck Rd
5.42
ND
ND
ND
ND
1.03
I-264 WB: Birdneck Rd to Va. Beach
5.39
ND
ND
ND
ND
1.03
I-664 EB: 39th St to I-64
3.99
ND
ND
ND
ND
1.04
I-664 WB: I-64 to 39th St
4.01
ND
ND
ND
ND
1.07
I-64 EB: Bainbridge Blvd to College Park Blvd
5.27
ND
ND
ND
ND
1.11
I-64 WB: College Park Blvd to Bainbridge Blvd
5.28
ND
ND
ND
ND
1.03

Table Notes:
ND=No detectors were in place during this time period.
The Travel Time Index (TTI) is a measure of total congestion. It is the ratio of the peak period travel time to the travel time under ideal conditions. A TTI value of 1.2 indicates that peak period travel takes 20 percent longer than under ideal conditions.




Table 4. Trends in the Buffer Index on on Hampton Roads Freeways
Corridor
Length
Buffer Index
2000
2001
2002
2003
2004
I-64, EB (I-564 to I-264)
7.90
ND
13%
28%
69%
69%
I-64, EB (I-264 to Ches. City Line)
3.95
ND
0%
34%
43%
81%
I-64, WB (Ches. City Line to I-264)
3.90
ND
6%
ND
49%
53%
I-64, WB (I-264 to I-564)
7.95
ND
11%
0%
50%
73%
I-64, HOV (I-564 to I-64)
9.20
3%
0%
ND
65%
58%
I-264, EB (I-64 to Va. Beach)
7.55
ND
0%
4%
ND
46%
I-264, WB (Va. Beach to I-64)
7.55
ND
0%
8%
ND
88%
2.40
2%
0%
ND
ND
184%
I-564, WB (I-64 to Naval Station)
2.90
55%
0%
ND
ND
8%
I-64 EB: 8th View St. to I-564
3.69
ND
ND
ND
ND
13%
I-64 EB: I-664 to S. Willard Ave
3.88
ND
ND
ND
ND
65%
I-64 WB:I-564 to 8th View St
3.94
ND
ND
ND
ND
68%
I-64 WB: S. Willard Ave to I-664
3.89
ND
ND
ND
ND
19%
I-264 EB: Va. Beach to Birdneck Rd
5.42
ND
ND
ND
ND
8%
I-264 WB: Birdneck Rd to Va. Beach
5.39
ND
ND
ND
ND
7%
I-664 EB: 39th St to I-64
3.99
ND
ND
ND
ND
8%
I-664 WB: I-64 to 39th St
4.01
ND
ND
ND
ND
22%
I-64 EB: Bainbridge Blvd to College Park Blvd
5.27
ND
ND
ND
ND
39%
I-64 WB: College Park Blvd to Bainbridge Blvd
5.28
ND
ND
ND
ND
8%

Table Notes: ND=No data.



Table 5. VMT and Lane-Miles for the Norfolk-Hampton Roads Urban Area
Year
Annual VMT (millions)
Percent Change
Lane Miles
2001
3,525
N/A
653.8
2002
3,622
2.76%
652.8
2003
3,852
6.35%
672.2
2004
3,849
-0.07%
672.1
Source: HPMS Universe data.

3.3.2 Planning Functions

3.3.2.1 Hypothesis #3: Availability of Archived Data Will Improve Accuracy of Regional Planning Models

Goal: Improved Regional Planning

Discussion: For the evaluation period, it can be said that ADMS Virginia did not improve the accuracy of regional planning models. Data quality and coverage problems were noted by HRPDC as the major barrier to use in regional planning models. Even if high quality data were present, the fact that only a small percentage of area freeways are currently covered by roadway surveillance is a limiting. However, HRPDC noted the potential for improving the accuracy of regional planning models by accessing the ADMS. These include:

In addition to providing data inputs for the travel demand forecasting model, HRPDC expects that the ADMS will provide hourly speeds and volumes for the DynaMIT traffic simulation model. The ADMS currently provides these data in DynaMIT input format.

3.3.2.2 Hypothesis #4: Availability of Archived Data Will Reduce Cost of Regional Planning Models

Goal: Improved Regional Planning

Discussion: Because the ADMS had not been used to supply data for regional planning models at HRPDC, this hypothesis could not be tested. However, in interviews, HRPDC staff stated that the direct cost of collecting input data for regional planning models would not be reduced by use of the ADMS. Rather, the ADMS would be used to collect data on the covered highway segments, allowing data collection on additional segments. In other words, HRPDC would expect their data collection costs to remain constant with use of the ADMS, but they would expand their collection coverage. For HRPDC, this would mean primarily conducting travel time runs (floating cars) because VDOT takes traffic counts on the freeways. If VDOT could use the ADMS to collect volume data on covered freeways (rather than having to deploy portable equipment), there would be a direct cost savings.

3.3.3 General Archive Functions

3.3.3.1 Hypothesis #5: The ADMS Provides a Mechanism for Improving the Quality of Traffic Data

Goal: Improved data quality

Discussion: In most TMCs, only cursory review of field detector data is performed - the level of checking is usually only if detectors are communicating with the TMC or not (on or off). Sometimes, field detectors will assign and communicate error codes and the TMC software will check for outlandish values. However, these procedures detect only the grievous errors, allowing more subtle ones to slip by.

STL took the issue of data quality very seriously from beginning of the project and designed into the system a series of sophisticated data quality control checks. STL defined these checks as follows:

  1. Maximum occupancy threshold - fail if occupancy > 95%
  2. Overall maximum volume threshold - fail if volume > 3100 vehicles/lane/hour
  3. Positive volume with zero speed - fail if volume is positive and speed zero
  4. Maximum volume threshold with a reported occupancy of zero - fail if occupancy is zero and volume > (volume when occupancy = 2%). This situation appears because occupancy is truncated to an integer and may result in a zero value, when in reality it is not.
  5. Average Effective Vehicle Length - this test is applied only to data where all of speed, volume, and occupancy are positive. This test is based on:

    AEVL = 10 * u * h / q where
    AEVL = Average Effective Vehicle Length
    U = speed (km/h)
    H = occupancy (%)
    Q = hourly equivalent volume (vehicles/lane/hour)

    Data fail this test if AEVL >18 or AEVL < 2.7

  6. Records containing zeros for all three values (volume, occupancy, and speed) are considered to be "bad."

If data fail the QC tests - or are missing to begin with - the data are flagged and imputation is conducted (see below under Hypothesis #7).

As demonstrated above and discussed by the interviewees, data quality has been a serious problem in Hampton Roads since at least 2002. The data were reviewed for quality using the QC process from the Mobility Monitoring Program; results are shown in Table 6.

Table 6. Quality Control Test Results on Speed Data, Hampton Roads, 2000 2004
Quality Attribute
2001
2002
2003
2004
Percent complete 35% 6% 39% 46%
Percent Valid 43% 31% 58% 47%
Percent of VMT Covered 9% 17% 18% 28%
Percent of Freeway Miles 11% 9% 10% 29%
Table Notes: (1) Validity is reported as the percentage of submitted data values that passed the quality control rules. (2) Completeness is reported as the percentage of data values available for use. It is calculated as the ratio of total available data values to total expected data values.


The results for 2004 are encouraging in the sense that the percent complete is the highest it's ever been in Hampton Roads. They are discouraging in the sense that quality still lags behind that of many TMCs as illustrated in Table 7.

In summary, ADMS Virginia provides the basis for improving data quality by producing information that can be applied by users in their applications and feedback to TMS personnel about the quality of data reported from the field. However, unless that information is acted upon by TMC and leads to improved maintenance of field detectors, data quality will not be improved. In fairness, this activity lies outside of the purview of ADMS Virginia and the evaluators found that the system itself does exactly what it is supposed to do in the realm of data quality.

Table 7. Summary of 2003 Freeway Archived Data Completeness
Participating City
Completeness of Analysis Data (percent)
Volume
Speed
Albany, NY
38
37
Atlanta, GA
57
54
Austin, TX
77
59
Baltimore, MD
63
57
Charlotte, NC
55
57
Cincinnati, OH-KY
44
41
Dallas, TX
46
44
Detroit, MI
61
62
El Paso, TX
33
33
Hampton Roads, VA
49
39
Houston, TX
N/A
56
Los Angeles, CA
98
98
Louisville, KY
82
76
Milwaukee, WI
80
77
Minneapolis-St. Paul, MN
83
79
Orange County, CA
97
93
Orlando, FL
--
--
Philadelphia, PA
89
88
Phoenix, AZ
63
60
Pittsburgh, PA
77
74
Portland, OR
84
83
Riverside-San Bernardino, CA
70
67
Sacramento, CA
88
83
Salt Lake City, UT
44
38
San Antonio, TX
67
66
San Diego, CA
95
92
San Francisco, CA
97
92
Seattle, WA
80
81
Washington, DC
33
33
Source: Texas Transportation Institute and Cambridge Systematics, Monitoring Urban Freeways in 2003: Current Conditions and Trends from Archived Operations Data, November 2004.


3.3.3.2 Hypothesis #6: The ADMS Is Portable To Other Areas

Goal: To Provide Transferability with A Minimum of Customization

Discussion: As discussed, the ADMS was successfully transported to the NoVA District of VDOT with minimal disruption. The success of this transfer relied on the facts that (1) the schema developed for Hampton Roads was a thorough representation of how archived data should be stored and (2) developing custom translation programs to populate the schema. There were some problems with geolocation for some NoVA field equipment, but once these were worked out, the ADMS performed properly. All of the applications developed for Hampton Roads were able to function for NoVA, presumably because the data collected by NoVA was similar in scope to that collected in Hampton Roads.

With regard to the schema, the Evaluation Team found it to be very comprehensive and provided a strong engine for ADMS Virginia applications. One shortcoming that could be easily fixed is the expansion of the incident data definitions and inclusion of data on work zones. In fairness, the expanded data for these events are not currently collected by operators, so the ADMS would have no source of the data. However, in the near future, these types of data are likely to become more important to operators. FHWA has initiated a pilot project that explores collection of data needed to support incident performance measures and evaluation of incident management programs.9 There is also a current FHWA project exploring work zone performance measures and the data need to support them.10 The expanded data can include all of the following, but even a subset of them would aid in performance measurement:

Incident Data

Data on the so-called "Incident Timeline" would allow operators greater flexibility in operations planning. Decomposing total incident duration into discrete "sub-events" is very useful for performance monitoring; tracking the duration of the sub-events can help identify areas that require improvement.

Work Zone Data

Work Zone Characteristics -- The actual and planned changes in the roadway environment created by the work zone. Used to measure the extent of work zones in time (duration) and space (amount of existing highway removed for the work zone), and their impact on safety and mobility. Also used in traveler information services to alert motorists to expected work zone conditions.

3.3.3.3 Hypothesis #7: The ADMS Development Process Has Met the Needs of the Stakeholders

Goal: Exemplary or "Model" ADMS Design

Discussion: On this evaluation point, the Team found ADMS Virginia to be an exemplary deployment of an archived data management system as specified in the National ITS Architecture. It is consistent with ASTM Standard E2259, Standard Guide for Archiving and Retrieving ITS-Generated Data on the standard's primary "guiding principles" as shown in Table 8. Several of ADMS Virginia's features are worth highlighting because the Evaluation Team expects these to serve as state-of-the-practice in guiding the development of other ADMSs.

Table 8. Consistency of ADMS Virginia with ASTM Standard E2259
Guiding Principle from Standard
ADMS Virginia Consistency
Reliance on User Needs and Requirements Process Highly consistent. A formal user requirements process was pursued in the design of ADMS Virginia
Providing for Diverse Needs and Requirements of Different Stakeholders Highly consistent. A wide variety of stakeholders were identified and involved in the user requirements process
Get Archived Data from Other Centers Highly consistent. The traditional traffic monitoring data was included in ADMS Virginia
Anticipate a Variety of Data Sources Highly consistent. Traffic and event data were included in ADMS Virginia
Retention of Original Source Data Highly consistent. Data as received from field detectors can be maintained by ADMS Virginia
Manage Archive to Account for Data Quality Highly consistent. An exemplary feature of ADMS Virginia (see text)
Establish and Maintain Metadata Highly consistent. An exemplary feature of ADMS Virginia (see text)
Process User Requests for Data and Information Highly consistent. The graphical user interface allows for easy access to the ADMS
Support Analysis of Archived Data Highly consistent. Many pre-packaged analyses were included in ADMS Virginia’s functionality
Prepare Data for Periodic Government Reporting Systems Somewhat consistent for traffic data where AADT values are computed, but not directly linked to data formats for other systems (e.g., HPMS)

Metadata

The Evaluation Team found the design and use of metadata in ADMS Virginia to be superb. Traditional metadata - what ASTM E2259 calls "archive structure metadata" - is readily available to users - descriptions of data elements and data relationships.

"Processing documentation metadata" was also included in ADMS Virginia. In fact, the Evaluation Team found this to be the first implementation of this concept in an ADMS. This is information about how the data were processed. Documentation on QC and imputation procedures is readily available. More importantly, the "flagging" of data as having failed QC or having been imputed is a major advancement of ADMSs. Finally, the calculation of the "normality index" - specifically develop for ADMS Virginia - is a highly innovative feature that could be use d in future deployments. This index provides users with information on how the currently viewed data deviates form "normal" or "expected" values for that location and time.

The final type of metadata specified in ASTM E2259 is "data collection system metadata" - information about the equipment and conditions under which data were collected. This type of metadata was not included in ADMS Virginia because it was not available (i.e., not collected by operators). Provision could have been made in the data structure for it, but we see no reason why ADMS Virginia should provide this if there is no reasonable chance of data being supplied by operators.

Imputation

From a user's perspective, having missing traffic data filled in via reasonable imputation methods is a powerful feature of an ADMS. This is particularly true for volume data, because most traffic measures involve summing volumes over time and space. (In contrast, speeds can be treated as a sample since aggregations typically deal with average speeds.) Testing of various imputation algorithms at STL has been ongoing for some time,11 and represent the state of the art in this field. Because imputation is transparent to end users, the interviews did not reveal any preferences or experiences with using imputed data, other than users would prefer high quality measurements to begin with. When data are imputed in ADMS Virginia, metadata flags are set, and users have the option in the applications to use or not use imputed data in calculations of statistics and performance measures. These features provide end users with options for computing measures such as AADT - either they can select imputed data and allow the system to compute the measures, or they can download unimputed data and use their own methods for accounting for missing data.

System Development Costs

In addition to the $300,000 annual maintenance and enhancement budget provided by VDOT, the actual development effort (in terms of hours only) is provided in Table 9.

Tables 9. Software Development Level of Effort for ORCI
Open Roads Labor Category
Hours
Build 1
Build 2
Build 3
Build 4
PM/Admin
131
164
98
39
Sr. Software Engineer
417
108
49
10
Software Engineer
394
502
584
452
Programmer
148
1011
575
324
Total Hours/Build
1090
1785
1306
825
Total Hours
5006


User Access

Table 10 shows the number of active users of ADMS Virginia as of May 25, 2004. "Active user" is defined as anyone who has established an account and run at least one query. The table also identifies new users since the release of Build 3.

Table 10. User of ADMS Virginia, as of May 25, 2004
User Group
Organization
New User
Active Users
Project Stakeholder Hampton Roads Smart Traffic Center
9
Hampton Roads Planning District Commission
3
City of Hampton
2
Hampton Roads Transit
1
VDOT – Central Office
9
Virginia Transportation Research Council
1
City of Norfolk
1
NOVA Safety Service Patrol
1
NOVA Smart Traffic Signal System
1
Researcher MIT
2
NC State University
1
University of Maryland
1
Auburn University
2
UVA – Smart Travel Lab
2
Texas A&M University
2
University of Delaware
1
University of Kentucky
1
External User Kimley-Horn and Associates, Inc.
1
P. B. Farradyne
2
Airsage
1
DMJM + Harris
1
ESRI
1
FDOT
3
Geodecisions
1
Illinois DOT
1
Maryland SHA
1
Minnesota DOT
1
Battelle
Y
1
Mitretek Systems
Y
1
New York DOT
Y
1
PBS&J
Y
2
FHWA FHWA
4
Evaluation Team SAIC
1
Cambridge Systematics, Inc.
1
Development Team UVA – Smart Travel Lab
10
Open Roads Consulting, Inc.
2
George Mason University
1
Source: Earnest, Ken, Build 3 – Performance Analysis Report, June 7, 2004


There were a total of 77 active users that are categorized into one of six user groups: Project Stakeholder, Researcher, External User, FHWA, Evaluation Team, and Development Team. There were 36 additional users (scattered among the six user groups) that have established an account but have not used the system.

Figure 4 shows the types of queries submitted by users during the initial phases of Build 3. AT least at this stage, the predominant usage is the downloaded of measurement data fro individual traffic detectors. This is consistent with the interviews of planning personnel who said their primary use was (and would continue to be for the short-term), data to feed other applications.

Figure shows types of queries submitted and the number of each during Build 3. The station data download query was made 158 time during the build. Other query types and the number of queries made for each type include: Norfolk data (25), traffic flow spatial (23), current cond. active incs. (16), traffic flow timeline (9), OM data quality (6), special events table (5), DynaMIT Simulation (4), M AADT analysis (4), MM traffic download (3), std. weather download (3), traffic forecast (3), traffic forecast accuracy (3), planning - air quality (2), std. incident download (2), MM traffic flow timeline (1), std. incident plot (1), and std. TMS download (1).
Figure 4. Types of Queries Submitted During Build 3 (4/1/04 - 5/25/04)

3.3.3.4 Hypothesis 8: The ADMS Has Satisfactorily Fused Data from Different Sources

Goal: Applications and Queries Can Access and Use Disparate Forms of Data

Discussion: ADMS Virginia has successfully fused traffic, incident, and weather data into a variety of applications. It has done so by rectifying any potential location referencing problems, as discussed in the interviews in the previous section. The applications that plot or use these different data sources all perform satisfactorily, based on the limited experience of the users (mostly in exploratory fashion rather than for use in planning or operations applications). The successful fusion of traffic, incident, and weather data will allow more complex analyses of system conditions oin the future, such as decomposing total congestion into its component sources and documenting the performance benefit from operations strategies (such as incident management.)





2The HRSTC is staffed primarily with contractor personnel, with oversight by VDOT personnel. The operations contractor is currently responsible for entering incident data.
3Note: there are no current plans for this type of traveler information system.
4The ADMS Virginia developers have been aware of the speed of access/query turnaround time issues and have been working to improve them since this interview was done.
5Since the evaluation interviews, the NoVA District has conducted such an analysis.
6Weather data has recently been added to the NoVA part of ADMS Virginia.
7htpp://mobility/tamu.edu/mmp
8This shows the importance of checking the output of field equipment with spot checks in the field. Post hoc QC tests are limited in the data problems that they can catch.
9Focus States Initiative: Traffic Incident Management Performance Measures, http://ops.fhwa.dot.gov/incidentmgmt/
10http://ops.fhwa.dot.gov/wz/decision_support/perf_measurement.htm
11Conklin, James, Data Imputation Strategies for Transportation Management Systems, Masters Thesis, University of VA, May 2003.

Previous | Next