Final Report
Model Deployment of a Regional,
Multi-Modal 511 Traveler Information System
Multi-Modal 511 Traveler Information System
Appendix B
Sample Weights
The construction of survey weights is a standard survey practice. Weighting of survey data is typically performed to adjust the relative importance of any one response to reflect that not all survey respondents were selected with the same probabilities, to reduce bias in survey estimates from differing patterns of response, and to align sample respondent distributions to known population distributions to improve coverage and precision.
Evaluation of the Need for Weights
The survey was designed as a stratified systematic probability sample with time-of-day and day-of-week as stratification factors. The sample size was proportionally allocated to each stratum based upon historical call volumes, and the same intercept rate was to be employed in each stratum. Therefore, each respondent would have a roughly equal probability of selection regardless of the stratum. However, due to logistical constraints and other factors, the intercepts were not conducted at a uniform rate, resulting in unequal probabilities of selection that need to be adjusted for with the survey weights. Additionally, the response rates for this survey varied significantly between the strata (see Figure B-1). Both of these factors resulted in the distribution of completed interviews differing from the distribution of callers into the 511 system (see Figure B-2 and Figure B-3). In particular, a higher percentage of interviews were completed during the end of the week than the beginning of the week compared to the distribution of all calls, and a higher percentage of completed interviews were from the evening rush compared to the distribution of all calls.
Figure B-1. Box-and-Whisker Plot of Response Rates by Day-of-Week
(mean indicated by "+")
Figure B-2. Distribution of 511 Calls and Completed Interviews by Day-of-Week
Figure B-3. Distribution of 511 Calls and Completed Interviews by Time-of-Day
Construction of the Weights
Weights for this survey were constructed using a three-step process in which each step modifies an interim weight developed in the previous step. These steps are outlined below.
Step 1. Calculate Base Weight: The base weight in stratum i was calculated as the reciprocal of the probability of selection.
Step 2. Calculate Adjustment for Response Rate: The base weight in each stratum was adjusted to account for differences in the stratum-specific response rates by multiplying the base weight by the reciprocal of its stratum-specific response rate.
Step 3. Normalize Weights: In this survey, the true number of unique callers to the 511 system is not known because many callers use the system multiple times. It is not possible to completely identify all of these multiple users through a unique identifier, such as the telephone tag in the 511 server logs, because this information is not available for every call. Thus, it is impossible to align the survey distributions of individuals who completed the survey to the overall population of individuals. Therefore, the final step in creating the survey weights was to normalize the weights back to the size of the sample (i.e., number of completed surveys) by dividing the calculated weight by the average weight. This maintains all of the relative adjustments for differing probabilities of selection and response, but will result in weighted survey estimates with sample totals instead of population totals. Note that the number of strata is represented by s in the two equations that were used to normalize the weights.
