CHAPTER 9.0 HOUSEHOLD SURVEY PRETESTS
Note: portions of this chapter come from the 2008 NCHRP Report 571, with review and contributions by Nancy McGuckin and Rob Sheldon.
9.1. Pilot Surveys and Pretests
9.1.1 Requirements for Pretests or Pilot Surveys
Pretests and pilot surveys are the process of testing various aspects of the survey design, protocol, instruments, analysis, etc., on a small sample of the population prior to fielding the main survey. The intention of pretests and pilot surveys is to determine whether everything in the intended survey will work and produce the expected results. In some instances, pretests or pilot surveys may be conducted to compare two or more methods for some element of the survey process and to determine which to choose. In other cases, there is no comparison test involved, although it may be anticipated that some refinements to elements of the survey process will result. Further elaboration on pretests and pilot surveys can be found in Section 6.1 of the Technical Appendix.
It is recommended that the terms pilot survey and pretest be defined as follows:
1. Pilot Survey—a complete run through or dress rehearsal of the entire survey process, including drawing the sample, conducting the survey, coding the data, and performing basic analysis of the data. A pilot survey is conducted on a small sample of the same population that will be sampled for the main survey. As distinct from a pretest, the pilot survey involves a test of every element of the main survey, conducted in exactly the same way as is planned for the main survey. A pilot survey may also be used to test two or more different survey procedures and compare the results in order to assist in selection of one for the main survey. In such a case, each version to be tested is subjected to every step of the main survey.
2. Pretest—a test of any element, or sequence of elements of a survey, but comprising less than the full survey execution. For example, the instrument may be pretested by having a small subsample of respondents complete the instrument and then reviewing limited aspects of the completed instruments to determine whether any design changes are warranted. Any aspect of survey design and implementation may be subjected to a pretest. Pretests may also be used to compare alternatives for an element or elements of a survey. The main distinction between a pretest and a pilot survey is that pretests do not involve testing all aspects of the planned main survey, but may be limited to subsets of the protocol, instrument, sampling, etc. During the design phase, several sequential pretests could be conducted to test various refinements of the instrument, protocol, sampling, etc.
Second, it is recommended that one or more pretests and/or one or more pilot surveys should be an essential step in all transportation surveys unless there are specific circumstances that render such a step unnecessary and unlikely to produce useful information.
It is further recommended that the following guidelines with respect to pilot tests and pretests be adopted:
1. In any survey in which interviewers will interact with respondents, the pilot survey or pretest should include listening in to interviewers to determine how they interact with potential respondents, how well they keep to the script of the survey, and whether the script causes difficulties in conversational style.
2. In any survey that uses interviewers or observers, there should be a debriefing with those used in the pilot survey or pretest to determine whether difficulties were experienced in handling survey procedures, questionnaires or other materials, scripts, etc.
3. If it has been 10 years or more since the last time a survey was done, a pilot survey should always be undertaken because the changes in population that will have occurred will render any past experience irrelevant.
9.1.2 Sample Sizes for Pretests and Pilot Surveys
Because we recommend that pretests and/or pilot surveys be conducted in all future travel surveys, it is appropriate to establish the required sample size of these initial tests or surveys. For further discussion, the reader is referred to Section 6.2 of the Technical Appendix.
It is recommended that the following standardized procedures be adopted by the profession:
1. Whenever possible, the main sample should be drawn first and the pilot survey or pretest sample drawn only from those households or persons who were not drawn for the main sample. When the pilot survey or pretest is being conducted to determine the sample size required for the main survey, two options are possible. The first option is that a main sample can be drawn that is expected to be more than sufficient in size. The pilot survey or pretest sample can then still be drawn subsequently from those households or persons who will not be included in the main sample under any likely circumstances. The second option is to draw the pilot survey or pretest sample at random from the total population and then be sure to exclude all such drawings from the population for drawing the main sample. The former of these two is the preferred method.
2. No pretest or pilot survey should use a sample of less than 30 completed households or respondents. Exercises using smaller samples than this should be regarded as preliminary tests and pre-pilot surveys and should always be followed by a pretest or pilot survey with at least a 30 respondent sample size.
3. The minimum sample sizes shown in should be used in all pilot surveys and appropriate pretests.
9.2 Pretesting
The complexity and size of household travel/activity surveys makes careful and thorough pretesting an essential element of the survey implementation process. The pretest offers the survey team the opportunity to analyze the household travel/activity survey design elements before it is too late to change them. However, survey teams often fail to take full advantage of the pretests by failing to conduct them in earnest or by conducting them too late in the survey implementation process.
The section summary lists the three overriding questions with regard to pretesting. Essentially, the questions are: 1) what does one pretest?; 2) how does one conduct the pretests?; and 3) how does one analyze the results?
9.2.1 Survey Design Components Analyzed in the Pretest
As discussed in Chapter 2.0 of this manual and reiterated below, one pretests as many aspects of the survey as possible by performing the exact same steps as the actual survey. Household travel/activity survey pretests provide the survey team with the opportunity to:
· Refine fieldworker, interviewer and office worker procedures and logistics;
· Test and revise question wording, sequencing, and formatting;
· Determine the range of potential responses to questions (such as how many trip or activity spaces are needed on diaries) and identify unexpected responses and respondent behavior;
· Identify problems with the sample population and the sampling frame;
· Develop preliminary estimates of the variance in key variables to help establish final sample sizes;
· Estimate response rates;
· Estimate the survey completion time and cost; and
· Compare alternative approaches to gathering certain data items.
9.2.2 The Pretesting Process
Despite the potential advantages of careful and exhaustive pretesting for household travel/activity surveys, all too often the pretest is the first part of the survey process that is cut back when time and cost pressures build.
The ideal approach for conducting pretests is outlined in Chapter 2.0. The process has three steps: the office pretest, the questionnaire pretest, and the survey dry-run.
In the office pretest, survey team members ask 10 to 12 colleagues and other experts who are not directly involved with the household travel/activity survey to review proposed procedures, and more importantly, to examine the survey questions and the questionnaire. The office pretest can be conducted formally or informally, and usually is a very effective way to identify problems with the questionnaire and with specific questions.
Based on the advice of this expert group, the survey team should revise the questionnaire content, wording, sequencing, length, and format. If the revisions are serious, the revised questionnaire should be brought back to the experts for another round of corrections.
Once the expert group is fairly comfortable with the questionnaire, the survey team should perform a questionnaire pretest. Household travel/ activity survey questionnaires, and particularly heavily-formatted question types like diaries, should be tested on non-experts because they are often confusing to people without knowledge of transportation planning or people who are unfamiliar with survey questionnaires, and because the surveys often rely upon respondents’ understanding of technical (and sometimes ambiguous) terms and expressions. During the questionnaire pretest, respondents are administered the survey and are asked to describe any problems or areas of confusion that they encountered.
Often, these pretests are personally-administered even when the ultimate survey will not be. It is becoming increasingly popular to conduct this portion of the pretest as part of a formal or informal focus group of 10 to 15 participants. This allows the survey team to observe first-hand how respondents react to the survey; something which is usually impossible for mail surveys and only partially possible for telephone surveys.
Based on what the non-experts say about the questionnaire, the survey team may decide to revise the questionnaire, and to re-start the pretesting process. More likely, the revisions will be minor, and the survey team will be ready to take the survey instrument to the third phase of the pretest, the household travel/activity survey dry-run, or pilot survey.
In this part of the pretest, the survey team actually completes the survey on a small number of respondents, following all planned survey procedures as closely as possible. Survey teams for large household travel/ activity surveys usually conduct pilot surveys with 100 to 300 respondents, often testing slight variations in design. Smaller survey efforts and most market research efforts have pretests with 30 to 50 respondents.
In the pilot survey, or dry-run pretest, all the field implementation and data processing tasks should be performed in an identical manner to the full survey effort. This effort should be “a cradle-to-grave” testing of the entire survey study, from drawing the sample, to conducting the survey, to geocoding responses, to analysis of responses (including perhaps trip linking, sample weighing and expansion, and imputation of missing data). The pretest will be unable to test issues that arise when the survey is applied to the high volume of final respondents, but ideally, the pretest can be used to ensure that all aspects of the survey effort are in place and are operating as expected prior to the beginning of any data collection. Note that if a full dry-run is performed, almost all of the final survey procedures and programs will be developed, improving the efficiency of analysis of the full survey.
9.2.3 Evaluation of the Pilot Test Results
The analysis of pilot study results is described in detail in two recent publications by E.S. Ampt et al. (1985) This section briefly describes the analysis of the eight survey design components listed above, but survey teams can expand upon these analyses whenever the pilot survey data permit.
9.2.3.1 Analysis of Survey Procedures and Logistics
Once the pilot survey is completed, the survey team should compare the actual amount of time and level of effort required to complete individual components of the survey effort to the levels that were predicted prior to the study. Those components that cost more or took longer than expected should be evaluated in detail. The survey team should determine whether the planned procedures had problems and/or whether the individuals responsible for completing the procedures did not complete them correctly.
The survey team should also consider whether inefficiencies in the survey implementation process are likely to be magnified by the much larger final survey effort. For instance, if preparing labels for a mail survey based on a recruitment call is a little slow with a pilot test of a few hundred responses, it may be a major problem with a survey with thousands of responses. Some aspects of the pilot survey process will necessarily be different from the main survey effort, but many procedural and logistical survey components can be evaluated.
If problems are identified, the survey team will likely want to change training procedures or to re-design staff responsibilities. It is important that the actual survey staff work on the pretest. If interviewers are being used, the ones that are scheduled to conduct the survey should also conduct the pretest. If only experienced interviewers are used in the survey pretest, the pretest will not accurately reflect actual conditions.
9.2.3.2 Analysis of Survey Questions and Questionnaires
Richardson, Ampt, and Meyburg recommend the following analysis of the questionnaire (Richardson, Ampt and Meyburg, 1995):
1. Do instructions appear to have been read and followed?
2. Are definitions clear? Are there any consistent misinterpretations on the part of either the respondent or the interviewers?
3. Are questions clear and unambiguous? Are there signs that respondents or interviewers have misunderstood the intent of the question?
4. Do the cover letters appear to have been read and understood? Was the survey information telephone number used by any respondents?
5. Do the answers to individual questions indicate any problems? Are special techniques such as attitude rating scales producing valid answers? Too much bunching of answers may indicate a leading question or badly chosen categories. Too many “Don’t know” responses might indicate a vague question, a confusing question, or an unimportant question. Too many refusals to a question may indicate that it should be asked more delicately, the order of questions should change, or the question should be omitted.
6. Is there any evidence that the questionnaire is too long? Too many unanswered questions or hurried answers towards the end of the questionnaire indicate that perhaps the questionnaire is too long for the amount of interest shown in the subject matter.
7. Is the sequencing of questions clear? Are interviewers asking and/or respondents answering questions that do not pertain to them? Are more branching and skipping instructions needed? Is it clear what question should be answered next after a branching question?
Questionnaire problems should be handled in the same way as for the earlier pretests. If major questionnaire revisions are needed, the questionnaire should be redesigned and the survey team should consider restarting the pretest task. Otherwise, minor questionnaire problems should simply be addressed.
9.2.3.3 Analysis of Responses
The survey team should evaluate completed questionnaires for layout problems. All responses should be easily read by coders and data entry specialists. The survey team should ensure that enough space is provided for potential responses, and that the questionnaires do not require interviewers or respondents to try to fit written responses in extremely small areas. A common problem on factual questions is that an “other” response requests a more detailed description, but the layout of the form makes it difficult or impossible for the respondent to comply. If a diary form is used, the survey team should ensure that the number of potential activities and trips on the diary is adequate for the diary period without being excessive (causing the survey to appear even more challenging).
The survey team should also determine whether any questions or parts of the survey forms have been systematically overlooked by respondents, and conversely whether sections of the questionnaire that should be skipped are being completed.
9.2.3.4 Analysis of the Adequacy of the Survey Population Definition and the Sampling Frame
Because the output of the pilot survey, like the output of the actual survey, will usually be the input datafiles for one or more travel models, the survey team can evaluate the pilot survey returns to determine whether the survey population and sampling frame have been defined properly for the likely analyses. The survey team should determine whether individuals outside of the scope of potential analyses are being asked to complete the survey. If they are, the survey team should consider ways to redefine the survey population, to improve the sampling frame, or perform better screening procedures. There is no sense in expending resources on individuals or households that will not factor into the final analyses.
Similarly, the survey team should attempt to identify segments of the survey population (geographic areas, income levels, household types, etc.) that did not seem to be getting contacted in the pilot survey. In most cases, missing population segments are caused by the small pilot survey sample sizes, but the survey team should ensure that these segments are, in fact, represented in the sampling frame. Usually, Census data sources can be used to identify missing segments of the population of interest.
9.2.3.5 Analysis of Sample Size Estimates
The pilot survey provides preliminary information on how precisely the actual survey sample sizes will measure certain parameters. As discussed in Chapter 5.0 and Section 6.5, the actual precision of survey-derived parameters can only be known after the final survey is complete and the data collection resources have been depleted. To estimate the necessary sample size that will provide adequate precision before the survey, the survey team must estimate the variance in key variables. The pilot survey represents one of the best sources of information on the likely final variances.
The survey team should calculate the variances and coefficients-of-variation for the key pretest results, and estimate the likely final survey precision levels. The survey team may determine the need for either more or less completed surveys based on this information. The necessary sample size to produce a desired confidence level and a maximum sampling error is given by Equation 5.8a:
(Eq.5 8a)
where:
CV = Coefficient of Variation
Z = Z-Statistic
d = Relative precision level
2 = Variance of the random variable in the population
m = Mean of the random variable in the population
The pilot test provides the survey team with reasonably accurate estimates of what the variance, 2, and the coefficient of variation, CV, will be in the final survey. By plugging in the pilot test derived coefficients of variation and the desired precision and confidence levels into the above equation, the survey team can determine the minimum final sample size. Based on this calculation, the survey team can increase or decrease the survey sample size target or adjust desired levels of precision and confidence.
Shiffler and Adams have found that for small sample pilot surveys, the variance and coefficient of variation estimates tend to systematically understate the final survey’s actual variance and coefficient of variation (Schiffler and Adams, 1987). The authors suggest applying the correction factors shown in Table 6.20 to calculated sample size estimates that are based on small sample pilot surveys. For a pilot survey with ten responses, Shiffler and Adams suggest that it is necessary to increase the sample size calculated with Equation 5.8a and the pilot survey’s coefficient of variation by 7.1 percent.
It is recommended that survey teams perform pilot surveys with more responses than are shown in the table, but because household travel and activity survey samples are usually stratified, it is likely that individual strata will have pilot survey responses in the ranges shown.
Of course, whenever possible, the survey team should use more than the minimum calculated sample size because, even if the above correction factors are applied, there is no way to know if the final variance and coefficient of variation will be higher than for the pilot study. The above corrections simply equalize the probability that the calculated sample size will be sufficient.
9.2.3.6 Analysis of Response Rates
The survey teams should carefully track the levels and types of non-response encountered in the pilot survey. The biggest question in this regard is whether response levels are different than the survey team has anticipated. Depending on the survey method, variations in response rates can have a significant effect on the survey cost and completion schedule. In addition, higher than expected non-response levels may severely weaken the confidence one can have in survey-derived parameters.
With 30 to 50 completed questionnaires, the survey team can begin to get a picture of what the actual response rates and survey completion times are likely to be. This will allow the team to get a much more accurate estimate of the survey fieldwork costs and the amount of time the survey will take. If these actual measures are significantly different than had been estimated, the survey team will want to consider drastically changing the survey method and techniques. Of course, this will entail re-performing the survey design, sampling, and organization tasks, to some extent.
In addition to measuring non-response levels, the survey team should, to the extent possible, surmise the causes of the non-response. If any identifying information is available for both the pretest respondents and non-respondents (such as the location of their homes or type of housing unit), the survey team may be able to determine whether non-response bias is present. Some practitioners have used follow-up interviews with pilot survey non-respondents to develop strategies in the final survey design that address the potential biases.
Table 9.1 Shiffler’s and Adam’s Correctional Factors Recommended Sample Size
9.2.3.7 Analysis of Survey Completion Time and Cost
If the pilot survey is carefully monitored, the survey team will be able to obtain fairly accurate estimates of the times and costs of each element of the survey fieldwork. Of particular interest, in this regard, is the time of any interviews that are conducted. Interview time is an important design parameter because interviewers are generally paid by the hour, and because as interview times increase, so do the rates of incompletions and non-responses.
If the final survey times and costs are estimated to be longer and/or higher than the sponsoring agency has available, the survey team can:
· Change survey procedures;
· Omit some questions to shorten the survey;
· Lower sample size targets;
· Increase the data collection budget; or
· Cancel or postpone the final survey work.
Note that if contractors and consultants are being used, the sponsoring agency should include in the contract an allowance for this decision to be made. Often, agreements are formulated in which it is to neither party’s advantage to consider the full range of choices. Under these conditions, the usefulness of the survey pretest is diminished.
9.2.3.8 Analysis of Alternative Approaches and Methods
During the survey design effort, it is common for survey team members and/or other experts to disagree on the best approach to be used for certain elements of the survey design, or for the survey team to simply not be sure what is the best approach. The pilot survey is the ideal place to select between alternatives because slightly different procedures or instruments can be given to different groups of pretest respondents, and then can be compared somewhat objectively.
The recent NCTCOG household activity survey pretest involved comparing:
· Two different diary recording periods (24-hour versus 48-hour);
· Two different diary formats (complete diary versus simplified diary);
· Two different data retrieval methods (CATI versus mailback); and
· Three different types of incentives.
To perform these types of controlled experiments, the pilot survey sample size may need to be increased, especially if the different elements being compared are likely to interact with each other.
9.2.4 Pilot Survey Schedule
Usually, the pilot survey results do not imply the need for drastic changes in the survey design prior to the final survey, but because revisions of some type are to be expected, it is essential that the survey team complete the pretest well in advance of the planned fieldwork period. In addition, the survey team needs to allocate sufficient time and money resources to pretesting, so that they are prepared to make changes in the questionnaire and procedures based on the outcome of the pilot survey.
REFERENCES
