10 Things to Know About Survey Implementation

1 Decide whether or not to work with a local firm

There are two discrete skill sets needed on the ground when implementing a survey. The first skill set is focused on administration/logistics, and the second skill set is focused on research design. The first set of skills is needed for administration: budgeting, creating route plans, recruitment, and management of staff. Administration requires a level of familiarity with local conditions; for example, the ability to quickly estimate costs and troubleshoot logistical issues are important here. The second skill set requires knowledge of research methods to ensure that survey implementation is consistent with the research design. Researchers must be able to recognize any deviations from the protocol and address them in a way that leads to as little bias as possible. Important here is a deep understanding of the survey protocols and possible alternatives, in the case that changes need to be made on the ground.

If you will not be present during survey administration, you will need to either hire a firm or individuals who can work together to cover both sets of needs. There are clear advantages to hiring a firm if you have the budget, the biggest being that firms coordinate internally and balance both sets of needs, ensuring that logistics accommodate the design and vice versa. A possible drawback is that firms frequently have their own protocols, and these default procedures are usually at a lower standard than the latest protocol being used in academia. Upgrading protocols is a costly process and firms may push back against the use of stricter, or simply different, practices.

If you will be present during enumeration but you have a small budget, or do not feel you could manage the entire implementation (administration and design) yourself, a good alternative is to hire a field coordinator from a survey or research firm on a consultant basis. This person can help with administration while you take on the design-related work. Additionally, hiring someone for administration locally can do a lot to help with cross-cultural management. The types of management procedures that might work to motivate or sanction employees in the US may not work in another context, so someone who knows what is acceptable and effective can add a lot of value.

1.1 Contracts with local firms

When setting up contracts with local firms it is important to get the incentives right—thorough and good work should also be the most profitable for the firm. You can do a lot to set expectations and incentives in the contract. For example, pay on delivery where possible (although it is customary to pay some costs upfront to cover fixed expenses like transport and early salaries). You can also choose to impose financial penalties for late or low-quality data, but be sure to make these requirements clear up front and provide specific rules for what constitutes low-quality work and how late penalties will be assigned.

In addition to direct costs, it is reasonable for a local firm to charge overhead. This can vary from context to context, and it is best to check against the budgets of other similar projects to make sure the rate is reasonable.

2 Budgeting

As you prepare to begin survey implementation, the first and most important part of the survey is budgeting expected costs. It’s important to be thorough and detailed in putting together your budget; it will be critical throughout implementation and will be closely considered by potential funders.

The total cost of a survey is the sum of fixed costs, like transportation and equipment costs, and variable expenses like salaries, per diems, and administration costs.

2.1 Salaries

Estimating total salary costs before drawing the sample (needed in order to determine the teams and route plans) requires a bit of guesswork. One approach is to estimate the work-hours needed to conduct the survey (survey length x sample size) and divide by some estimated number of enumerators to come up with the number of enumerator days you will need to pay. The per diem may need to cover food and lodging, and make this clear to enumerators so they can plan accordingly. For surveys that will require long fieldwork, it is good practice to pay salary on a rest day each week although some enumerators prefer to work continuously in order to finish sooner and return home. This choice is context-specific.

2.2 Per Diems

Per diems cover enumerator’s expenses associated with doing fieldwork. This means lodging for overnight stays, all meals, and sometimes also transportation. Per diems should also be paid on rest days that fall in between work days. In the case that the variation in lodging and food costs is low, it is not important to change the per diem rate according to location. Teams will know when to save and when to spend.

2.2.0.1 Quick calculator:

((survey time to complete * sample size)/workable hours in a day)/# of enumerators = number of days

number of days * (daily rate + per diem) + supervisors = approx. total salary cost

2.3 Transportation

It’s important to, ex ante, be as accurate as possible in estimating the full cost of transportation as this is frequently both least flexible and most variable cost. Typically, it is good practice to build in contingency on the cost of fuel, as the price can change over the several months it takes to go from the grant application stage to the implementation stage. If you are budgeting before drawing your sample, pay particular attention to hard-to-reach areas in your population (islands, places without road access) and pad your transport line for the possibility you randomly sample enumeration areas that carry these higher costs.

2.4 Equipment

Later on in this guide we present the benefits of using personal digital assistants (PDAs) or tablets for data collection (see section 3). PDAs/tablets can be either purchased using survey funds or leased from a research firm, university, or other researchers.

3 Should you use PDAs or tablets instead of paper?¹

3.1 Why Personal Digital Assistants [PDAs] or Tablets are better (if you have the budget)

Use of a PDA/tablet allows the collection of more accurate and detailed data (Goldstein, 2012) because of:

Automated skip patterns
More detail, e.g. the ability to program a multi-stage code list
The ability to program some randomization algorithms (permuted block randomization, for example)
Sensitive questions can be recorded by the respondent themselves on the tablet (instead of the enumerator). There are even ways, using sound and video playback, to do this with illiterate respondents
The ability to audio record responses for later transcription

PDAs/tablets have lower error rates than paper-based surveys (Caeyers, 2010) and have superior quality control options including:

Real-time data upload
Real-time survey modification in the case of error or oversight in terms of questions included
Audio recording of portions of surveys to verify enumerator delivery
Timers that measure how long respondents spend on the entire survey and each individual question
Real-time validation checks to make sure numerical questions don’t have answers that are nonsensical
The ability to generate several orthogonal treatments within a single survey (either for multiple experiments or for conjoint experiments)

3.2 Tips for PDA or tablet use

Always buy extra equipment – chargers, battery packs, power strips, and tablets can go missing, be stolen, or get broken. In many countries you can’t buy extra equipment, even in capital cities, and it’s often more expensive and lower quality than what you can get at your home base. Buying 10-20% more equipment than you need can be expensive, but it is usually far cheaper than the salaries that you will have to pay for enumerators with equipment problems who do not have backups.
Pay close attention to battery life when you buy your equipment. If you want full days of enumeration, some of the cheaper tablets will not work.
Budget for extra battery packs for your enumerators to carry in the field, particularly if they will travel to rural areas where they may not always be able to charge the tablets every night.
Forecast how frequently teams will be able to upload recorded data. The PDAs/tablets need to be able to store data from completed interviews until uploading is possible. In rural environments this can mean quite a lot of memory is needed, particularly if the survey is long and/or complex.
Budget extra costs for charging of the PDAs or tablets in the field (i.e. paying for extra generator time from hotels) and for potential delays because of lack of power.
Budget extra time for exhaustive testing of the PDA/tablet once the survey is fully coded. Code failures can be disastrous the more complex the code/randomization becomes. Run through as many different responses to the survey as you can yourself, and do a “fake” pre-test during training in which you collect data and inspect it to make sure there are no errors.
Have someone available to make on-the-spot changes to the code in case problems are discovered in the field that stall enumeration until the change is implemented. In addition, give your enumerators enough paper versions of the survey to last for one or more days to serve as a holdover until the code is remedied.
If possible, name your variables in the survey software to avoid a really laborious process of manually re-naming later. This also makes it easier to inspect the data in real-time.
Code answer values (i.e. the values that will be outputted into the dataset) in advance so that you can standardize scales and easily clean the data (for example, use different negative numbers for “don’t know” and “refuse” options so a 10-character command in R can clean the whole dataset).
Take advantage of having more space for text by giving enumerators directions for complicated items in the displayed question text itself.

4 Establish a Management Structure

Parallel reporting chains can greatly improve the reliability of data by providing incentives to declare errors and mistakes. Under a parallel structure, oversight staff reports directly to survey manager, while surveying staff reports to field coordinator. The aim is to use the auditing and back checking reports in order to cleanly identify problems, and field supervision to then correct any issues.

4.1 Surveying

Field teams are made up of enumerators and a team leader. Team leaders report to a field manager, or in a case of a large survey, a regional supervisor.

The enumerator’s role is sampling and selecting households and respondents within enumeration areas, gain consent, and conduct the interview. The enumerator’s tasks include:
- Selecting the household. For an enumerator this is the first stage in the random selection process and is done according to a clearly specified procedure, which should be easily referenced in both the manual and the survey instrument itself.
- Selecting respondents. Once the household has been selected, the enumerator should follow a similarly-specified random (or systematic) selection process in order to select the subject.
- Consent. Enumerators need to know the definition of informed consent and how to make sure the respondent understands his/her rights during the interview.
- Conducting the interview. This will involve asking questions and closely following the instructions communicated in training and on the questionnaires. A hard copy of question-by-question instructions should be provided to enumerators for use as a reference.
- Controlling the interview situation. The enumerator must work towards reducing or eliminating suspicion and prejudice within the interview environment. This may involve asking bystanders not to congregate or dealing with sensitive situations with respect to other family members within the home.
- Avoiding bias. The enumerator’s personal views must not be reflected in the data collected. This means, among other things, that the enumerator must remain neutral and respectful during the data collection by not expressing his or her opinion and ensuring that the respondent trusts that the enumerator will respect his or her privacy. Emphasize during training that there really is no right answer and that the goal of the survey is to find out what people really think – because without knowing that, we can’t find solutions to problems.
- Presentation. Interpersonal skills such as manners, dressing, body language and ability to persuade are all important for data quality and will help in obtaining the target respondents for each day.
The team leader manages a group of enumerators and can conduct interviews him/herself. The team leader is responsible for:
- Logistics. The team leader is responsible for organizing the transport of teams, gathering materials, transporting paper instruments, and managing the technology.
- Permissions. Team leaders make contact with local authorities to introduce and explain the survey and get permission to work.
- Supervising. Team Leaders ensure the team arrives on time to the enumeration area and proceeds to oversee within-enumeration area selection of households.
- Correcting. Once interviewing has begun, team leaders should move between enumerators and check they are following protocols. Supervision should not, however, make subjects feel uncomfortable.
- Data Quality. Team Leaders check all questionnaires in the field and at the end of the day. If PDAs/tablets are used either the team leader or an RA will check data. The team leader ensures that data errors are fixed by revisiting respondents.

4.2 Oversight

Enumeration teams and field management can quite easily deviate from important protocols—these deviations can range from replacing sampled households based on the ease of getting respondents to creating fake data. In many cases, cutting corners is not easy to detect and can save money and time for the enumerators, field managers, and even the survey research firm. PDAs/tablets can reduce the number of total possible types of fraud, but some level of field supervision is always necessary. A parallel reporting structure, with independent oversight, can help guard against these deviations.

On the oversight side, there are two types of checks that should be conducted– audits and backchecks:

Auditors arrive at randomly selected villages on the days they are slated for enumeration, without advance warning to the team. When an auditor visits a team, they make sure the team is in the correct enumeration area, that they have sought consent from local leaders, complied with the household selection procedure, and that all team members are working. The auditor then tracks the performance of the team throughout the day— as they seek consent, build rapport, conduct within-enumeration area sampling, and survey respondents.
A backcheck consists of a revisit to a respondent who completed a survey not more than a few days prior. Using the data collected, they locate respondents and verify responses on a few key questions, for which the response was not likely to have changed (for example, age or household size) in the period since they were initially contacted. Backchecking can both help to identify enumerators who are not performing and establish an error rate. If phone numbers are being collected in the survey, this can be done more cheaply by telephone.

Conducting both audits and backchecks means that for each individual survey there is some non-zero probability that the work will be checked in some way. In the case that there are only backchecks, teams will never be monitored in terms of their adherence to protocols as they sample and conduct interviews. In the case that there are only audits, if a team is not visited on a particular day of work there is no chance to check that they actually interviewed subjects and recorded their responses accurately.

Auditors and backcheckers must report directly to survey management. Imagine an example: Say a village is difficult to find and the team of enumerators chooses a replacement (rather than resampling by the PIs), and the auditors visit the sampled village and uncover it was not surveyed. If this error is communicated to someone also managing the enumerators, their best response is to cover this up or try to fix it without the PI knowing. This prevents having to admit a management mistake, and having to add a day of work to revisit the original or resampled village. If the survey team is notified directly, there is an opportunity to fix the mistake and make personnel changes as needed. Unmonitored communication between auditing teams and enumeration teams can result in a lot of unauthorized fixes and unexplainable data patterns.

5 Recruit enumerators

5.1 Sources

If you are working alone, recruit experienced enumerators through contacts at survey firms, NGOs, or universities. It is important that enumerators are experienced, literate, educated, and able to build rapport with subjects. Hiring enumerators who are connected, in some way, with the survey leaders or local coordinator, e.g. through a youth organization or other social tie, can help immensely with oversight as the enumerators have bigger reputational costs if they shirk their duties.

5.2 Languages

The foremost requirement is that the enumerators speak the required local languages. We know that coethnicity between enumerators and subjects can reduce bias, so recruitment of coethnic interviewers, and balancing across the sample if using treatment and control groups, is important.

5.3 Gender parity

Having a team of mostly male enumerators interview a sample with equal numbers of men and women there can introduce response bias. For sensitive questions, such as questions on sexual behavior or violence, it is strongly recommended that women interview other women. If it is difficult to recruit experienced women enumerators, it usually makes sense to hold a special training for women candidates with less experience in order to ensure teams are balanced in the end.

6 You must pre-test!

There are several different and necessary phases of pre-testing:

The first pre-test is for instrument clarity and is typically conducted by the researcher(s) or the wider survey team. This step involves a thorough review of the instrument after all the questions have been written—the focus here is on order, clarity, and flow.
The second phase of pre-testing focuses on the translated version of the instrument; the key concern is that the questions are easily understood by native speakers while retaining the meaning intended by the researcher. This is best done by a group of translators or members of your survey team, as it often it takes several people to discuss and settle upon the widely understood meaning of words and phrases. If you are translating into multiple languages, it’s important to have people from every language represented for consistency, working together to ensure consistency across versions. Fellow enumerators can work here, but it is better to recruit people of varying education levels.
The third phase of pre-testing ensures that respondents understand the intended meaning of each question and are comfortable responding. The best way to get at this is to observe enumerators conduct practice surveys in the field. You’ll want to get as close as you can to the real sample for this test, incorporating different subgroups within the real sample, e.g. men and women, more and less educated. Instruct enumerators on how to identify and record issues, so that they can describe them after to the PI and survey manager, but also observe these practice interviews first-hand. Key concerns are clarity and length: respondents should not become confused, tired, unfocused, or disinterested. You want to understand how the questions are understood by respondents and if enumerators are changing questions at all during the survey. This step will also, importantly, give the project team a good idea of how long the average survey should take, so you can adjust expectations of how many surveys can be done per enumerator per day.

7 Training

7.1 Standards

Trainings establish consistent standards for data collection. If you are contracting a survey firm and are not on the ground yourself, training is the most important part of the process to personally attend. It’s a key moment to communicate quality standards, expectations, the intended meaning of each question, and teach important procedures that may be more technical than what the firm is used to, such as a list experiment. It is also a key moment to motivate the team, by communicating the project’s goals and importance. In order to ensure that the each member of the team is prepared to a certain standard, it is a good idea to test each team member at the end of the training period. There should be an expectation that some team members will be asked not to proceed any further with the project as a result of the test, which will emphasize the importance of taking training to heart and taking the test seriously.

7.2 Assessment

Trainings are a key moment for assessment as well. If you train teams together it is easy to spot management issues and leadership capabilities. A good practice is to train teams together, and select team leaders at the end of training—this gives you a few days to gauge skills and also incentivizes trainees to perform during the training.

7.3 Participation

Trainings set the tone for the rest of fieldwork. Beyond communicating standards and expectations, this is also a key moment to create a culture of participation. Encouraging trainees to speak out about issues with the survey can show that you are open to feedback and increase the chances that they will report adverse events or challenges during the actual data collection.

Training sections:

Protocol review, e.g. how to get permission to work in village/household, how to select respondent, etc.
Discussion of unforeseen contingencies, e.g. what happens when you get a refusal, when enumerators are targeted or threatened, when enumerators observe perverse reactions to or consequences of the survey
Question by question review
Interviewing techniques (rapport building, discussions)
Review translated instrument
Using a PDA/tablet and conducting the interview on a PDA/tablet
Using GPS
Group practice (enumerators interview each other and get feedback)
Field practice– at least one day of training (or more for complicated or long surveys) should be spent interviewing real people who are similar to the survey subjects
Certification exam

Training usually takes several more days than you expect. See below for a rough guide to realistic training schedules.

List of documents needed for training:

Instrument + translated instrument
Question-by-question guide
Manual with expectations, instructions for filling responses, tips on interviewing etc.
Protocol for sampling, consent, reporting, and unforeseen contingencies

Before being deployed to the field, each enumerator must:

Be able to correctly list, sample and interview individuals in the enumeration area
Understand their role
Understand and correctly follow interviewing protocols
Be informed about oversight procedures
Complete an IRB-approved module on human subjects protection

Data from mock surveys must be individually assessed and feedback given to each enumerator. You can check whether certain enumerators are entering data differently than their peers, for example by entering lots of “Don’t know” or “Refuse” answers, finding low prevalence of sensitive behaviors, or entering data that is logically inconsistent. However, there is a lot that you can’t tell from the data alone. Spending a lot of time observing enumerators while they run surveys can greatly increase the quality of the data by improving their training, allowing you to select the best enumerators more accurately, and allowing you to understand how the questions are being implemented in the field.

8 Employee incentives

To a large extent, the quality of the data collected is determined by the behavior the enumerators. Throughout the hiring, training, and field stages of the survey it is important to get the incentives right so that enumerators are motivated to do high quality work.

8.1 Expectations and Quality Control

Expectations should be laid out clearly during training, in a manual, and reiterated in clearly worded contracts (signed after training).

Basic expectations of enumerators:

Being on time
Adhering to within-enumeration area sampling and replacement scheme
Getting informed consent
Building rapport with subjects
Accurately recording responses
Communication with supervisors

8.2 Contracts and Payment

As much as possible, make payment dependent on delivery. Enumerators have less and less incentive to stick with the project towards the end of fieldwork. The marginal returns are lower and they may be concerned about finding new work. In order to offset this, it is good practice to withhold a portion of their total salary (+/- 30%) until the end of fieldwork, and sometimes until data has been thoroughly reviewed if using paper instruments that need to be entered manually. At the same time, enumerators are often living paycheck to paycheck and may have expenses to cover during their long absence in the field. It is important to pay an advance up front to allow enumerators to take care of personal expenses that may otherwise make them anxious and unhappy during fieldwork. Having a strong local manager who understands the enumerators financial situations can help you create incentives while still making sure that they perceive the compensation structure as fair and adequate.

8.3 Soft Incentives

Soft incentives help to keep teams happy and motivated throughout work. Some examples are:

Performance-based bonuses: Allowing managers to give performance-based bonuses for exceptional performance on a daily or weekly basis.
Certificates: Survey trainings often involve learning portable skills, like the use of tablets or PDAs. Certificates can help enumerators prove to new employers that that they have these skills.
Letters of Recommendation
Recommendations to other survey firms, NGOs, etc.
Wrap party

9 Map out your routes

Make route plans that set the order in which enumeration areas [EAs] will be covered, and then assign teams to routes. The basic idea is to minimize transportation costs while taking into account regional differences in languages or dialects that could mean that only some teams can work in certain areas.

A good way to visualize route plans is to mark sampled EAs on a map, using different colors to indicate different languages spoken in each EA. [SAMPLE PHOTO HERE]. Rely on team leaders to gauge travel time between sampled points and to suggest best routes.

10 Checking the data

Write code to clean data and check for errors, and run it on available data as soon as possible. This code should check for nonsensical responses or patterns, both within and across instruments (enumerator error).

If you are using tablets or PDAs, you should begin to run the cleaning and error script on the incoming data as soon as interviewing begins. If you are not using PDAs/tablets, you’ll need to develop a procedure for checking the paper instruments in the field (the job of the team leader), and also begin checking and cleaning of entered data as soon as possible.

10.1 Checking data entered from paper instruments

After interviewing the team leader needs to review all instruments for completeness and accuracy. If there are missing data or other inconsistencies, the team leader should send the enumerator back to revisit the respondent to correct all problems before leaving the area.

Once instruments are collected, data entry should commence as soon as possible. All data should be entered twice, and any discrepancies should be checked by a supervisor against the paper instrument.

10.2 Checking data gathered using PDAs or tablets

When using tablets or PDAs, checking the data is the responsibility of the RA and PIs. In addition to using a script that checks for patterns and outliers, it is also best practice to record selected portions of the interview and listen to a subsample of responses, both for errors and quality.

Footnotes

Thanks to Brandon de la Cuesta for help with this section↩︎