In this article, we share several suggestions on how to enable a more mindful approach to survey design. We have had the privilege of working with a variety of international partners in the past and wanted to share what we learned from these experiences. We have compiled eight ways that can help you design, administer, and analyze surveys that are relevant for your research and for the communities you work with. When compiling our suggestions, our objective was to ensure that these three ideas ran through all of them:
Respect for your local partners – Work with local partners every step of the way and adhere to a place-based approach.
Respect for your participants – Make sure that your surveys are accessible and safe for your participants.
Working together – Your participants and their communities are partners in research. Respect their time and knowledge rather than extracting from it.
Table of Contents
SURVEY DESIGN
Before we dive deeper into best practices for survey design, we want to mention that good survey design starts with diverse perspectives. A diverse perspective in the development stage of a survey comes from engaging with local partners, listening and learning from your participants, and working in diverse teams. This promotes diversity, inclusion, and fosters a more ethical and culturally sensitive survey.
1) Finding a representative sample
Be careful not to fall into the trap of surveying a certain group solely because they can easily be recruited to participate in your study. Excluding a population for any reason other than it is not appropriate for your research question is unethical. It is possible that your population of interest will be hard to reach and might require extra effort on your part to arrange for potential respondents to take your survey. Yet, it is vital to survey participants that are the most relevant for your research question. This will increase the generalizability and effectiveness of your study.
One of the best ways to recruit people that are in hard-to-reach places is through a local partner. Reach out to leaders in those communities, perhaps professors or community leaders, they might be able to help you recruit participants for your study through their network. One of the ways that you can connect with those community leaders is through social media. For example, a Facebook community page of your target population (e.g., a local youth environmental community group).
2) Check social norms and domestic laws
If you are investigating a population outside of your country or culture, be conscious of the fact that other people could have different social norms and domestic laws. Social hierarchies and gender roles could play much more important and bigger roles in the fabric of society than they do in your community. For example, in some countries, it is inappropriate for men to speak to unaccompanied women. There are also places where same-sex activity is illegal, in others it is legal but is condemned in society, thus, asking questions about sexual orientation might be legal but inappropriate, offensive, or can even harm the lives of participants who answer that question. Therefore, it is important to be mindful of social norms and domestic laws because disregarding them could cause harm, yet the first and foremost element of all survey design methods should not endanger participants and communities.
Also, in some countries, certain minority groups are not legally recognized by the country. An example of this is Rwanda, wherein an effort to heal from its history of genocide, the government since 2004 has enacted laws against genocide ideology and divisionism, and instead has aimed at unity and reconciliation among Rwandans. As such, it is illegal to ask questions about a person’s ethnicity or minority status. Social scientists conducting research with minority groups will need to be cognizant of these rules and their implications during their survey design process. Some of the ways that you can research social norms and domestic laws are:
Talking to locals - By talking to locals you can get valuable insight about your population of interest. They will be able to answer questions such as what is culturally appropriate to ask and what kind of topics to avoid.
Pre-testing – Once you pre-test your survey with your population of interest, you might discover that some questions do not make sense or are not culturally appropriate (we discuss this later in the blog). This is a great opportunity to communicate with your participants about how to improve your survey.
Survey design is an iterative process, but these iterations are far more useful when caught early in the process by talking to locals and pre-testing before the survey goes live. Once a survey is launched, you are limited with what can be changed. Taking more time upfront and meaningfully engaging with community members can lead to better outcomes for your survey work and fewer interpretation issues during analysis. However, it could happen that you find out about social norms or domestic laws only once the survey is launched:
In the field – For in-person surveys using enumerators, you might realize that a question is not phrased properly only once it is live. At this point, you cannot go back and change the question, but you can make a note of which question caused confusion or was not well received by your participants, so you can take that into account when you are analyzing your results. It may be possible to rephrase the question and use simpler words but be careful not to change the meaning of the question.
3) Get permission from authorities to conduct the survey
In some countries or large bureaucratic institutions, you might need to get permission from the authorities to conduct your study. Getting permission from the authorities makes conducting surveys a longer and more complex process. However, by adhering to cultural norms and respecting local authorities and customs, you will be able to carry out a more culturally sensitive survey and ensure that you are not endangering your participants. After doing so, you might find yourself in one of these two situations:
Participants might be forced to take the survey – In some cases, participants might be forced to take the survey by the authorities or they might feel pressured to do so. There could also be some sort of benefit within the organization (e.g., career advancement, rewards) if they take the survey. This means that your participants are giving you consent that is not entirely voluntary (check the informed consent section below).
For example, in Laos, surveys for youth need to be pre-approved by the Lao People’s Revolutionary Youth Union (LRPYU) so that the survey questions and results are not in conflict with their mission and mandate. In most cases, surveys aimed at understanding this community will need to be planned well ahead of time to have the proper approval and buy-in.
Participants might be discouraged from taking the survey - On the other hand, you might find yourself in a situation where the authorities actively discourage your participants from participating in your survey. In that case, you should inform the people of the views of the organization’s authorities and if they could face any consequences if they do decide to participate.
REFINE YOUR SURVEY
When you are refining your survey, be mindful of the context that you will administer it in. Given that every situation and community is unique, a big part of refining your survey should include working closely with your community of interest. This should include pre-testing your survey and learning from the recommendations of your community of interest on how you can improve your survey.
4) Choose the right words
One of the things to look out for when you are choosing the right words for your survey questions are binaries. Survey design has long used the idea of binary questions for demographic data, yet our understanding of identities is far more complex. Building in space for this complexity is key to capturing truly representative information with respondents. One of the ways to do that is to use open-ended questions and Likert scale questions in your survey. Pre-testing a survey is a key step in survey design. By testing your survey with real people within the partner community, you can identify design flaws that were not immediately apparent and provide you with the necessary insight to design a more culturally sensitive survey. Here are examples of issues you may discover that indicate that you need to re-word your questions:
Skipped questions – Skipped questions are one of the most telling indicators that your participants did not understand the question. People might feel embarrassed to admit that they do not understand what a question requires from them and simply skip it. This means that it is vital to use lay language, simple language that the general population is likely to understand when designing your survey. When questions are missed, not only will you as a researcher not get the information that you need, but you might also be forcing your participants to give uninformed consent (see below). For reference, only 20% of adult Canadians read at a college level and when people are ill or stressed, their literacy level drops even lower.
A lot of “I don’t know” or “neither agree nor disagree” answers – If your survey has an “I don’t know” option, your participants are probably going to choose that if they do not know an answer to your question. However, if you did not include this in your survey your participants might be choosing “neither agree nor disagree,” as it most closely resembles “I don’t know.”
A lot of “Not applicable” or “NA” answers - If participants feel or believe that the question does not apply to them, they might be choosing the “Not applicable” or “NA” options. This might indicate to you that the way that you have worded the question does not signal to your participant that it is relevant to them.
Lost in Translation:
In a survey we conducted on climate adaptation in the coastal region of Bangladesh, we asked participants “What’s one perceived threat to successful climate policies.” The responses we got were startling at first. Participants wrote “unconscious people” in the survey responses. In meeting with our enumerator, I had to check to see if we were interviewing people who are were not physically and mentally able to give consent. This was a case of meanings lost in translation. What the participants meant to say was that the biggest threat to successful climate policies were people who are not educated to the effects of climate change, or people who are ignoring it.
The reason why we brought up this example is to showcase why you must be careful when you are contextualizing your survey with language norms and translations. Simple language can go both ways and different things can be lost in translation.
5) Understand when “Neither Agree nor Disagree” is a Genuine Answer
It is important to note that you might also be genuinely getting a lot of “Neither Agree nor Disagree” responses in your survey. So be mindful of the possibility that these responses could also be honest answers from your participants. One of the ways to identify whether your participants genuinely “neither agree nor disagree” with a statement, or if they are trying to tell you that they do not understand the question, is how quickly they answer your survey. Watch out for:
Abnormally long answering times – If you have the option of tracking how long participants spend on a particular question, watch out for long hesitation times. If participants look stuck on a question, this might be an indicator that the question is too complex, confusing, and needs rewording. However, this could also mean that your participants are simply distracted by something else.
Abnormally short answering times - If you think that your participants are answering your survey too quickly, it may be because they do not understand the questions. They might also not have a genuine interest in answering properly and honestly. They could be in a rush, forced to take the survey, or might not recognize any benefit of taking the survey to them.
In addition to unearthing problems with individual questions, pre-testing also helps you improve your survey through debriefing your participants. During this stage, you can discuss the survey with a small number of participants before administering it on a larger scale. These debriefing sessions are an opportunity to ask your participants about:
Question clarity
Length of the survey
Sensitivity and relevance of your questions
By pre-testing your survey and then debriefing your participants you also get the opportunity to:
Test question styles - If you have asked the same question in two different ways (e.g., a Likert scale question or an open-ended question), you can also ask your participants which version of the question was easier to answer during debriefing. You may discover that your participants strongly preferred one style of a question to another, thus, giving you insight to design a survey that is better suited for a given population.
IMPLEMENT YOUR SURVEY
When you are implementing your design and actively administering your survey, remember that you are working with people. Get informed consent, work with available infrastructure, and above all, remain empathetic and respectful throughout the process.
6) Get Truly Informed Consent
It comes as no surprise that obtaining ongoing, informed consent is crucial to conducting an ethically sound and culturally sensitive survey. Ensuring that you understand the process and ongoing nature of this relationship with participants is an important element of fostering ethical and safe experiences for participants. This means that you must provide as much information for your participants as they need so they can make a fully informed decision whether to participate in your survey or not. You should also provide the objective of the survey and what benefits the participants may receive by completing it. Without this information, participants might not feel the need to do the survey or feel that they are safe to participate in it. Some of the ways that you can ensure that your participants are giving you informed consent and that they fully understand what taking part in your survey entails are:
Allow participants to skip questions if they are feeling uncomfortable answering them.
Make sure that your participants know that they can withdraw from the survey at any time and write that into your survey (e.g., make sure that this is worded clearly and that there is an option to opt-out on every page).
Explain what the survey is about, why you are conducting it, what type of questions are going to be asked, if you will ask questions about sensitive topics, a privacy statement, and if there are any risks to the participants.
Share the time and date that the survey will be completed and available.
Remind them that their identity is kept confidential, and clearly mention if otherwise.
Provide details on how the results will be collected and assessed.
If or how the results will improve the livelihoods of people that are participating in your survey.
Give your participants opportunities to raise concerns and address those concerns accordingly.
Also, be mindful of these issues that could compromise the voluntary consent of your participants:
Monetary Incentives - Incentives are a great way to encourage more people to take part in your survey but be wary of offering large sums of money. Not only could that incentivize people to take a part in your survey for the wrong reasons, but it might also put pressure on people that are financially disadvantaged to participate in your survey. This could lead to risks for them participating that they might be willing to overlook due to the monetary reward.
Social Incentives - Avoid creating an impression that it is “cool” to participate in your study or that participating can in any way enhance someone’s social image. For example, if a student feels (and this might not be the reality) that participating in a professor's survey can enhance their grade or on the contrary, their studies will suffer if they do not participate, their consent will be compromised.
Ability to consent - Some participants may not be able to consent due to limited or impaired cognitive abilities as they will not be able to make a fully informed decision whether to participate in your survey or not. It is possible that by law, these people will be allowed to consent to participate in your survey. Yet you should consider whether these people can:
- See the risks and the benefits of them participating in this survey.
- Understand how the conditions of the survey affect them (e.g., how much time will be required from them, the difficulty of the survey).
If the people that you want to include in your survey cannot provide informed consent (e.g., children), the next step is to seek permission from an authorized third party (e.g., a parent or guardian).
7) Work around internet availability
If your population of interest does not have reliable access to the internet, you will have to explore other ways of getting them to take your survey. Some of the ways that you can do this are:
Carry out your survey in-person – To carry out a survey in-person in a hard-to-reach population you will need to rely on local partners. Find trusted enumerators that are experienced in socio-demographic research techniques that will deliver your survey to the right people.
Telephone survey - If your population of interest does not have access to the internet, you still might be able to conduct your survey remotely by phone. While it is a great option if you do not want to or cannot travel to the area where your population of interest is, sometimes the fastest way to get your results is to administer your survey in person.
Instant messaging software – In many of the communities Kai Analytics has worked in, Facebook messenger was the DeFacto way to communicate. Even with spotty internet connectivity, we found out the farmers we interacted with are still able to asynchronously communicate back and forth. Ensuring that you have the data privacy measures in place and the appropriate messaging software for the country context you are working in are important considerations.
Alternative forms of survey delivery can be very helpful if delivering a survey through email is not an option. To decide which delivery method is the most appropriate, think of the context you or your partners will be working in.
8) Demonstrate empathy
Demonstrate empathy with the way you design your survey, administer it, and debrief your participants. Actively listen to what your participants have to say and make them feel safe and welcome to express their ideas at any point. If you end up travelling to provide aid or go on field missions, remember that it is not your mandate to instruct or impose your own beliefs on the people you are meant to help. You are there to gather information that may be beneficial to them. Avoid having them think that you consider people that participate in your surveys inferior to you in any way.
Actively listen to what your participants have to say. Whoever your prospective participants might be, remember that behind the survey responses that you will get are real people. Act according to the principle of Respect for Persons (Read the Belmont Report | HHS.gov). The main idea of this principle is that it is unacceptable to treat individuals solely as means (objects or things) to an end (a research goal). If you are asking questions about events such as natural disasters or extreme climate conditions, be mindful of the fact that the people that are responding to your surveys might have suffered immeasurable losses due to these events. These topics can be sensitive and hard to talk about for some of your participants. A participant’s welfare comes above all else in human research.
Is there something that we missed? While we are still learning, it is important to open dialogue on better approaches to survey design. In the future, we want to explore:
how to market your survey,
accessibility, and
how to make surveys more accommodating to people with hearing, visual, and mobility challenges.
Let's learn together, email us at hello@kaianalytics.com and we would love to continue the discussion. Not one person has all the answers but together we can build a more robust approach to research!
If you want to learn about how to analyze your survey with text analysis, check out our article on the subject.
For further reading, check out the following resources:
Home (sshrc-crsh.gc.ca) (CAN)
P.S. We want to say a big thank you to Johanna Wilkes and Stephany Carrillo for their review and feedback on this blog. Johanna is a PhD candidate in the Global Environmental Stream at the Balsillie School of International Affairs. Stephany is an international civil servant.
Kai Analytics helps non-governmental organizations (NGOs) and international non-profit organizations (NPOs) conduct effective global surveys, collecting and analyzing insights from disadvantaged and marginalized communities in Asia and Sub-Saharan Africa to give a real voice to policy. Learn more about our services and the countries we serve here International Development | Kai Analytics