This is the first blog in the series of ‘Best Practices in Crafting Effective Surveys, Data Analysis and Visualization’.
Asking the right questions that prod the respondent to think in the direction you want them to think in, is both a science and an art. The manner in which a survey questionnaire is designed, as well as the choice and order of words can make a significant difference. ‘The devil is in the details’, so in order to derive details and cull out actionable insights, the survey writer has to play the role of ‘a devil’s advocate’ and make an effort to gauge the respondent’s psyche, to design a questionnaire that alludes to attaining the desired research objectives. One has to be mindful of the fact that meaningful research is a factor of apt questions and great answer choices. Hence, the fruits you will reap (as answers) will largely depend upon the seeds you will sow (as questions).
A short, clear, concise and focused survey questionnaire will elicit greater responses, not only in terms of numbers but quality as well. Enumerated below is an indicative list of 15-points that could come handy while designing survey questions:
- Two-pronged approach - Survey questions need to fulfill dual goals – while the research objectives of the interviewer/surveyor should be attainable, supported by detailed analysis and interpretation, the questionnaire should also be user-friendly, brief, inclusive, relevant and one that provide accurate information to motivate the respondent to complete it.
In practice, the first page of the survey should include the company name, logo and survey title, followed by an introductory paragraph delineating the key objectives/expected outcomes, process followed in conducting the survey, target audience, average time taken to complete the survey, incentives offered for participants etc. This can be followed by a statement assuring complete confidentiality of data and thanking participants for their responses.
- Value the respondents’ time and effort - Highlight the fact that you appreciate the participants’ contribution to the survey and cherish their opinions. Also, accentuate how their views will not only help the organization improve its products/services, but at the same time yield actionable outcomes for the respondents/their organizations. Further, before specifying the average time taken to complete the survey to your target respondents, request some colleagues who are unversed with the survey to take it, and time their responses. This would give a fair idea of the usual time required to fill in the survey.
- Keep your Eyes on the Prize - Design the questions keeping the output in mind. This will help you scope the survey better and obtain desired outcomes, while keeping the ‘main message’ at the heart of the survey.
- Numbers Don't Lie - While designing a survey questionnaire, the writer’s aim should be to metricise the survey as much as possible. This will not only helping in gaining actionable insights by standardizing responses on preset parameters, but is also likely to yield a higher response rate.
- Baby Steps - Divide the entire survey into sub-sections. Breaking the survey down into smaller parts ensures that respondents do not feel overwhelmed by the length of the survey and it is easily comprehensible. Moreover, it also enables ease and granularity of analysis, as smaller themes feed into the overall key messages/takeaways derived from the output.
- Funnel Method - Organize the questionnaire in a manner wherein general, broad company information questions are at the top, followed by key demographic and classification questions. These are low-hanging fruits, and get the respondent warmed-up for upcoming critical questions. Next, delve into the key questions centred around the main theme of the survey, such as questions around structure, processes, challenges etc. and tail into result-oriented questions such as key success factors (KSFs), outcomes, major metrics tracked etc. to end the survey.
- Strike a Balance - Ensure there is a good mix of qualitative and quantitative questions in the questionnaire. However, limit the number of open-ended questions as deriving actionable insights from those is a tedious task. In fact, attempt to give some indicative answer options for qualitative questions as well. This will stimulate respondents to answer the question and give them a starting point to start thinking in the desired direction. If the survey platform allows, you can always add a ‘Comments’ box in the end to gain additional insights. For instance,
Q1. What were the key reasons for voluntary attrition in your organization?
- Moving to a better position elsewhere
- Limited career growth opportunities
- Lack of cultural 'fitment'
- Compensation-related issues
- Discord with the top-team over the vision/direction of the company
- Personal issues
- Others (please specify in the Comments box below)
- Close the Loop - Provide mutually exclusive ranges/intervals as choices for questions with answers in number/percentage format, such as questions on revenue, workforce, market cap etc. Participants are more likely to answer questions that offer ranges rather than type in a number, especially while sharing information deemed confidential. For example,
Q2. What was the total revenue of your company (pertaining to India operations) in 2020-21*? Please note that the figures mentioned below are in INR crores.
- <500
- 500-1,000
- 1,000-2,500
- 2,500-5,000
- 5,000-10,000
- 10,000-25,000
- 25,000
*If your organization follows a financial year other than April to March, please interpret 2020-21 to be your last completed financial year.
- Explain terms and avoid suggestive words - Don’t assume that the respondent knows what you mean. Clarify terms/phrases wherever there is the slightest scope of different interpretations and specify that these definitions are only valid for the purpose of that particular survey. These definitions can be provided as a footnote at the end of the question they pertain to (refer to the footnote mentioned after Q2. above). Moreover, to gather unbiased results, it is imperative to avoid the use of leading words/phrases such as should do, must act, will ban etc.
- Mix It Up - Have a good mix of varied question types like radio button (single answer, ‘yes/no’), multiple-choice checkbox, slider, ranking, dropdown, rating, textbox, matrix etc. format questions which break the monotony.
- Follow It Up - Use display logic/piping to gather specific details and deep-dive into a particular area. Instead of one long question, ask multiple short follow-on questions.
For example, after a ‘Yes/No’ type question, you can ask a follow-up question on the reasons for respondents saying ‘Yes’ or ‘No’, and can further ask them to rate those reasons is ascending/descending order of importance in a separate follow-up question.
- Utilize Balanced Scales - Rating scale questions should ideally carry a scale with odd-number of choices, such as 3, 5 or 7 options. Further, scale points should correspond to conceptually equidistant points rendering a balanced scale. At the time of analysis, this would ensure apt reading of the scale and aid statistical computations such as mean, median, percentiles etc.
For example, a 5-point scale can be annotated as ‘Extremely Satisfied, Somewhat Satisfied, Neutral, Somewhat Dissatisfied, Extremely Dissatisfied’. Another instance could be the following:
- Preset Answer Formats - Some survey platforms allow setting the format of answer options to text, percentage, numeric etc. This is a good way to standardize responses.
- Capture Key Metrics - Include questions on key fundamental metrics like revenue, workforce, ownership type, domain-industry/services, orientation-B2B/B2C/Both, sector etc. that will help in classifying the data by various cuts at the time of analysis. Further, in case there are some key industry- or function-specific metrics, it would be worthwhile to gather those too.
Illustratively, demographics such as employee age, tenure with the company, gender etc. and compensation data, attrition rate, increments etc. are some of the key metrics tracked by the HR community.
- Check the ‘look and feel’ of the survey - The survey layout should be consistent in terms of font type and size, font colours etc. and should be compatible with not only desktops/laptops, but also handheld devices such as tabs and mobiles, to enable respondents to fill the survey on-the-go.
Similar to the famous quote, ‘Customer is King’, in case of surveys, ‘Respondents are Kings’. Their responses will ‘make or break’ the survey. Thus, a well-drafted questionnaire is a key ingredient feeding into incisive research outcomes.
To know more about the different types of survey questions, guidelines for data analysis/interpretation and techniques of data presentation, watch for the next blog in this series.
