library / lib332f223040366190
Survey & Questionnaire Design: Collecting Primary Data to Answer Research Questions
Jane Bourke, Ann Kirby, Justin Doran · 2016
In a sentence
A practice-oriented guide to designing surveys and questionnaires that collect valid, reliable primary data to answer well-formulated research questions.
This concise, practice-focused ebook walks researchers, students, and managers through the full arc of survey design—from formulating a sharp research question and hypothesis, through writing clear factual and non-factual questions, choosing appropriate levels of measurement, coding responses, and ensuring validity and reliability, to laying out a self-completion questionnaire. Drawing on the authors' years of teaching survey methods at University College Cork and grounded in established survey-methodology literature (Fowler, Dillman, Sudman & Bradburn, Oppenheim), it pairs concrete examples, exercises, video links, and a sample small-business questionnaire to help readers avoid the common pitfalls that produce confused respondents and useless data. If you want to collect primary data that actually answers your question, this book gives you the practical decision rules to do it.
The four lenses
- Science
- Statistics
- Systems
- Strategy
Tags
The model
A framework linking survey design levers (research question clarity, question wording, measurement choices, coding, layout, testing) through respondent cognitive states (comprehension, ability, willingness) to data quality outcomes (validity, reliability, response rate).
Research Question Claritydesign lever
The degree to which the research question, hypothesis, and objectives are clearly formulated, specific, and well-aligned so they can guide the design and types of questions, the sample, and the data needed for the survey.
Question Wording Qualitydesign lever
The extent to which individual survey questions are short, simple, standardised, free of jargon, double-barrels, leading, negative, assumptive, and sensitive wording, so each respondent experiences the same clear question.
Level of Measurement Choicedesign lever
The choice of nominal, ordinal, interval, or ratio scale for each question, which determines the precision of data captured, the analysis available, and the difficulty respondents face in providing the information.
Coding Qualitydesign lever
The rigour with which responses are translated into numbers using mutually exclusive, unambiguous, consistently applied codes, with appropriate pre-coding, post-coding, missing-data codes, and logic/range checks to prevent transcription and coding decision errors.
Questionnaire Layout Qualitydesign lever
The quality of the questionnaire's physical and visual organisation—cover letter, clear instructions, good first question, meaningful order, consistent answer spaces, white space, and avoidance of awkward formats—that motivates respondents and eases completion.
Instrument Pre-testing and Field-testingdesign lever
The extent of pre-testing (expert reviews, focus groups, cognitive testing, interviewer debriefing, observational interviews, behaviour coding) and field-testing (pilot tests, dress rehearsals) used to detect problems and minimise measurement error before full distribution.
Respondent Comprehensionpsychological state
The degree to which a respondent understands the question being asked in the way the researcher intended, reflecting the first golden rule that wording, vocabulary, and clarity allow a consistent interpretation across respondents.
Respondent Ability to Answerpsychological state
The respondent's capacity to provide an accurate answer, affected by task difficulty, cognitive ability, whether they actually know the information, the reference period, and ability to recall information from memory.
Respondent Willingness to Answerpsychological state
The respondent's willingness to give correct and valid answers, reduced by sensitive or prestige-biased topics such as income, weight, alcohol or drug use, and increased by salient, non-threatening question framing.
Data Validityoutcome metric
The extent to which the answers obtained correspond to what they are intended to measure, encompassing content, face, criterion, and construct validity so the instrument measures the right concept for the research hypothesis.
Data Reliabilityoutcome metric
The accuracy or precision of the measuring instrument—its capacity to extract the same response from respondents with similar characteristics—assessed via test-retest, internal consistency, alternative form, and split ballot methods.
Response Rateoutcome metric
The proportion of intended respondents who complete and return the questionnaire, influenced by question length, complexity, sensitivity, layout, motivation, and the balance between multiple short questions and long complex ones.
How they connect
- research question clarity → influences question wording quality
- research question clarity → influences measurement level choice
- question wording quality → predicts respondent comprehension
- question wording quality → predicts respondent willingness to answer
- measurement level choice → influences respondent ability to answer
- respondent comprehension → predicts data reliability
- respondent comprehension → mediates data validity
- respondent ability to answer → predicts data validity
- respondent willingness to answer → predicts data validity
- coding quality → predicts data reliability
- questionnaire layout quality → predicts response rate
- question wording quality → influences response rate
- instrument testing → predicts data validity
- instrument testing → predicts data reliability
The process
This playbook outlines a systematic approach to survey research, beginning with the foundational design of the data collection instrument and proceeding through data preparation and validation. The process starts with designing a clear, engaging, and logically structured questionnaire to maximize response rates and data quality. Once the survey is designed, the playbook emphasizes the importance of understanding and identifying the correct levels of measurement for each data point, a critical step that informs both question construction and subsequent analysis. After data is collected, the focus shifts to translating responses into a usable format through a structured coding process, which handles both simple closed-ended questions and more complex open-ended responses. The final stage is a rigorous quality control process, involving multiple types of error checks to clean the coded data, ensuring the final dataset is accurate, reliable, and ready for valid statistical analysis.
Designing a Questionnaire Layout
To enhance the quality and reliability of data collected through surveys by optimizing the layout and structure of the questionnaire.
When to use: During the research instrument design phase, before a survey is administered to respondents.
Step 1Set clear objectives for the information to be collected.
Entry: Research goals have been defined.
Exit: A clear list of information requirements is documented.
In: Research objectives · Out: Information collection objectives
ch09
Step 2Prepare a reliable survey instrument with carefully constructed questions.
Entry: Information objectives are set.
Exit: A draft of all survey questions is complete.
In: Information collection objectives · Out: Draft questionnaire
ch09
Step 3Group related questions logically and sequence them to maintain respondent engagement.
Entry: Draft questions are written.
Exit: Questions are ordered logically within the questionnaire.
- Deciding on the sequence of questions based on their sensitivity or complexity.
In: Draft questionnaire · Out: Sequenced questionnaire draft
ch09
Step 4Design the visual layout to be clean and engaging.
Entry: Questions are sequenced.
Exit: A formatted questionnaire layout is created.
In: Sequenced questionnaire draft · Out: Formatted questionnaire layout
ch09
Step 5Test the questionnaire layout with a small pilot group.
Entry: A complete draft of the questionnaire is ready.
Exit: Feedback from the pilot group is collected.
In: Formatted questionnaire layout · Out: Pilot test feedback
ch09
Step 6Revise the layout based on feedback from pilot testing.
Entry: Pilot test feedback has been analyzed.
Exit: A final, revised questionnaire is ready for deployment.
In: Pilot test feedback · Out: Final questionnaire
ch09
Levels of Measurement Identification
To categorize survey data into appropriate measurement levels (Nominal, Ordinal, Interval, Ratio) for accurate data interpretation and selection of statistical analysis methods.
When to use: During questionnaire design to formulate questions correctly and before data analysis to select appropriate statistical tests.
Step 1Review each survey variable and its potential responses.
Entry: A draft of the survey questions or collected survey data is available.
Exit: All variables for categorization are identified.
In: Survey questions, Survey responses · Out: List of variables to categorize
ch06
Step 2Determine the appropriate level of measurement for each variable.
Entry: Variables are identified.
Exit: Each variable is assigned a measurement level.
- Deciding if data is categorical (Nominal), ranked (Ordinal), or has meaningful intervals (Interval/Ratio).
In: List of variables to categorize, Definitions of measurement levels · Out: Categorized variables
ch06
Step 3Document the measurement level for each variable.
Entry: All variables have been assigned a measurement level.
Exit: A complete record of variable measurement levels is created.
In: Categorized variables · Out: Updated codebook or data dictionary
ch06
Coding Process for Survey Responses
To systematically translate qualitative and categorical survey responses into a numerical format suitable for quantitative statistical analysis.
When to use: After data has been collected and before statistical analysis begins.
Step 1Prepare a codebook defining each variable and its corresponding numerical codes.
Entry: Survey data has been collected.
Exit: A draft codebook is created.
In: Final questionnaire, Collected survey responses · Out: Codebook
ch06
Step 2Assign codes to closed-ended questions.
Entry: Codebook is prepared and closed-ended responses are available.
Exit: All closed-ended responses are numerically coded.
In: Survey responses, Codebook · Out: Partially coded dataset
ch06
Step 3Develop a coding scheme for open-ended questions.
Entry: Open-ended responses are available for review.
Exit: A set of thematic codes for open-ended responses is defined in the codebook.
In: Open-ended survey responses · Out: Coding scheme for open-ended data
ch06
Step 4Apply the coding scheme to all open-ended responses.
Entry: Coding scheme is finalized.
Exit: All open-ended responses are numerically coded.
In: Open-ended survey responses, Coding scheme · Out: Fully coded dataset
ch06
Error Prevention and Cleaning in Coded Data
To identify, minimize, and mitigate errors during and after the coding of survey responses to ensure the integrity and reliability of the final dataset.
When to use: After the initial data coding is complete and before conducting the main statistical analysis.
Step 1Train coders on all coding guidelines and rules.
Entry: A codebook and coding process are defined.
Exit: Coders have demonstrated understanding of the coding rules.
In: Codebook, Coding guidelines · Out: Trained coders
ch06
Step 2Conduct valid range checks on numeric data.
Entry: Data has been coded into a numerical format.
Exit: All data points are confirmed to be within valid ranges, or outliers are flagged.
In: Coded dataset · Out: List of out-of-range errors
ch06
Step 3Apply filter and logic checks to verify response coherence.
Entry: Range checks are complete.
Exit: All logical inconsistencies are identified and flagged for review.
In: Coded dataset · Out: List of logical errors
ch06
Step 4Inspect a sample of coded data for accuracy.
Entry: A significant portion of data has been coded.
Exit: The error rate of the sample is within acceptable limits.
In: Coded dataset · Out: Quality assessment report
ch06
Step 5Correct identified errors and document all changes.
Entry: Errors have been identified and flagged.
Exit: A clean, validated dataset is ready for analysis.
In: List of out-of-range errors, List of logical errors · Out: Cleaned dataset, Data cleaning log
ch06
A candidate measure
Survey & Questionnaire Design: Collecting Primary Data to Answer Research Questions — derived measurement candidates
Research Question Clarity
expert clarity rating; checklist compliance score
self-report suitability: medium
Question Wording Quality
guideline-violation count; expert quality rating
self-report suitability: low
Level of Measurement Choice
per-item scale classification
self-report suitability: low
Coding Quality
coding error rate; inter-coder agreement
self-report suitability: none
Questionnaire Layout Quality
layout checklist score
self-report suitability: low
Instrument Pre-testing and Field-testing
number of testing methods used; revisions made
self-report suitability: medium
Respondent Comprehension
cognitive-interview comprehension coding; misinterpretation frequency
self-report suitability: medium
Respondent Ability to Answer
item non-response rate; response latency
self-report suitability: medium
Respondent Willingness to Answer
sensitive-item non-response rate; refusal rate
self-report suitability: low
Data Validity
criterion correlation; content coverage rating
self-report suitability: none
Data Reliability
test-retest correlation; internal consistency
self-report suitability: none
Response Rate
completed returns / distributed
self-report suitability: none
The story
The reader A researcher, student, or manager who wants to collect primary data that genuinely answers their research question.
External problem
They need to design a survey that produces valid, reliable, analysable data.
Internal problem
They feel uncertain and overwhelmed about whether their questions are good enough and fear collecting useless data.
Philosophical problem
It's wrong to assume anyone with common sense can write a questionnaire—poor design wastes effort and misleads.
The plan
- Formulate and clarify your research question, hypothesis, and objectives.
- Review the literature and choose a deductive or inductive approach.
- Design clear, standardised factual and non-factual questions.
- Choose appropriate levels of measurement and code responses systematically.
- Test the instrument for validity and reliability.
- Lay out the questionnaire for clarity and high response.
Success
- A valid, reliable questionnaire that yields analysable data and answers the research question.
At stake
- Confused respondents, unreliable answers, low response rates, and data that cannot answer the research question.
Chapter by chapter
ch05FACTUAL & NON-FACTUAL QUESTIONS
This chapter explores the distinction between factual and non-factual questions and their implications in measurement and gathering data, arguing that the type of question posed can drastically shape the outcomes and interpretations in research.
ch06LEVELS OF MEASUREMENT & CODING
This chapter elucidates the four levels of measurement in research—nominal, ordinal, interval, and ratio—and elucidates the coding process necessary for translating survey responses into quantitative data.
ch09QUESTIONNAIRE LAYOUT
This chapter delves into the crucial elements of designing a questionnaire, emphasizing how proper layout can significantly impact data collection and, ultimately, research success.
Questions this book answers
- How do you formulate and clarify a research question, hypothesis, and objectives?
- How do you write survey questions respondents can comprehend, answer, and are willing to answer?
- What is the difference between factual and non-factual questions and how do you design each?
- What are the levels of measurement and how do they shape coding and analysis?
- How do you code survey responses for statistical analysis?
Glossary
- Research Question Clarity
- The clarity, specificity, and alignment of the research question, hypothesis, and objectives that guide survey design decisions.
- Question Wording Quality
- The degree to which individual questions are short, simple, standardised, and free of design flaws.
- Level of Measurement Choice
- The scale type (nominal, ordinal, interval, ratio) selected for each question.
- Coding Quality
- The rigour of translating responses into unambiguous, mutually exclusive numeric codes with appropriate checks.
- Questionnaire Layout Quality
- The quality of visual and structural organisation of the questionnaire that aids completion and motivation.
- Instrument Pre-testing and Field-testing
- The extent of pre-testing and field-testing activities performed to minimise measurement error.
- Respondent Comprehension
- The degree to which a respondent understands the question as the researcher intended.
- Respondent Ability to Answer
- The respondent's capacity to provide an accurate answer given knowledge, recall, and task difficulty.
Related in the library