library / lib9fb7870a8da6e47b
Survey Research Methods - Fowler
In a sentence
A practical guide to the principles and procedures of survey research, focusing on identifying and minimizing various sources of error to produce high-quality statistical descriptions of populations.
This book is an accessible and comprehensive guide to the practice of survey research for social scientists, students, and anyone who needs to collect, analyze, or interpret survey data. It demystifies the entire survey process, from sampling and question design to data collection and analysis, by focusing on a "total survey design" perspective. This approach emphasizes that the quality of survey data is only as strong as its weakest link and teaches readers how to identify and manage the various sources of error—sampling error, nonresponse bias, and measurement error—that can compromise results. Whether you're conducting interviews in person, by phone, or using mail or internet surveys, this text provides the essential principles and practical procedures needed to produce credible, accurate, and defensible statistics about any population.
The four lenses
- Science
- Statistics
- Systems
- Strategy
Tags
The model
This model, inferred from the book's 'Total Survey Design' perspective, illustrates how a set of core survey design choices and procedures directly influence the quality of sample representativeness and measurement, which in turn determine the final accuracy, precision, and credibility of survey data.
Sampling Method Qualitydesign lever
The rigor of the methodological choices related to sampling, including the comprehensiveness of the sample frame, the use of probability selection methods, the specific sample design (e.g., stratified, clustered), and the adequacy of the final sample size for the study's analytical goals.
Nonresponse Procedure Qualitydesign lever
The quality and intensity of procedures used to minimize nonresponse and to understand its potential for bias. This includes the number and timing of contact attempts, use of incentives, interviewer training on refusal conversion, and efforts to gather data about nonrespondents.
Data Collection Mode Choicedesign lever
The researcher's selection of the primary data collection method (e.g., in-person interview, telephone, mail, internet, or a mixed-mode strategy), which involves trade-offs between cost, sampling coverage, response rates, and potential measurement effects.
Questionnaire Design Rigordesign lever
The degree of systematic effort invested in developing, evaluating, and pre-testing survey questions to ensure they are reliable and valid. This includes defining objectives, using cognitive interviews, behavior coding, and field pre-testing.
Interviewer Management Qualitydesign lever
The quality of the systems for recruiting, training, supervising, and monitoring interviewers to ensure they perform their roles (enlisting cooperation, asking questions, probing) in a standardized, non-biasing manner. Null if no interviewers are used.
Sample Representativenessbehavioral pattern
The degree to which the final set of respondents from which data is collected accurately mirrors the characteristics of the target population. This is a function of both the initial sample frame coverage and the final rate and nature of nonresponse.
Measurement Qualitypsychological state
The degree to which the answers obtained from respondents are reliable (consistent across comparable situations) and valid (correspond to the true values being measured). This is influenced by question design, data collection mode, and interviewer performance.
Survey Data Accuracyoutcome metric
The degree to which survey estimates correspond to the true values of the population. Accuracy is inversely related to Total Survey Error, which is the sum of all systematic biases (e.g., coverage error, nonresponse error, measurement bias).
Survey Data Precisionoutcome metric
The level of random variability in survey estimates, primarily determined by sample size and sample design features like clustering. A more precise estimate has a smaller standard error or confidence interval, indicating less random sampling error.
Survey Credibilityoutcome metric
The perceived trustworthiness and defensibility of the survey results among users of the data. Credibility is a function of the transparency and rigor of the methods employed, as well as the final accuracy and precision of the data produced.
How they connect
- sampling method quality → influences survey data precision
- sampling method quality → influences sample representativeness
- nonresponse procedure quality → influences sample representativeness
- questionnaire design rigor → influences measurement quality
- interviewer management quality → influences measurement quality
- interviewer management quality → influences sample representativeness
- data collection mode choice → moderates sample representativeness
- data collection mode choice → moderates measurement quality
- sample representativeness → influences survey data accuracy
- measurement quality → influences survey data accuracy
- survey data precision → influences survey credibility
- survey data accuracy → influences survey credibility
A candidate measure
Survey Research Methods - Fowler — derived measurement candidates
Sampling Method Quality
Use of probability sampling (binary: yes/no); Stated coverage rate of the sample frame (percentage); Use of stratification or clustering (categorical); Final sample size (count)
self-report suitability: none
Nonresponse Procedure Quality
Minimum/maximum number of contact attempts per case; Use of incentives (binary: yes/no; or amount); Use of refusal conversion attempts (binary: yes/no); Execution of a nonresponse bias study (binary: yes/no)
self-report suitability: none
Data Collection Mode Choice
Primary data collection mode (categorical: mail, phone, web, in-person); Use of a mixed-mode design (binary: yes/no); Sequence of modes in a mixed-mode design
self-report suitability: none
Questionnaire Design Rigor
Documented use of focus groups (binary: yes/no); Documented use of cognitive interviews (binary: yes/no); Documented use of behavior coding of pretest interviews (binary: yes/no); Documented use of a field pretest (binary: yes/no)
self-report suitability: none
Interviewer Management Quality
Number of hours of initial training; Use of supervised practice interviewing (binary: yes/no); Systematic monitoring of interviews during production (binary: yes/no); Supervisor-to-interviewer ratio
self-report suitability: none
Sample Representativeness
Response rate (calculated using AAPOR standards); Contact rate; Refusal rate; Comparison of respondent demographics to population benchmarks (e.g., chi-square test)
self-report suitability: none
Measurement Quality
Item nonresponse rates for key questions; Reliability coefficients (e.g., Cronbach's alpha) for multi-item scales; Frequency of problematic interviewer-respondent interactions from behavior coding; Correlations with other variables in expected directions (construct validity)
self-report suitability: none
Survey Data Accuracy
The absolute or relative difference between the survey estimate and a benchmark value.; Bias ratio (estimate of bias divided by standard error).
self-report suitability: none
Survey Data Precision
Standard error of the mean or proportion; Width of the 95% confidence interval; Design effect (DEFF) for key estimates
self-report suitability: none
Survey Credibility
Adherence to AAPOR Transparency Initiative disclosure standards (checklist); Publication of results in a peer-reviewed journal (binary: yes/no); Number of citations to the study
self-report suitability: none
The story
The reader Social science researchers, students, market analysts, or public policy professionals who need to produce or understand reliable quantitative descriptions of a population. They want to conduct credible surveys and make accurate statistical claims, but they are often unsure how to navigate the complex methodological choices involved.
External problem
Conducting a survey that produces inaccurate, biased, or unreliable data, leading to flawed conclusions and wasted resources.
Internal problem
Feeling uncertain and overwhelmed by the many decisions in survey design (sampling, mode, question wording), and fearing that their research will be criticized for methodological weaknesses or that they will misinterpret data from other sources.
Philosophical problem
It's just plain wrong that important decisions are made based on poorly executed surveys that misrepresent reality, undermining the credibility of social research and leading to bad policy or business outcomes.
The plan
- Understand the Sources of Survey Error: Learn to identify the different types of error (sampling, nonresponse, measurement) that can affect your data.
- Master the Core Methodologies: Follow clear guidance on the three pillars of a survey: sampling, question design, and data collection.
- Adopt a Total Survey Design Approach: Learn to make informed trade-offs between cost and accuracy across all aspects of your survey to optimize data quality.
Success
- The reader confidently designs and executes methodologically sound surveys.
- They can produce precise, defensible statistics, critically evaluate research done by others, and make a credible contribution to their field or organization.
At stake
- The reader continues to conduct or use surveys with hidden flaws, leading to inaccurate conclusions.
- Their research is dismissed for methodological errors, and they risk making poor decisions based on bad data.
- The credibility of their work and of social research in general is undermined.
Chapter by chapter
ch02Designing Questions to Be Good Measures
This chapter elucidates the critical role of question design in survey research, emphasizing its influence on the reliability and validity of survey measurements.
- Well-designed survey questions are the linchpin of reliable and valid research outcomes, fundamentally influencing data quality.
- The distinction between factual and subjective questions requires careful consideration, as each type presents unique challenges and biases.
- Pre-testing is an essential step in question design, allowing researchers to uncover ambiguities that could compromise data integrity.
- Use of standardized scales in subjective measures can enhance the comparability and reliability of survey responses.
ch03Introduction
This chapter outlines the foundational principles and practices of survey research, emphasizing how effective survey design influences the accuracy and credibility of data collected from samples.
- Surveys are a fundamental tool for gathering vital statistics that influence policy and public opinion.
- Credible data in surveys is achieved through rigorous sampling, standardized question design, and skilled interview practices.
- Total survey design emphasizes that the effectiveness of survey outcomes relies on the harmony of its components, challenging researchers to not neglect any aspect.
- Misleading data can stem from poor survey design; thus, careful attention to methodological details is essential.
ch04Types of Error in Surveys
This chapter delineates the dual sources of error—bias and variance—in survey methodologies that can obscure the accuracy of inferences drawn about target populations.
- Understanding the difference between bias and variance is essential to get accurate insights from surveys.
- Sampling error represents random variability, while bias leads to systematic misrepresentations in survey results.
- Neglected populations can skew survey data, affirming the importance of an inclusive sample frame.
- Survey questions must be designed to minimize misunderstanding and bias to capture true respondent experiences.
ch05Sampling
This chapter explores the critical components of sampling design in research, emphasizing the necessity for proper sampling techniques to ensure representativeness and precision in survey estimates.
- The reliability of survey findings is significantly determined by the quality of the sampling process employed.
- Comprehensive and accurately defined sample frames are essential for producing valid results.
- Probability sampling methods such as simple random, systematic, and stratified sampling enhance precision in estimates.
- Multistage sampling approaches are invaluable for populations lacking comprehensive lists of individuals.
ch06p01Nonresponse: Implementing a Sample Design (part 1/2)
This chapter explores the critical issue of survey nonresponse, dissecting its impact on data validity and proposing strategic methods to enhance response rates and reduce bias in sampling.
- Effective survey research begins with earnest efforts to minimize nonresponse, significantly impacting data quality.
- Understanding the three dimensions of nonrespondent categories is crucial for implementing strategies to improve response rates.
- Calculating response rates must be transparent and include all relevant segments of the sample to avoid misrepresentation.
- Persistence in contacting respondents across varied timelines boosts engagement and represents a best practice in survey methodology.
ch06p02Nonresponse: Implementing a Sample Design (part 2/2)
This chapter scrutinizes the intricacies of survey question formulation, emphasizing the dangers of ambiguous wording and the necessity of maintaining standardized questioning methods to ensure data integrity.
- Strong survey questions begin with clarity and specificity, reducing the risk of nonresponse.
- Avoid introducing optional probes; they invite ambiguity and diverging interpretations.
- The order in which questions and response options are presented can fundamentally affect the reliability of responses.
- Tailor questions to the individual circumstances of respondents, utilizing prior information from the interview process.
ch07Designing Questions to Be Good Measures
This chapter emphasizes the critical importance of crafting survey questions that accurately measure respondents' intended meanings, ensuring both reliability and validity in survey research.
- Clear question design is critical for obtaining reliable and valid survey data; ambiguity leads to unreliable conclusions.
- Interviewers' modifications to questions can introduce variability—try to provide structured scripts and avoid complex phrasing.
- Proper definitions of terms can significantly enhance the understanding of survey questions across diverse populations.
- Utilize pilot testing to uncover potential issues in question interpretation before finalizing the survey instrument.
ch08Evaluating Survey Questions and Instruments
This chapter addresses the critical evaluation of survey questions and instruments, arguing for meticulous pre-testing and refinement to ensure valid data collection.
ch09Survey Interviewing
The effectiveness of survey data collection significantly hinges on the skills and management of interviewers, whose roles encompass gaining cooperation, training respondents, and ensuring standardized interactions, ultimately influencing data quality.
- Interviewers are not just conduits; they play a crucial role that directly impacts data quality.
- Engaging interviewers' interpersonal skills can significantly improve respondent cooperation.
- Effective training emphasizes the importance of communication clarity, pacing, and appropriate probing.
- Ongoing supervision is indispensable for maintaining high interview standards and ensuring data integrity.
ch10Preparing Survey Data for Analysis
This chapter outlines the essential steps and best practices required to transform survey answers into analyzable data formats, emphasizing the importance of accurate coding, data entry, and cleaning procedures to ensure the reliability of research findings.
- Proper coding is a vital step for turning survey answers into data; it necessitates clear rules and consistency to avoid errors.
- Establishing distinct codes for various types of missing data can significantly enhance the clarity and usability of a dataset.
- Engaging coders in continuous training and independent verification is essential to minimizing coding errors and ensuring reliable results.
- Utilizing technology in data entry can streamline processes and facilitate error identification, making data collection tasks more efficient.
ch11Analyzing Survey Data
This chapter examines critical analytic issues in survey data, focusing on how to address nonresponse, item nonresponse, weighting, and sampling error adjustments to improve the accuracy of survey estimates.
- Nonresponse in surveys can severely bias results if not adequately addressed, and adjustments are essential for accurate population estimates.
- Weighting serves as a critical tool to correct for skewed gender or demographic representation in survey data, providing more reliable averages.
- Item nonresponse can distort analytical outcomes, necessitating the exploration of imputation methods to fill in missing data.
- Different imputation models can yield varying results; thus, running comparative analyses can validate your approach before finalizing the data.
ch12Ethical Issues in Survey Research
This chapter addresses the ethical considerations critical to conducting survey research, focusing on protecting the rights and well-being of participants throughout the research process.
- Ethical survey research must prioritize the rights and wellbeing of participants, adhering to the guidelines established by the Belmont Report.
- Researchers can often conduct surveys without requiring signed consent forms, provided that respondents are thoroughly informed about their rights and the research objectives.
- Confidentiality is a cornerstone of ethical research; rigorous protocols should be established to protect respondent information.
- Researchers must balance the delivery of participant incentives to avoid any coercive pressure in the decision to participate.
ch13Providing Information About Survey Methods
This chapter argues for the necessity of detailed methodological reporting in survey research, emphasizing transparency regarding sampling procedures, nonresponse analysis, and measurement validity to enhance the credibility of survey findings.
- Robust methodological reporting is essential to enhance the credibility and reliability of survey research findings.
- Transparency regarding sampling procedures, nonresponse rates, and question validity can significantly affect the interpretation of survey data.
- A comprehensive methodological appendix is not just a best practice; it is an obligation to facilitate understanding and replication of results.
- Low response rates necessitate in-depth nonresponse analysis to assess potential distortions in survey estimates.
ch14Survey Error in Perspective
This chapter articulates the importance of the Total Survey Design framework in mitigating survey errors while evaluating the intricate balance of cost versus methodological rigor in survey research.
Related in the library