library / lib7f791ee8e3f26dcd
Design, Evaluation, and Analysis of Questionnaires for Survey Research
Irmtraud N. Gallhofer, Willem E. Saris
In a sentence
A systematic, science-based method for designing survey questions, predicting their measurement quality before data collection, and correcting for measurement error in analysis.
This book transforms questionnaire design from an 'art' into a scientific activity. Saris and Gallhofer present a complete program: a three-step procedure for operationalizing complex concepts into concrete survey requests, a thorough mapping of the design choices researchers face (response scales, item structure, batteries, data collection mode), and a rigorous framework for estimating the reliability, validity, and method effects of survey questions using multitrait-multimethod (MTMM) experiments. The crowning achievement is the SQP (Survey Quality Predictor) program, built on a meta-analysis of thousands of MTMM experiments across dozens of countries and languages, which predicts the quality of any survey question from its coded characteristics—before it is ever fielded. The authors then show how to use these quality estimates to correct for measurement error in substantive analyses and in cross-cultural comparisons, demonstrating that ignoring measurement quality leads to seriously biased conclusions about relationships and means.
The four lenses
- Science
- Statistics
- Systems
- Strategy
The model
A causal-path model in which question design choices and contextual conditions (language, country, mode) influence the psychological response process and method reactions, which in turn determine measurement quality (reliability, validity, method effects), and ultimately the accuracy of substantive estimates of relationships and means.
Question Design Choicesdesign lever
The set of controllable formulation decisions for a survey request, including request type (direct/indirect, WH-word), response scale type and number of categories, labeling, balance, use of batteries, additional components, and fixed reference points.
Contextual Conditionscontextual condition
The survey context not chosen freely by the designer, including the country, the language of administration and translation, the data collection mode, and the position of the question within the questionnaire.
Cognitive Response Processpsychological state
The internal brain process triggered by a request that converts the respondent's underlying opinion on the concept of interest into a preliminary reaction, characterized by an intercept and slope linking the latent concept to the reaction.
Method Reactionpsychological state
The systematic, respondent-specific reaction to the particular method used to express an answer (e.g., scale type, agree/disagree format), producing common method variance shared across items measured with the same method.
Reliabilityoutcome metric
The strength of the relationship between the observed response and its true score, reflecting the degree to which the measure is free of random measurement error; the reliability coefficient squared is the proportion of observed variance due to the true score.
Validityoutcome metric
The strength of the relationship between the latent trait of interest and the true score, reflecting freedom from systematic method error; its complement is the method effect, since validity squared plus method effect squared equals one.
Total Measurement Qualityoutcome metric
The overall strength of the relationship between the observed variable and the latent concept of interest, equal to the product of reliability and validity (quality coefficient squared); it determines how much observed variance reflects the intended construct.
Item Nonresponseoutcome metric
The extent of missing values on a survey item, which disrupts analysis and can render results unrepresentative of the population; treated as a basic quality criterion alongside bias.
Response Biasoutcome metric
A systematic difference between the real values of the variable of interest and the observed scores corrected for random error, reflected in shifted response distributions across methods.
Composite Score Qualityoutcome metric
The quality of an aggregated measure (sum or weighted composite) of multiple concepts-by-intuition used to represent a concept-by-postulation, derived from the qualities of its component indicators and the chosen weights.
Accuracy of Substantive Estimatesoutcome metric
The degree to which estimated relationships between variables and comparisons of means reflect true population values rather than artifacts of measurement error; improved by correcting correlations and estimates using quality information.
Cross-Cultural Comparabilityoutcome metric
The extent to which measures have equivalent meaning across countries and languages (configural, metric, scalar, or cognitive equivalence), enabling valid comparison of means and relationships across groups.
How they connect
- question design choices → influences cognitive response process
- question design choices → influences method reaction
- question design choices → predicts reliability
- question design choices → predicts validity
- contextual conditions → moderates reliability
- contextual conditions → moderates cognitive response process
- cognitive response process → influences validity
- method reaction − influences validity
- method reaction → influences response bias
- reliability → predicts total measurement quality
- validity → predicts total measurement quality
- total measurement quality → predicts composite score quality
- total measurement quality → influences analytic accuracy
- composite score quality → influences analytic accuracy
- item nonresponse − influences analytic accuracy
- contextual conditions → influences cross cultural comparability
- total measurement quality → influences cross cultural comparability
- cross cultural comparability → influences analytic accuracy
The process
This book provides a comprehensive playbook for designing, evaluating, and analyzing survey research with a strong emphasis on maximizing measurement quality. The overall process begins with foundational planning, including selecting the appropriate data collection mode and considering cross-cultural language needs. It then moves into the detailed craft of question formulation and questionnaire design, focusing on clarity, structure, and the avoidance of common biases. A core theme is the rigorous, proactive evaluation of survey questions before fielding. The book details advanced methods like the Multitrait-Multimethod (MTMM) and Split-Ballot MTMM designs to empirically assess question reliability and validity. It also introduces a predictive, tool-based approach (SQP) for estimating question quality based on coded characteristics, allowing for iterative improvement. Once a high-quality instrument is developed, the playbook covers the practical administration of the survey across various modes. The final phase shifts to post-collection analysis, where the previously gathered quality metrics are used to statistically correct for measurement error, leading to more accurate and reliable findings. The book also provides specialized analytical processes for handling complex, multi-item constructs and for testing the measurement equivalence of instruments across different cultural groups, ensuring that cross-national comparisons are valid. In essence, the playbook guides the researcher through a full lifecycle: from conceptualization and careful instrument design, through a robust quality assurance loop, to sophisticated data analysis that accounts for and corrects the imperfections inherent in the measurement process. This systematic approach aims to elevate the scientific rigor and trustworthiness of survey-based research.
Plan Survey Administration Strategy
To establish the high-level strategy for survey data collection, including selecting the appropriate mode(s) and ensuring linguistic and cultural appropriateness for the target population.
When to use: At the beginning of a survey project, before questionnaire design begins.
Step 1Identify all available data collection modes (e.g., PAPI, telephone, mail, CATI, CAPI, CASI, web).
Entry: Research objectives and target population are defined.
Exit: A list of potential data collection modes is created.
In: Research objectives, Target population definition · Out: List of available data collection modes
ch08
Step 2Evaluate each mode based on cost, expected data quality, and accessibility for the target population.
Entry: A list of potential modes is available.
Exit: A comparative analysis of modes is complete.
- Decide on the trade-offs between cost and expected data quality for each mode.
In: List of available data collection modes, Budget constraints, Target population characteristics · Out: Comparative analysis of data collection modes
ch08
Step 3Choose the primary data collection mode or a mixed-mode strategy that balances quality and cost-effectiveness.
Entry: Comparative analysis of modes is complete.
Exit: A final data collection mode is selected.
- Choose a single mode or a mixed-mode approach.
In: Comparative analysis of data collection modes, Budgetary constraints · Out: Selected mode of data collection
ch08
Step 4For multi-regional or multi-national surveys, plan for translation and cultural adaptation.
Entry: The target populations and languages are identified.
Exit: A process for translation and cultural adaptation is established.
- Choose final wording for translations based on expert feedback.
In: Source questionnaire draft, List of target languages and cultures · Out: Plan for producing translated and culturally appropriate questionnaires
ch08
Formulate Survey Questions
To design clear, unbiased, and effective survey questions that accurately measure intended concepts by transforming abstract ideas into specific, answerable requests.
When to use: During the questionnaire development phase, after the research concepts have been defined.
Step 1Identify the core concept to be measured (e.g., satisfaction, intention, behavior).
Entry: Research objectives are clearly defined.
Exit: The specific concept for the question is identified.
In: Research objectives · Out: Defined concept for measurement
ch04p02
Step 2Create assertions that reflect the concept and then transform them into questions.
Entry: Concept for measurement is defined.
Exit: A draft question is formulated.
- Choose between direct or indirect request framing.
In: Defined concept for measurement · Out: Draft survey question
ch04p02
Step 3Refine the question for specificity and clarity.
Entry: A draft question exists.
Exit: The question is specific and detailed.
- Decide when a classification is sufficiently detailed or needs more refinement.
In: Draft survey question · Out: Refined survey question
ch04p03
Step 4Incorporate a clear time reference to improve recall accuracy.
Entry: The question measures a behavior or event over time.
Exit: The question includes a specific time reference.
- Decide which time reference (past, present, or future) is most appropriate.
In: Refined survey question · Out: Time-referenced survey question
ch04p03
Step 5Review and reformulate the question to avoid common pitfalls.
Entry: A draft question is formulated.
Exit: The question is reviewed and free of common formulation errors.
- Determine if a question is double-barreled and needs splitting.
- Choose whether to use a balanced or unbalanced format.
- Decide if a conditional clause is necessary and clear.
In: Draft survey question · Out: A clear, single-concept, balanced, and neutral question
ch04p03 · ch05
Step 6Address potential for social desirability bias for sensitive topics.
Entry: The question topic is identified as potentially sensitive.
Exit: Mitigation strategies for social desirability bias are incorporated into the question design or administration plan.
- Choose a data collection method that minimizes social desirability effects.
In: Finalized question wording · Out: Survey question and administration plan designed to reduce bias
ch04p03
Design Questionnaire Structure and Layout
To organize survey questions into a coherent, logical, and user-friendly questionnaire that minimizes respondent burden and maximizes data quality.
When to use: After individual survey questions have been drafted and before pre-testing begins.
Step 1Identify the main topics of the survey and group related questions together.
Entry: A list of drafted survey questions is available.
Exit: Questions are grouped by topic.
In: List of survey questions · Out: Thematic groups of questions
ch08
Step 2Establish a logical order for the topics to guide the respondent through the survey.
Entry: Questions are grouped by topic.
Exit: A logical sequence of topics is defined.
- Choose the order of topics to minimize cognitive load and order effects.
In: Thematic groups of questions · Out: Ordered sequence of question blocks
ch08
Step 3Structure individual survey items with all necessary components.
Entry: Questions are drafted.
Exit: Each survey item is fully specified with all its components.
- Choose which optional components (e.g., motivation, definitions) to include for each item.
- Decide whether to embed answer categories within the question or list them separately.
In: Draft survey questions · Out: Fully structured survey items
ch06
Step 4Design and format batteries of requests for efficiency.
Entry: A group of related questions with a uniform response format is identified.
Exit: A question battery is designed and formatted.
- Decide how to group related requests into an effective battery.
In: Thematic groups of questions · Out: Formatted question battery
ch04p03
Step 5Develop a clear and consistent visual layout for self-administered questionnaires.
Entry: The questionnaire is for self-administration (paper or digital).
Exit: A visually clear and consistent questionnaire layout is designed.
- Determine how to present information visually to facilitate easy completion.
In: Ordered list of survey items · Out: A well-structured questionnaire layout
ch08
Evaluate Survey Question Quality
To empirically assess and improve the quality (reliability and validity) of survey questions before final deployment using a variety of pre-testing and experimental methods.
When to use: After a draft questionnaire is ready and before it is fielded for the main data collection.
Step 1Conduct pre-tests with a small sample group from the target population.
Entry: A draft questionnaire is complete.
Exit: Pre-test data and initial respondent feedback are collected.
In: Draft questionnaire · Out: Pre-test data, Respondent feedback
ch09
Step 2Perform cognitive testing to understand respondent interpretation.
Entry: Pre-test is being conducted or has just been completed.
Exit: Insights into how respondents understand and process the questions are gathered.
In: Draft questionnaire, Pre-test participants · Out: Qualitative data on question interpretation and cognitive processes
ch09
Step 3Assess the cognitive difficulty of survey items.
Entry: Draft questions are available for review.
Exit: Items are ranked by cognitive difficulty and problematic items are identified for revision.
- Select which items require revision based on their assessed cognitive difficulty.
In: Survey item drafts, Expert panel or coding scheme · Out: Cognitive difficulty assessment for each item, Revised survey items
ch09
Step 4Design and implement a Multitrait-Multimethod (MTMM) experiment to quantify reliability and validity.
Entry: Key concepts and potential question formats are identified.
Exit: An MTMM experiment is designed.
- Determine which traits and methods to include in the experiment.
In: List of concepts (traits) to measure, Alternative question formats (methods) · Out: MTMM experimental design
ch09
Step 5Execute the MTMM experiment and analyze the results.
Entry: MTMM experiment is designed and a sample is available.
Exit: Quantitative estimates of reliability, validity, and method effects are produced.
- Determine if the model fit is acceptable.
- Assess if the estimated parameters are statistically valid.
In: Collected MTMM data · Out: Reliability coefficients, Validity coefficients, Method effect coefficients
ch10
Step 6Revise survey questions based on all evaluation feedback.
Entry: All evaluation data has been collected and analyzed.
Exit: The final, improved questionnaire is ready for deployment.
- Decide whether to revise, replace, or delete a question based on its evaluation results.
In: Cognitive testing feedback, MTMM quality estimates · Out: Finalized questionnaire
ch09
Predict Survey Question Quality Using SQP
To systematically code the characteristics of survey questions and use a predictive tool (Survey Quality Predictor - SQP) to estimate their quality (reliability and validity) without conducting a full MTMM experiment for every question.
When to use: During questionnaire design to evaluate and compare different question formulations, or retrospectively to estimate the quality of questions in existing datasets.
Step 1Access the SQP tool and select a question to analyze.
Entry: Access to the SQP tool.
Exit: A specific question (new or existing) is selected for analysis.
- Choose to analyze an existing question or create a new one.
In: Survey question text · Out: Selected question in the SQP interface
ch15
Step 2Systematically code the characteristics of the selected question.
Entry: A question is selected in the SQP interface.
Exit: All relevant characteristics of the question are coded and saved.
- Determine the correct code for each characteristic based on the provided framework.
In: Selected question, SQP coding framework · Out: A set of coded characteristics for the question
ch13 · ch14 · ch15
Step 3Request a quality prediction from the system.
Entry: The question's characteristics are fully coded.
Exit: A quality prediction is generated.
In: Coded characteristics of the question · Out: Predicted quality estimates (reliability and validity)
ch13 · ch14 · ch15
Step 4Review the prediction and any suggestions for improvement.
Entry: A quality prediction has been generated.
Exit: The question has been iteratively improved based on predictive feedback.
- Choose how to refine the question based on the system's suggestions.
In: Predicted quality estimates, Improvement suggestions · Out: An improved version of the survey question, Final quality prediction
ch15
Administer Survey
To collect data from respondents using the finalized questionnaire according to the chosen administration mode.
When to use: After the survey instrument has been designed, evaluated, and finalized.
Step 1Prepare the survey materials for the chosen administration mode.
Entry: Finalized questionnaire and selected administration mode.
Exit: All survey materials are ready for deployment.
In: Finalized questionnaire · Out: Mode-specific survey materials (e.g., printed booklets, programmed software)
ch07 · ch08
Step 2Administer the survey to respondents according to the mode-specific protocol.
Entry: Survey materials are prepared and a sample of respondents is available.
Exit: Data collection is completed for the sample.
- For oral interviews, decide whether to use visual aids (show cards) for complex items.
In: Mode-specific survey materials, Sample of respondents · Out: Collected survey data
ch07
Step 3Record or capture the responses.
Entry: Respondent provides an answer.
Exit: The answer is accurately recorded.
In: Respondent's answer · Out: Recorded response
ch07
Correct for Measurement Errors in Analysis
To obtain more accurate estimates of the relationships between variables by statistically correcting for measurement errors identified in the survey instruments.
When to use: During statistical analysis of survey data, after data has been collected and cleaned.
Step 1Obtain quality estimates (reliability) for the measurement instruments used.
Entry: Survey data has been collected.
Exit: A quality estimate is available for each variable in the analysis.
- Decide whether to use MTMM experiment data or SQP predictions for quality estimates.
In: List of variables for analysis · Out: Quality estimates (reliability coefficients) for each variable
ch17
Step 2Calculate the observed correlation matrix from the survey data.
Entry: Cleaned survey dataset is available.
Exit: An observed correlation matrix is computed.
In: Survey data · Out: Observed correlation matrix
ch17
Step 3Adjust the correlation matrix for measurement error.
Entry: Observed correlation matrix and quality estimates are available.
Exit: An adjusted correlation matrix is created.
In: Observed correlation matrix, Quality estimates · Out: Adjusted correlation matrix corrected for measurement error
ch17
Step 4Analyze the adjusted correlation matrix using a Structural Equation Modeling (SEM) program.
Entry: Adjusted correlation matrix is ready.
Exit: SEM analysis is complete.
In: Adjusted correlation matrix, Specified structural model · Out: SEM model results (e.g., path coefficients)
ch17
Step 5Interpret the results, which now represent the relationships corrected for measurement error.
Entry: SEM analysis is complete.
Exit: Substantive conclusions are drawn from the corrected model.
In: SEM model results · Out: Revised estimates of effects among variables, Substantive interpretation of findings
ch17
Evaluate and Measure Complex Constructs (Concepts-by-Postulation)
To validate and measure complex, abstract constructs (Concepts-by-Postulation, CP) that are not directly observable and are instead defined by their relationship to multiple simpler, observable indicators (Concepts-by-Intuition, CPI).
When to use: During the analysis phase, when the goal is to measure a complex construct and use it in further statistical analysis.
Step 1Theoretically define the complex construct (CP) and its relationship to its observable indicators (CPIs).
Entry: A complex construct needs to be measured.
Exit: A theoretical model specifying the relationship between the CP and its CPIs is defined.
In: Research theory, List of potential indicator variables (CPIs) · Out: Specified measurement model
ch16
Step 2Test the structure of the measurement model using factor analysis.
Entry: Survey data for the indicators is collected and the measurement model is specified.
Exit: A validated factor model that fits the data is established.
- Decide to retain, reject, or modify the model based on goodness-of-fit statistics and residuals.
- Decide between competing models (e.g., one-factor vs. two-factor).
In: Survey data, Specified measurement model · Out: Validated factor model, Model fit statistics
ch16
Step 3Estimate a composite score for the construct based on the observed indicators.
Entry: A validated factor model exists.
Exit: A composite score for the construct is calculated for each respondent.
- Choose a weighting method for the composite score (e.g., unweighted, regression weights, Bartlett weights).
In: Survey data for indicators, Factor model results (for weights) · Out: Composite scores for the complex construct
ch16
Step 4Assess the quality (reliability and validity) of the composite score.
Entry: Composite scores are calculated.
Exit: The quality of the composite score as a measure of the latent construct is determined.
In: Composite scores, Latent variable scores from SEM · Out: Quality assessment of the composite measure
ch16
Test for Cross-Cultural Measurement Invariance
To determine whether a survey instrument measures the same construct in the same way across different groups, such as countries, cultures, or languages, ensuring that comparisons between groups are valid.
When to use: During the data analysis phase of a multi-group survey project, before making direct comparisons of scores or model parameters across groups.
Step 1Establish a baseline measurement model for the construct of interest.
Entry: Survey data from multiple groups is available.
Exit: A well-fitting baseline model is established for each group.
In: Multi-group survey data, Specified measurement model · Out: Group-specific model fit statistics
ch18p01
Step 2Test for configural invariance (same factor structure).
Entry: A baseline model is established.
Exit: Configural invariance is confirmed.
In: Multi-group survey data · Out: Model fit for the configural invariance model
ch18p01
Step 3Test for metric invariance (equal factor loadings).
Entry: Configural invariance is established.
Exit: Metric invariance is confirmed or rejected.
- Determine if the change in model fit (e.g., via a Chi-square difference test) is statistically significant.
In: Configural invariance model results · Out: Model fit for the metric invariance model
ch18p01
Step 4Test for scalar invariance (equal intercepts).
Entry: Metric invariance is established.
Exit: Scalar invariance is confirmed or rejected.
- Determine if the change in model fit from the metric model is statistically significant.
In: Metric invariance model results · Out: Model fit for the scalar invariance model
ch18p01
Step 5If full invariance is rejected, investigate partial invariance.
Entry: A test for full invariance (metric or scalar) has failed.
Exit: A final model (either full or partial invariance) is selected.
- Decide which parameters to free based on modification indices and theoretical considerations.
In: Model results from a failed invariance test · Out: A final, well-fitting partial invariance model
ch18p01
Step 6Incorporate power analysis to assess the reliability of the test results.
Entry: Invariance tests have been conducted.
Exit: The robustness of the invariance conclusion is assessed.
- Decide if a detected misspecification is substantively relevant or a minor artifact of high statistical power.
In: Model parameters, Modification indices · Out: An assessment of the statistical power of the invariance test
ch18p01
A candidate measure
Design, Evaluation, and Analysis of Questionnaires for Survey Research — derived measurement candidates
Question Design Choices
SQP coded characteristics; scale length; labeling completeness
self-report suitability: none
Contextual Conditions
country code; language code; mode code; position index
self-report suitability: none
Cognitive Response Process
estimated cognitive slope; estimated cognitive intercept
self-report suitability: low
Method Reaction
method factor loading; common method variance
self-report suitability: none
Reliability
reliability coefficient; SQP-predicted reliability
self-report suitability: none
Validity
validity coefficient; SQP-predicted validity
self-report suitability: none
Total Measurement Quality
quality coefficient squared; SQP-predicted quality
self-report suitability: none
Item Nonresponse
missingness rate
self-report suitability: none
Response Bias
distribution differences across methods; deviation from factual benchmarks
self-report suitability: none
Composite Score Quality
composite-latent correlation; invalidity due to method in composite
self-report suitability: none
Accuracy of Substantive Estimates
corrected vs uncorrected effect sizes; corrected vs uncorrected explained variance
self-report suitability: none
Cross-Cultural Comparability
equality of slopes/intercepts; JRule judgments; power-aware misspecification indicators
self-report suitability: none
The story
The reader A survey researcher or social scientist who wants to design questionnaires that accurately measure the concepts they care about and yield trustworthy results.
External problem
Survey questions contain measurement error that biases estimates of relationships and means, and there is no easy way to know a question's quality before fielding it.
Internal problem
The researcher feels uncertain whether their data truly measure what they intend and anxious that their conclusions may be artifacts of poor questions.
Philosophical problem
Treating questionnaire design as an unteachable 'art' is wrong when scientific methods can make the consequences of design choices known and controllable.
The plan
- Operationalize complex concepts into concrete requests using the three-step procedure.
- Make informed choices about response scales, item structure, batteries, and data collection mode.
- Predict the quality of each question with SQP before fielding and improve weak questions.
- Estimate reliability, validity, and method effects where possible via MTMM designs.
- Correct correlations and estimates for measurement error in substantive analysis.
- Test equivalence and correct for measurement quality before cross-cultural comparison.
Success
- The researcher fields higher-quality questions, knows their measurement quality in advance, obtains less biased estimates of relationships and means, and can make valid cross-cultural comparisons.
At stake
- The researcher unknowingly uses low-quality questions, draws biased conclusions, attributes measurement artifacts to substantive differences, and produces non-comparable cross-national results.
Chapter by chapter
ch01CONCEPTS-BY-POSTULATION AND CONCEPTS-BY-INTUITION
This chapter explores the dichotomy between concepts formed through intuition and those developed through formal postulation, arguing that both play critical roles in the understanding and development of knowledge.
ch02FROM SOCIAL SCIENCE CONCEPTS-BY-INTUITION TO ASSERTIONS
This chapter examines the transition from intuitive social science concepts to formal assertions, emphasizing clarity in communication and the importance of structuring thoughts effectively.
ch03THE FORMULATION OF REQUESTS FOR AN ANSWER
This chapter systematically explores how to craft effective requests for answers, focusing on the importance of clarity in communication and the nuances of question formulation.
ch04p01SPECIFIC SURVEY RESEARCH FEATURES OF REQUESTS FOR AN ANSWER (part 1/3)
This chapter examines critical components of survey requests, focusing on selecting requests from databases and identifying problematic requests while delineating fundamental features connected to research goals.
- Careful selection of survey requests is vital for obtaining accurate data reflective of research objectives.
- Requests derived from prior successful studies can offer valuable templates for constructing new questions.
- Identifying and addressing problematic requests can enhance the overall reliability of survey outcomes.
- Articulating requests clearly not only respects respondents' time but also enhances response quality.
ch04p02SPECIFIC SURVEY RESEARCH FEATURES OF REQUESTS FOR AN ANSWER (part 2/3)
This chapter outlines the specific structures and linguistic principles that govern the formation of requests for answers in survey research, emphasizing their implications for data accuracy and respondent interpretation.
- Precise linguistic structures underpin effective survey question formulation, directly impacting data quality.
- Awareness of how extensions to simple assertions can alter meaning is crucial in survey design.
- Clarity and specificity in requests for answers prevent ambiguous interpretations that jeopardize data reliability.
- Utilizing a systematic approach to structuring requests helps ensure alignment with intended research concepts.
ch04p03SPECIFIC SURVEY RESEARCH FEATURES OF REQUESTS FOR AN ANSWER (part 3/3)
This chapter dissects the intricacies of constructing effective survey requests, emphasizing the significance of detailed classification, time references, social desirability, and the challenges of double-barreled requests.
ch05OTHER FEATURES OF SURVEY REQUESTS
This chapter examines various components of survey request formulations, including comparative and absolute judgments, conditional clauses, and balanced versus unbalanced requests, highlighting their implications for data quality and respondent engagement.
ch06THE STRUCTURE OF OPEN-ENDED AND CLOSED SURVEY ITEMS
This chapter delineates the components and structures of survey items, contrasting open-ended and closed survey requests to enhance comprehension and data quality in survey methodologies.
ch07SURVEY ITEMS IN BATTERIES
This chapter examines the structure and function of survey item batteries in social science research, emphasizing how these items, often grouped under a single introduction and request, can lead to varying interpretations and response qualities.
ch08MODE OF DATA COLLECTION AND OTHER CHOICES
This chapter examines critical decisions influencing survey research outcomes, emphasizing modes of data collection, question placement, layout design, and language use in questionnaires, with a focus on their effects on data quality and response accuracy.
- The choice of data collection mode significantly influences survey quality; thoughtful selection based on budget and population access is paramount.
- Question ordering within questionnaires can trigger psychological effects that impact respondent answers—sequence strategically for improved accuracy.
- Self-administered questionnaires must have intuitive layouts to minimize misinterpretation; clarity is especially critical when no interviewer is present.
- Language considerations in multilingual surveys go beyond simple translation; cultural context must guide the phrasing of questions for accurate comprehension.
ch09CRITERIA FOR THE QUALITY OF SURVEY MEASURES
The chapter scrutinizes the hidden complexities in the design and evaluation of survey measures, arguing that choices in item structure and data collection significantly influence data quality.
- Survey item design is not merely procedural; it critically impacts data quality.
- Employing the MTMM framework allows for more reliable assessments of survey quality.
- Minor changes in question phrasing can lead to major shifts in respondent answers, reflecting the importance of intentionality in item development.
- Proper evaluation of item nonresponse and bias is essential for ensuring representative survey data.
ch10ESTIMATION OF RELIABILITY, VALIDITY, AND METHOD EFFECTS
This chapter details the processes for estimating reliability, validity, and method effects in measurement models, emphasizing the importance of proper specification and the utility of the multiple traits and multiple methods (MTMM) approach.
ch11SPLIT-BALLOT MULTITRAIT–MULTIMETHOD DESIGNS
The chapter explores the limitations of classical multitrait-multimethod (MTMM) designs in survey research, proposing the split-ballot multitrait-multimethod (SB-MTMM) design as a solution to reduce response burden and improve data quality by minimizing repeated questioning.
- The classical MTMM design's repetitiveness can lead to decreased response quality due to participant fatigue and memory bias.
- The SB-MTMM design offers an innovative solution, allowing for fewer observations without sacrificing data reliability or validity.
- Implementing a two- or three-group SB-MTMM design can significantly enhance data quality by strategically balancing respondent engagement with methodological rigor.
- Researchers must weigh the trade-offs between respondent burden and data completeness when selecting survey designs.
ch12THE EMPIRICAL IDENTIFIABILITY AND EFFICIENCY OF THE DIFFERENT SB-MTMM DESIGNS
This chapter evaluates the empirical identifiability and efficiency of different Split-Ballot Multi-Trait Multi-Method (SB-MTMM) designs, highlighting the conditions under which each design succeeds or fails in estimating model parameters.
ch13MTMM EXPERIMENTS AND THE QUALITY OF SURVEY QUESTIONS
This chapter explores the use of Multi-Trait Multi-Method (MTMM) experiments to assess the quality of survey questions, revealing ongoing challenges in measurement error and the evolution of methodologies since 1979.
ch14MTMM EXPERIMENTS AND THE QUALITY OF SURVEY QUESTIONS
This chapter delves into the methodology and findings of Multi-Trait Multi-Method (MTMM) experiments aimed at assessing the quality of survey questions, emphasizing the significant variation in question quality across different contexts and languages.
- The quality of survey questions is highly variable across different languages and countries; understanding these differences is critical for accurate data interpretation.
- MTMM experiments have shown that even slight changes in question formulation can lead to substantial differences in reliability and validity coefficients.
- Statistical modelling and coding systems can uncover critical insights about question design that are often overlooked in traditional survey methodologies.
- Developed algorithms provide a robust means to predict the quality of new questions, which can help researchers avoid measurement errors.
ch15THE SQP 2.0 PROGRAM FOR PREDICTION OF QUALITY AND IMPROVEMENT OF MEASURES
The SQP 2.0 program provides a robust framework for assessing and improving the quality of survey questions using empirical data drawn from multitrait-multimethod (MTMM) experiments and a vast database.
- The SQP 2.0 program represents a groundbreaking tool that predicts and improves survey question quality, which is vital for researchers aiming to enhance data integrity.
- By leveraging MTMM experiences, researchers can obtain accurate quality estimates for their survey instruments, minimizing the risk of error.
- The intuitive design of SQP encourages active engagement in the coding process, fostering a greater understanding of what constitutes a quality survey question.
- Iterative question revision based on SQP suggestions leads to significant improvements in clarity and reliability, exemplified by increased quality scores.
ch16THE QUALITY OF MEASURES FOR CONCEPTS-BY-POSTULATION
This chapter intricately explores how the measurement quality of concepts-by-postulation (CP) can be effectively derived from the reliability and validity of the underlying concepts-by-intuition (CPI).
- The quality of measures for concepts-by-postulation must frequently be derived from the solidity of the concepts-by-intuition that underpin them.
- Using reflective indicators, the perceived correlations among constructs like political efficacy must be adequately tested before further analysis.
- A model's fit to data does not equate to its correctness without consideration of underlying measurement errors or method effects.
- Weighted procedures significantly outperform unweighted approaches in deriving meaningful composite scores and correlations.
ch17CORRECTION FOR MEASUREMENT ERRORS
Measurement errors are inevitable in data collection, and neglecting to correct for them can lead to biased estimates of relationships between variables, impacting research validity and conclusions drawn from survey data.
- Measurement errors are a constant in survey research, but they can be effectively corrected to enhance data quality.
- The quality of survey questions directly affects the validity of research findings and should be evaluated rigorously.
- Failure to correct for measurement errors can significantly distort the relationships identified in analyses, leading to incorrect conclusions.
- This chapter's method for correcting measurement errors is accessible and should be implemented as standard practice by researchers.
ch18p01COPING WITH MEASUREMENT ERRORS IN CROSS-CULTURAL RESEARCH (part 1/2)
This chapter addresses the challenge of measurement errors in cross-cultural research, emphasizing the difficulty of comparing results across different countries due to variations in the meaning and interpretation of measures.
- Measurement errors in cross-cultural research can significantly alter research outcomes and conclusions.
- Functional equivalence in measures is critical for valid comparisons across cultures.
- Researchers must apply rigorous standards to evaluate measurement invariance to avoid misleading conclusions.
- The power of statistical tests is crucial in determining equivalence; significant results require an analysis of substantive relevance.
ch18p02COPING WITH MEASUREMENT ERRORS IN CROSS-CULTURAL RESEARCH (part 2/2)
This chapter navigates the critical challenges of measurement errors in cross-cultural research, emphasizing strategies to enhance the reliability and validity of data collected across diverse cultural contexts.
- Measurement errors are a critical concern in cross-cultural research, often highlighting disparities in interpretation across different cultural groups.
- The multitrait-multimethod (MTMM) approach is invaluable for identifying and correcting for potential biases that may arise from differing response styles.
- Researchers must ensure question clarity to foster reliability, recognizing that even slight ambiguities can lead to significant data distortions.
- Engaging both quantitative and qualitative methods enriches the dataset and helps capture the nuance of cultural attitudes.
Questions this book answers
- How can complex social science concepts be translated into survey questions that actually measure what they are intended to measure?
- What design choices affect the quality of survey questions, and how large are these effects?
- How can the reliability, validity, and method effects of a survey question be estimated?
- How can the quality of a question be predicted before data are collected?
- How should measurement error be corrected in the analysis of survey data?
Glossary
- Question Design Choices
- The controllable formulation and structural decisions a researcher makes when constructing a survey request for an answer.
- Contextual Conditions
- Aspects of the survey context not freely chosen by the designer that affect question performance.
- Cognitive Response Process
- The mental process converting a respondent's latent opinion into a preliminary reaction to a request.
- Method Reaction
- The respondent's systematic reaction to the particular method used to express an answer.
- Reliability
- The strength of the relationship between the observed response and its true score.
- Validity
- The strength of the relationship between the latent trait of interest and the true score.
- Total Measurement Quality
- The overall strength of the relationship between the observed variable and the latent concept of interest.
- Item Nonresponse
- The extent of missing answers to a survey item.
Related in the library