Design, Evaluation, and Analysis of Questionnaires for Survey Research (Wiley Series in Survey Methodology)

The model

A causal-path model in which question design choices and contextual conditions (language, country, mode) influence the psychological response process and method reactions, which in turn determine measurement quality (reliability, validity, method effects), and ultimately the accuracy of substantive estimates of relationships and means.

A candidate measure

Design, Evaluation, and Analysis of Questionnaires for Survey Research (Wiley Series in Survey Methodology) — derived measurement candidates

The story

The reader A survey researcher or social scientist who wants to design questionnaires that accurately measure the concepts they care about and yield trustworthy results.

Chapter by chapter

1. ch01CONCEPTS-BY-POSTULATION AND CONCEPTS-BY-INTUITION

   This chapter explores the dichotomy between concepts formed through intuition and those developed through formal postulation, arguing that both play critical roles in the understanding and development of knowledge.

2. ch02FROM SOCIAL SCIENCE CONCEPTS-BY-INTUITION TO ASSERTIONS

   This chapter examines the transition from intuitive social science concepts to formal assertions, emphasizing clarity in communication and the importance of structuring thoughts effectively.

3. ch03THE FORMULATION OF REQUESTS FOR AN ANSWER

   This chapter systematically explores how to craft effective requests for answers, focusing on the importance of clarity in communication and the nuances of question formulation.

4. ch04p01SPECIFIC SURVEY RESEARCH FEATURES OF REQUESTS FOR AN ANSWER (part 1/3)

   This chapter examines critical components of survey requests, focusing on selecting requests from databases and identifying problematic requests while delineating fundamental features connected to research goals.

   • Careful selection of survey requests is vital for obtaining accurate data reflective of research objectives.
   • Requests derived from prior successful studies can offer valuable templates for constructing new questions.
   • Identifying and addressing problematic requests can enhance the overall reliability of survey outcomes.
   • Articulating requests clearly not only respects respondents' time but also enhances response quality.

5. ch04p02SPECIFIC SURVEY RESEARCH FEATURES OF REQUESTS FOR AN ANSWER (part 2/3)

   This chapter outlines the specific structures and linguistic principles that govern the formation of requests for answers in survey research, emphasizing their implications for data accuracy and respondent interpretation.

   • Precise linguistic structures underpin effective survey question formulation, directly impacting data quality.
   • Awareness of how extensions to simple assertions can alter meaning is crucial in survey design.
   • Clarity and specificity in requests for answers prevent ambiguous interpretations that jeopardize data reliability.
   • Utilizing a systematic approach to structuring requests helps ensure alignment with intended research concepts.

6. ch04p03SPECIFIC SURVEY RESEARCH FEATURES OF REQUESTS FOR AN ANSWER (part 3/3)

   This chapter dissects the intricacies of constructing effective survey requests, emphasizing the significance of detailed classification, time references, social desirability, and the challenges of double-barreled requests.

7. ch05OTHER FEATURES OF SURVEY REQUESTS

   This chapter examines various components of survey request formulations, including comparative and absolute judgments, conditional clauses, and balanced versus unbalanced requests, highlighting their implications for data quality and respondent engagement.

8. ch06THE STRUCTURE OF OPEN-ENDED AND CLOSED SURVEY ITEMS

   This chapter delineates the components and structures of survey items, contrasting open-ended and closed survey requests to enhance comprehension and data quality in survey methodologies.

9. ch07SURVEY ITEMS IN BATTERIES

   This chapter examines the structure and function of survey item batteries in social science research, emphasizing how these items, often grouped under a single introduction and request, can lead to varying interpretations and response qualities.

10. ch08MODE OF DATA COLLECTION AND OTHER CHOICES

    This chapter examines critical decisions influencing survey research outcomes, emphasizing modes of data collection, question placement, layout design, and language use in questionnaires, with a focus on their effects on data quality and response accuracy.

    • The choice of data collection mode significantly influences survey quality; thoughtful selection based on budget and population access is paramount.
    • Question ordering within questionnaires can trigger psychological effects that impact respondent answers—sequence strategically for improved accuracy.
    • Self-administered questionnaires must have intuitive layouts to minimize misinterpretation; clarity is especially critical when no interviewer is present.
    • Language considerations in multilingual surveys go beyond simple translation; cultural context must guide the phrasing of questions for accurate comprehension.

11. ch09CRITERIA FOR THE QUALITY OF SURVEY MEASURES

    The chapter scrutinizes the hidden complexities in the design and evaluation of survey measures, arguing that choices in item structure and data collection significantly influence data quality.

    • Survey item design is not merely procedural; it critically impacts data quality.
    • Employing the MTMM framework allows for more reliable assessments of survey quality.
    • Minor changes in question phrasing can lead to major shifts in respondent answers, reflecting the importance of intentionality in item development.
    • Proper evaluation of item nonresponse and bias is essential for ensuring representative survey data.

12. ch10ESTIMATION OF RELIABILITY, VALIDITY, AND METHOD EFFECTS

    This chapter details the processes for estimating reliability, validity, and method effects in measurement models, emphasizing the importance of proper specification and the utility of the multiple traits and multiple methods (MTMM) approach.

13. ch11SPLIT-BALLOT MULTITRAIT–MULTIMETHOD DESIGNS

    The chapter explores the limitations of classical multitrait-multimethod (MTMM) designs in survey research, proposing the split-ballot multitrait-multimethod (SB-MTMM) design as a solution to reduce response burden and improve data quality by minimizing repeated questioning.

    • The classical MTMM design's repetitiveness can lead to decreased response quality due to participant fatigue and memory bias.
    • The SB-MTMM design offers an innovative solution, allowing for fewer observations without sacrificing data reliability or validity.
    • Implementing a two- or three-group SB-MTMM design can significantly enhance data quality by strategically balancing respondent engagement with methodological rigor.
    • Researchers must weigh the trade-offs between respondent burden and data completeness when selecting survey designs.

14. ch12THE EMPIRICAL IDENTIFIABILITY AND EFFICIENCY OF THE DIFFERENT SB-MTMM DESIGNS

    This chapter evaluates the empirical identifiability and efficiency of different Split-Ballot Multi-Trait Multi-Method (SB-MTMM) designs, highlighting the conditions under which each design succeeds or fails in estimating model parameters.

15. ch13MTMM EXPERIMENTS AND THE QUALITY OF SURVEY QUESTIONS

    This chapter explores the use of Multi-Trait Multi-Method (MTMM) experiments to assess the quality of survey questions, revealing ongoing challenges in measurement error and the evolution of methodologies since 1979.

16. ch14MTMM EXPERIMENTS AND THE QUALITY OF SURVEY QUESTIONS

    This chapter delves into the methodology and findings of Multi-Trait Multi-Method (MTMM) experiments aimed at assessing the quality of survey questions, emphasizing the significant variation in question quality across different contexts and languages.

    • The quality of survey questions is highly variable across different languages and countries; understanding these differences is critical for accurate data interpretation.
    • MTMM experiments have shown that even slight changes in question formulation can lead to substantial differences in reliability and validity coefficients.
    • Statistical modelling and coding systems can uncover critical insights about question design that are often overlooked in traditional survey methodologies.
    • Developed algorithms provide a robust means to predict the quality of new questions, which can help researchers avoid measurement errors.

17. ch15THE SQP 2.0 PROGRAM FOR PREDICTION OF QUALITY AND IMPROVEMENT OF MEASURES

    The SQP 2.0 program provides a robust framework for assessing and improving the quality of survey questions using empirical data drawn from multitrait-multimethod (MTMM) experiments and a vast database.

    • The SQP 2.0 program represents a groundbreaking tool that predicts and improves survey question quality, which is vital for researchers aiming to enhance data integrity.
    • By leveraging MTMM experiences, researchers can obtain accurate quality estimates for their survey instruments, minimizing the risk of error.
    • The intuitive design of SQP encourages active engagement in the coding process, fostering a greater understanding of what constitutes a quality survey question.
    • Iterative question revision based on SQP suggestions leads to significant improvements in clarity and reliability, exemplified by increased quality scores.

18. ch16THE QUALITY OF MEASURES FOR CONCEPTS-BY-POSTULATION

    This chapter intricately explores how the measurement quality of concepts-by-postulation (CP) can be effectively derived from the reliability and validity of the underlying concepts-by-intuition (CPI).

    • The quality of measures for concepts-by-postulation must frequently be derived from the solidity of the concepts-by-intuition that underpin them.
    • Using reflective indicators, the perceived correlations among constructs like political efficacy must be adequately tested before further analysis.
    • A model's fit to data does not equate to its correctness without consideration of underlying measurement errors or method effects.
    • Weighted procedures significantly outperform unweighted approaches in deriving meaningful composite scores and correlations.

19. ch17CORRECTION FOR MEASUREMENT ERRORS

    Measurement errors are inevitable in data collection, and neglecting to correct for them can lead to biased estimates of relationships between variables, impacting research validity and conclusions drawn from survey data.

    • Measurement errors are a constant in survey research, but they can be effectively corrected to enhance data quality.
    • The quality of survey questions directly affects the validity of research findings and should be evaluated rigorously.
    • Failure to correct for measurement errors can significantly distort the relationships identified in analyses, leading to incorrect conclusions.
    • This chapter's method for correcting measurement errors is accessible and should be implemented as standard practice by researchers.

20. ch18p01COPING WITH MEASUREMENT ERRORS IN CROSS-CULTURAL RESEARCH (part 1/2)

    This chapter addresses the challenge of measurement errors in cross-cultural research, emphasizing the difficulty of comparing results across different countries due to variations in the meaning and interpretation of measures.

    • Measurement errors in cross-cultural research can significantly alter research outcomes and conclusions.
    • Functional equivalence in measures is critical for valid comparisons across cultures.
    • Researchers must apply rigorous standards to evaluate measurement invariance to avoid misleading conclusions.
    • The power of statistical tests is crucial in determining equivalence; significant results require an analysis of substantive relevance.

21. ch18p02COPING WITH MEASUREMENT ERRORS IN CROSS-CULTURAL RESEARCH (part 2/2)

    This chapter navigates the critical challenges of measurement errors in cross-cultural research, emphasizing strategies to enhance the reliability and validity of data collected across diverse cultural contexts.

    • Measurement errors are a critical concern in cross-cultural research, often highlighting disparities in interpretation across different cultural groups.
    • The multitrait-multimethod (MTMM) approach is invaluable for identifying and correcting for potential biases that may arise from differing response styles.
    • Researchers must ensure question clarity to foster reliability, recognizing that even slight ambiguities can lead to significant data distortions.
    • Engaging both quantitative and qualitative methods enriches the dataset and helps capture the nuance of cultural attitudes.