library / lib185ac0c01875305e
Experimental Quasiexperimental Designs Shadish
In a sentence
A comprehensive guide to designing and interpreting experimental and quasi-experimental studies to draw valid inferences about cause, effect, and their generalization to broader populations, settings, treatments, and outcomes.
Building on the classic works of Campbell and Stanley (1963) and Cook and Campbell (1979), this book is the definitive resource for researchers, evaluators, and students who need to establish credible cause-and-effect relationships. It provides a sophisticated yet practical framework centered on a four-part validity typology—statistical conclusion, internal, construct, and external validity—and a systematic process of identifying and ruling out plausible threats to each. The authors offer a rich toolkit of design elements for constructing strong studies, especially quasi-experiments for field settings where randomization is not feasible. Going beyond its predecessors, the book presents a novel, grounded theory of causal generalization, offering principles and methods for extending findings beyond the specific context of a single study. For anyone serious about evidence-based causal claims in the social, behavioral, health, or policy sciences, this text is an indispensable guide to methodological rigor and thoughtful inference.
The four lenses
- Science
- Statistics
- Systems
- Strategy
The model
This is a meta-model of the book's core argument, portraying how research design choices (Design Levers) function to reduce specific threats to validity (Inferential Integrity), thereby strengthening the four types of validity (Intermediate Outcomes) that collectively support a robust, Generalized Causal Inference (Final Outcome).
Use of Randomizationdesign lever
The use of a formal chance process, such as a coin toss or a random number table, to assign research units to different treatment conditions. This is the defining feature of a randomized experiment.
Control of Assignment Mechanismdesign lever
The degree to which the researcher controls the process by which units are assigned to conditions, particularly through a known, replicable, and fully modeled rule, as in a regression-discontinuity design.
Use of Structural Design Elementsdesign lever
The incorporation of various non-randomized design features to improve inference, such as control groups, pretests, multiple observations over time (time-series), switching replications, or nonequivalent dependent variables.
Use of Generalization-Focused Samplingdesign lever
The deliberate selection of instances of persons, settings, treatments, or outcomes to be either typical of a target category or intentionally heterogeneous, in order to support generalizations.
Use of Multiple Studiesdesign lever
The synthesis of findings from multiple studies, either through a directed program of research or through literature reviews (narrative or meta-analytic), to explore causal generalization.
Reduction of Internal Validity Threatspsychological state
The degree to which plausible alternative explanations for the observed cause-effect relationship (e.g., selection, history, maturation, regression, attrition, testing, instrumentation) have been ruled out by design or measurement.
Reduction of Statistical Conclusion Threatspsychological state
The degree to which threats to validly inferring covariation between cause and effect (e.g., low statistical power, violated test assumptions, unreliability of measures) have been minimized.
Reduction of Construct Validity Threatspsychological state
The degree to which threats to validly inferring the higher-order constructs that research operations represent (e.g., inadequate explication, construct confounding, mono-operation bias) have been minimized.
Reduction of External Validity Threatspsychological state
The degree to which threats to generalizing a causal relationship across variations in persons, settings, treatments, and outcomes (i.e., interaction effects) have been explored and found to be minimal.
Statistical Conclusion Validityoutcome metric
The validity of inferences about the correlation (covariation) between the presumed cause and the presumed effect, including its magnitude and statistical significance.
Internal Validityoutcome metric
The validity of inferences about whether the observed covariation between a treatment and an outcome reflects a causal relationship as those variables were manipulated or measured in the specific study context.
Construct Validityoutcome metric
The validity of inferences about the higher-order constructs that the particular persons, settings, treatments, and outcomes used in a study represent.
External Validityoutcome metric
The validity of inferences about whether a cause-effect relationship holds over variations in persons, settings, treatment variables, and measurement variables.
Generalized Causal Inferenceoutcome metric
The ultimate scientific goal of establishing a causal relationship that is not only valid in the local study context but is also well-characterized and understood in terms of its generalizability across populations, settings, treatments, and outcomes.
How they connect
- use of randomization → influences reduction of internal validity threats
- control of assignment mechanism → influences reduction of internal validity threats
- use of structural design elements → influences reduction of internal validity threats
- use of structural design elements → influences reduction of statistical conclusion threats
- use of generalization focused sampling → influences reduction of construct validity threats
- use of generalization focused sampling → influences reduction of external validity threats
- use of multiple studies → influences reduction of construct validity threats
- use of multiple studies → influences reduction of external validity threats
- reduction of internal validity threats → influences internal validity
- reduction of statistical conclusion threats → influences statistical conclusion validity
- reduction of construct validity threats → influences construct validity
- reduction of external validity threats → influences external validity
- statistical conclusion validity → influences generalized causal inference
- internal validity → influences generalized causal inference
- construct validity → influences generalized causal inference
- external validity → influences generalized causal inference
The story
The reader A social, behavioral, health, or policy researcher, evaluator, or graduate student who wants to conduct studies that produce credible, defensible evidence about whether their interventions, programs, or policies cause desired outcomes.
External problem
It is difficult to design studies that convincingly demonstrate a causal relationship, especially in complex field settings where full control is impossible and random assignment is often impractical or unethical.
Internal problem
They feel uncertain about their conclusions, frustrated by ambiguous findings, and worried that their hard work will be dismissed as methodologically weak and its conclusions ignored.
Philosophical problem
It is wrong that important societal decisions are made, and resources are allocated, based on weak or non-existent evidence about what truly works.
The plan
- Master the four types of validity (statistical conclusion, internal, construct, external) and the logic of identifying and ruling out specific threats to your causal inferences.
- Learn a flexible toolkit of design elements (e.g., control groups, pretests, time-series) to construct strong quasi-experiments tailored to your specific research context.
- Understand the theory and practice of implementing randomized experiments, including how to handle common problems like attrition and failed treatment delivery.
- Apply a powerful, grounded theory of generalization to extend your causal findings to other people, places, times, and treatments.
Success
- Designing and conducting studies that yield clear, credible, and defensible causal conclusions.
- Feeling confident in interpreting and communicating research findings to both scientific and lay audiences.
- Contributing to a cumulative body of knowledge that informs evidence-based policy and practice.
- Becoming a respected expert in rigorous, cause-probing research methodology.
At stake
- Continuing to produce ambiguous or confounded research findings that are easily criticized and dismissed.
- Wasting time and resources on studies that fail to answer the core causal questions they were designed to address.
- Important decisions continue to be based on anecdote and ideology, because rigorous evidence is lacking.
- Feeling perpetually insecure about the validity of one's own research conclusions.
Related in the library