library / libf956094990621cc1
Methods of Meta Analysis Hunter Schmidt
In a sentence
A comprehensive guide to psychometric meta-analysis, a set of statistical methods for correcting error and bias in research findings to reveal the true underlying relationships across studies.
Researchers in the social and behavioral sciences are often faced with a bewildering landscape of conflicting findings on any given topic. Traditional narrative reviews and simplistic vote-counting methods fail to resolve these conflicts and often lead to erroneous conclusions, stalling scientific progress. 'Methods of Meta-Analysis' presents a powerful solution: psychometric meta-analysis. This book argues that much of the apparent inconsistency in research literatures is not real, but is instead the result of correctable statistical and measurement artifacts, such as sampling error, measurement error, and range restriction. It provides a rigorous, step-by-step framework for identifying, quantifying, and correcting for these distortions. By applying these methods, researchers can move beyond a superficial summary of flawed studies to estimate the true, construct-level relationships that would be observed under ideal research conditions. This book is the definitive guide for any researcher who wants to build a truly cumulative science by making sense of the vast and often confusing body of accumulated evidence in their field.
The four lenses
- Science
- Statistics
- Systems
- Strategy
The model
This is a causal model explaining the generation of observed findings in a research literature. It posits that the true relationship between constructs, which may itself vary across contexts (due to real moderators), is systematically attenuated by measurement and statistical artifacts within each study. This attenuated population parameter is then further distorted by random sampling error to produce the observed effect size. The variance of observed effect sizes across studies is therefore a composite of true variance, variance from sampling error, and variance from differences in artifacts across studies.
True Relationship Magnitude and Variancecontextual condition
The actual distribution of construct-level effect sizes (ρ or δ) across the population of settings, defined by a mean and a standard deviation. The variance (SD > 0) reflects the influence of genuine moderator variables.
Study Design Artifactsdesign lever
Methodological characteristics of a primary study that systematically or randomly distort the observed effect size. This includes measurement error (unreliability), range variation, dichotomization of continuous variables, and imperfect construct validity.
Study Sample Sizedesign lever
The number of participants (N) in a given primary research study.
Attenuated Population Parameterpsychological state
The hypothetical population effect size (ρo or δo) for a single study, after the true effect size has been attenuated by the specific systematic artifacts (e.g., measurement error, range restriction) present in that study, but before the introduction of sampling error.
Sampling Errorpsychological state
The random, unsystematic deviation of an observed sample statistic (r or d) from its population parameter (ρo or δo) due to the study being conducted on a finite sample of participants.
Observed Effect Sizeoutcome metric
The descriptive statistic (e.g., correlation r, or standardized mean difference d) reported in a primary research study, which is the result of the true effect being distorted by both systematic artifacts and random sampling error.
Variance of Observed Effect Sizesoutcome metric
The total observed variance across a set of primary study effect sizes. This variance is a composite of true effect size variance, sampling error variance, and variance due to differences in artifacts across studies.
How they connect
- true relationship magnitude and variance → influences attenuated population parameter
- study design artifacts − predicts attenuated population parameter
- study sample size − predicts sampling error
- attenuated population parameter → predicts observed effect size
- sampling error → influences observed effect size
- true relationship magnitude and variance → predicts variance of observed effect sizes
- study design artifacts → predicts variance of observed effect sizes
- sampling error → predicts variance of observed effect sizes
A candidate measure
Methods of Meta Analysis Hunter Schmidt — derived measurement candidates
True Relationship Magnitude and Variance
The final mean correlation (ρ-bar) or d-value (δ-bar) from a psychometric meta-analysis.; The final standard deviation of the correlation (SDρ) or d-value (SDδ) from a psychometric meta-analysis.
self-report suitability: none
Study Design Artifacts
Reliability coefficients (alpha, test-retest, inter-rater).; Ratio of sample SD to reference population SD (u).; Proportions in each category of a dichotomized variable.; Correlation between a proxy measure and the intended construct (construct validity).
self-report suitability: none
Study Sample Size
Reported N.
self-report suitability: none
Attenuated Population Parameter
This is a latent statistical parameter and is not directly measured. It is an intermediate calculation in the meta-analytic process.
self-report suitability: none
Sampling Error
The variance of this latent error term is estimated using a formula based on N and effect size.
self-report suitability: none
Observed Effect Size
Pearson correlation coefficient (r).; Standardized mean difference (d).; Statistics that can be converted to r or d (e.g., t, F, chi-square).
self-report suitability: none
Variance of Observed Effect Sizes
The sample-size-weighted variance of the collected effect sizes.
self-report suitability: none
The story
The reader Social and behavioral science researchers, academics, and graduate students who are struggling to make sense of the vast and often contradictory body of research in their fields. They want to move beyond simply listing study results to build a cumulative, quantitative science but feel frustrated by the apparent chaos in the literature.
External problem
The research literature on any given topic is filled with conflicting findings—some studies are statistically significant, others are not; effect sizes vary widely—making it impossible to draw clear conclusions using traditional narrative review methods.
Internal problem
Researchers feel confused, disheartened, and cynical about the possibility of scientific progress. They question whether their fields can ever achieve the cumulative knowledge seen in the physical sciences and feel their own research efforts are lost in an ocean of inconsistency.
Philosophical problem
It is fundamentally wrong that decades of collective research effort, often publicly funded, are being wasted because we lack the proper tools to distill the true knowledge from the noise of flawed individual studies.
The plan
- Identify and understand the statistical and measurement artifacts (e.g., sampling error, measurement error) that distort individual study findings.
- Learn the specific statistical procedures to correct for the biasing effects and spurious variance created by each artifact.
- Apply these methods to synthesize findings across studies, calculating the mean and standard deviation of the true, corrected effect sizes.
- Use these corrected estimates to test for the existence of real moderator variables and build cumulative, theory-driven knowledge.
Success
- Researchers can confidently establish the fundamental facts and relationships in their area of study.
- They can distinguish real moderators from artifactual noise, leading to more accurate and parsimonious theories.
- Their discipline advances as a cumulative science, with a solid, quantitative knowledge base.
- They are empowered to make credible, evidence-based contributions to both theory and practice.
At stake
- Researchers will remain trapped in a cycle of concluding 'the findings are mixed' and issuing endless calls for 'more research' that never resolves the underlying questions.
- The social and behavioral sciences will continue to be perceived as non-cumulative and 'soft,' losing credibility, funding, and impact.
- True relationships will remain obscured by artifactual noise, and scientific progress will stagnate.
Chapter by chapter
ch01p01Bare-Bones Meta-Analysis: Correcting for Sampling Error Only (part 1/2)
This chapter presents a focused exploration of sampling error in meta-analysis, arguing that the true goal is to identify relationships in research that would manifest in perfect studies, rather than merely reflecting observed results from flawed studies.
ch01p02Bare-Bones Meta-Analysis: Correcting for Sampling Error Only (part 2/2)
This chapter explores the methodology of bare-bones meta-analysis, detailing how to correct correlations for sampling error only while addressing the implications of various artifact corrections.
- Artifacts like measurement error and range restriction can lead to substantial underestimations of mean correlations in meta-analysis.
- A bare-bones meta-analysis provides significant but limited insights, as it only corrects for sampling error without addressing other critical artifacts.
- Effective corrections require understanding of both direct and indirect range restrictions and their impact on correlation measures.
- Weighting studies based on sample size and artifact attenuation is crucial for achieving accurate meta-analytic results.
ch02Meta-Analysis of Correlations Using Artifact Distributions
This chapter examines the methodology of artifact distribution meta-analysis, emphasizing its necessity in correcting correlations for measurement errors and range restrictions that affect study outcomes.
ch03Technical Questions in Meta-Analysis of Correlations
The chapter addresses critical methodological questions in meta-analysis of correlations, arguing for the superiority of correlation coefficients over squared correlations and advocating for the use of random effects models.
ch04Treatment Effects
This chapter examines the various artifacts that distort treatment effects in experimental studies, arguing for a shift from dichotomous interpretations of treatment efficacy to a quantitative understanding of effect sizes, thereby enhancing the reliability of research conclusions.
ch05p01Meta-Analysis Methods for d Values (part 1/2)
This chapter lays out the foundational mathematical frameworks for conducting meta-analyses on effect sizes from experimental studies, particularly focusing on the d statistic as a measure of group differences, while highlighting the crucial distinctions between experimental and observational studies.
ch05p02Meta-Analysis Methods for d Values (part 2/2)
This chapter explores the complexities and methodologies of adjusting the effect size statistic d for multiple artifacts during meta-analysis, offering a detailed framework for obtaining more accurate treatment effect estimates.
ch06Technical Questions in Meta-Analysis of d Values
This chapter explores the nuanced technical considerations in calculating and interpreting effect sizes (d values) in meta-analyses, addressing potential biases and methodological discrepancies across different experimental designs.
- Different experimental designs necessitate tailored approaches to appropriately compute and interpret d values in meta-analyses.
- Neglecting to adjust for partialled covariates in ANCOVA can lead to substantial overestimation of effect sizes.
- The use of fixed effects models may misrepresent study variability; random effects models should be favored for more accurate reflections of study heterogeneity.
- Credibility intervals are critical for understanding the distribution of effect sizes across studies, complementing confidence intervals that focus solely on mean estimates.
ch07p01General Technical Issues in Meta-Analysis (part 1/2)
This chapter critically examines the foundational technical issues surrounding meta-analysis, addressing misconceptions, methodological concerns in detecting moderators, and the implications of sampling errors.
- Large sample studies do not replace the need for meta-analysis; smaller studies contribute valuable insights essential for calibration and moderator analysis.
- Second-order sampling error remains a significant concern that can distort confidence intervals and effect size calculations in meta-analyses.
- Effectively detecting moderators requires robust methodologies and a framework that guards against capitalizing on chance findings.
- Systematic and hierarchical approaches in analyzing moderators lead to more accurate interpretations of the data.
ch07p02General Technical Issues in Meta-Analysis (part 2/2)
This chapter explores the complexities and implications of second-order sampling errors in meta-analysis, focusing on both homogeneous and heterogeneous cases of study effect sizes.
- Smaller study counts can lead to significant errors in effect size estimates due to primary second-order sampling errors.
- Homogeneity in effect sizes is a rare occurrence; most meta-analyses will confront heterogeneity.
- Chi-square tests for homogeneity may yield misleading results, particularly in contexts with small sample sizes.
- Increasing the number of studies can enhance the precision of meta-analytic estimates, reducing sampling error variance.
ch08Cumulation of Findings Within Studies
This chapter delves into the methodologies for handling multiple effect sizes within a single study in meta-analysis, asserting the critical role that replication types play in accurately estimating effects.
- Fully replicated designs can be treated as separate studies, offering robust independence when studies are executed across differential contexts.
- In cases of conceptual replication, it is advisable to combine measures into composite scores to minimize measurement error and enhance construct validity.
- The integration of subgroup analyses adds complexity and diminishes statistical power; care must be taken to evaluate the necessity of subgroup categorizations.
- Violations of statistical independence do not necessarily skew mean estimates in meta-analyses, yet they fundamentally alter the estimation of sampling error variance.
ch09Different Methods of Meta-Analysis and Related Software
This chapter critically examines eleven methods for conducting meta-analysis, evaluating their efficacy in synthesizing research findings and discussing applicable software tools.
ch10Locating, Evaluating, Selecting, and Coding Studies and Presentation of Meta-Analysis Results
This chapter navigates the intricacies of properly conducting literature searches, evaluating study methodologies, and presenting meta-analysis results, highlighting their significance in ensuring the validity and comprehensiveness of meta-analytic findings.
- A thorough literature search is vital; reliance on a limited number of journals can introduce significant biases.
- The assumption of methodological inadequacy must be empirically tested, as not all weaknesses result in biased findings.
- Detailed documentation of the search and study evaluation process enhances the credibility of meta-analytic reports.
- The coding process should be adaptable to the specific hypotheses of the meta-analysis and conducted with rigor.
ch11Availability Bias, Source Bias, and Publication Bias in Meta-Analysis
This chapter examines the pervasive issues of availability, source, and publication bias in meta-analysis, highlighting their implications for the validity of findings across various scientific disciplines.
- Availability bias significantly compromises the validity of meta-analyses, necessitating a proactive approach in identifying biases within the literature.
- Researchers should expand their lens beyond just publication bias to consider the complexities of source and availability biases in their studies.
- Empirical evidence suggests that many fields, especially psychology, are plagued by inflated effect size estimates due to selective publication practices.
- Employing a variety of methodological strategies can improve the detection and adjustment for biases, underscoring the necessity of robust triangulation in meta-analyses.
ch12Summary of Psychometric Meta-Analysis
This chapter articulates the critical need for a robust theoretical framework in conducting psychometric meta-analysis, emphasizing the distinction between simply summarizing flawed studies and accurately estimating population parameters.
ch13Appendix
This chapter details the technical specifications and enhancements of the Hunter-Schmidt Meta-Analysis Software Package Version 2.0, providing essential tools for performing psychometric meta-analyses.
ch14References
This chapter consolidates a comprehensive list of references that underpin the research and methodologies discussed throughout the book, highlighting the significance of each cited work in understanding the evolution and current practices of meta-analysis.
ch15Author Index
This chapter serves as a comprehensive guide to the authors referenced throughout the book, providing detailed page references for their contributions to the themes and concepts explored.
- The author index is crucial for enhancing reader engagement and comprehension of referenced ideas.
- Comprehensive citations reflect the breadth and depth of scholarship that underpins the book's content.
- A systematic approach to citations aids in bridging theory and practice within the field.
- Proper attribution of intellectual sources cultivates respect for academic integrity and knowledge advancement.
ch16Subject Index
This chapter serves as a comprehensive index for the book, providing detailed references for various concepts and methodologies discussed throughout, crucial for readers seeking to navigate complex topics in meta-analysis.
- The index serves as a vital navigational tool for understanding complex subjects in meta-analysis.
- Comprehensive coverage of terms ensures readers can easily find necessary information to support their research.
- A well-organized index enhances research efficiency and clarity, allowing for focused exploration of methodologies.
- Reference pages reinforce the interconnectedness of concepts, promoting an integrated understanding of the topic.