peopleanalyst

library / libc8eb6d9df6e05702

Statistical Power Analysis for the Behavioral Sciences

Jacob Cohen · 1988

In a sentence

A comprehensive handbook for behavioral scientists that explains the concept of statistical power and provides practical methods and tables to calculate it for various statistical tests, enabling more rational research planning and interpretation of results.

Most behavioral science research is plagued by underpowered studies, where researchers have a low probability of detecting real effects, leading to a literature filled with ambiguous 'not significant' findings. 'Statistical Power Analysis for the Behavioral Sciences' is the definitive guide to overcoming this problem, providing researchers with the conceptual framework and practical tools to rationally plan their studies. Jacob Cohen demystifies the four key parameters of statistical inference—significance level (α), power (1-β), sample size (n), and effect size (ES)—and provides extensive tables to calculate power for a given sample size or determine the sample size needed to achieve a desired power for a wide array of common statistical tests. By emphasizing the crucial concept of 'effect size,' the book shifts the focus from mere statistical significance to the magnitude of the phenomenon under study, empowering researchers to design more sensitive experiments, avoid wasted effort on studies doomed to fail, and more meaningfully interpret their results.

The four lenses

  • Science
  • Statistics
  • Systems
  • Strategy

The model

This model describes the fundamental relationship in statistical inference, where the statistical power of a study is determined by three key parameters: the chosen significance criterion, the size of the sample, and the magnitude of the effect in the population.

Significance Criterion (α)design lever

The pre-determined standard for rejecting the null hypothesis, representing the risk of a Type I error (falsely rejecting a true null hypothesis). It embodies both the alpha level (e.g., .05) and the directionality ('sidedness') of the test.

Sample Size (n)design lever

The number of independent observational units in a sample. It is a primary determinant of the reliability or precision of sample results and thus a key factor controlled by the researcher in planning a study.

Effect Size (ES)contextual condition

The degree to which the phenomenon under study is present in the population, or the degree to which the null hypothesis is false. It is a standardized, dimensionless index of the magnitude of an effect.

Statistical Power (1-β)outcome metric

The probability that a statistical test will lead to the rejection of the null hypothesis when it is false to a specified degree (i.e., for a given Effect Size). It is the probability of obtaining a statistically significant result and the complement of the Type II error rate.

How they connect

  • significance criterion influences statistical power
  • sample size influences statistical power
  • effect size influences statistical power

The story

The reader A behavioral or social science researcher who wants to conduct meaningful research that yields clear, publishable results and contributes to their field.

External problem

Conducting studies that frequently yield statistically non-significant results, making it difficult to publish or draw firm conclusions, and struggling to determine the appropriate sample size for studies.

Internal problem

Feeling frustrated, uncertain, and discouraged when hard work results in ambiguous findings, and worrying about wasting time and resources on studies that are destined to fail from the start.

Philosophical problem

It is wrong that so much scientific effort is wasted and progress is slowed because researchers lack the tools and understanding to design studies with adequate statistical power.

The plan

  1. Understand the fundamental concepts of power analysis: the interplay of alpha, beta, sample size, and effect size.
  2. For your specific statistical test, learn the appropriate effect size index (e.g., d, r, f) from the relevant chapter.
  3. Use the provided tables to either determine the power of your study or calculate the required sample size to achieve your desired power.

Success

  • Design studies with a rational basis for their sample size.
  • Achieve a high, predetermined probability of detecting the effects you are looking for.
  • Waste less time and resources on inconclusive research.
  • Produce clearer, more publishable results that make a solid contribution to scientific knowledge.
  • Feel confident and competent in statistical and research planning.

At stake

  • Continue to conduct underpowered studies that are likely to fail.
  • Accumulate a portfolio of ambiguous, unpublishable 'not significant' findings.
  • Contribute to the 'file drawer problem' and slow the progress of your scientific field.
  • Remain frustrated and uncertain about the true meaning of your research results.

Questions this book answers

What is statistical power and why is it important in research?
How are the four key parameters of statistical inference—significance criterion (alpha), sample size (n), effect size (ES), and power (1-beta)—interrelated?
How can a researcher calculate the power of a statistical test before conducting a study?
How can a researcher determine the necessary sample size to have a high probability of detecting an effect of a specific magnitude?
What constitutes a 'small,' 'medium,' or 'large' effect for various common statistical tests?

Glossary

Significance Criterion (α)
The standard of proof that a phenomenon exists, defined as the risk of a Type I error (α), which is the probability of mistakenly rejecting a true null hypothesis. It combines the probability level (e.g., .05, .01) and the directionality of the test (one-tailed or two-tailed).
Sample Size (n)
The number of independent units in a research sample. It is the primary determinant of the reliability and precision of sample statistics as estimates of population parameters.
Effect Size (ES)
The degree to which the phenomenon under study is present in the population, or the degree to which the null hypothesis is false. It is a dimensionless index representing the magnitude of a difference, relationship, or other effect.
Statistical Power (1-β)
The a priori probability that a statistical test will yield a statistically significant result, thereby correctly rejecting the null hypothesis when a specific alternative hypothesis (a given effect size) is true. It is the complement of the Type II error rate (1 - β).

Tools these methods power