The Turnaround That Wasn't

The slide at the leadership offsite is a win, and everyone in the room can feel it. A year ago the company pulled the twenty lowest-scoring managers off the annual engagement survey — the bottom decile, the ones whose teams were quietly miserable — and put them through a development program: a coach, a workshop series, a 360. The new survey is in, and their scores have jumped. Not a little: the biggest year-over-year gains anywhere in the company. The chart shows two bars per manager, last year stubby and this year tall, and the story tells itself. The program works. Scale it. Fund it for everyone.

Here is the problem, and it has nothing to do with whether the program was any good. The managers were chosen because they scored at the bottom. And a group chosen for scoring at the bottom will, on average, score higher the next time you measure them — whether or not anything was done to them. The offsite is celebrating an effect that would have shown up if the twenty managers had spent the year on a beach. Without a comparison group of equally-low managers who got nothing, the slide cannot tell the difference, and neither can the room.

They say: target the bottom and measure the lift

It is the most natural improvement design in the world, which is exactly why it's everywhere. Find the worst — the lowest-engagement teams, the bottom-quartile sales reps, the highest-incident sites, the employees a model has flagged as high attrition risk. Aim the intervention there, because that's where the need is and where the upside looks biggest. Measure before, measure after, report the gain. The logic feels not just defensible but responsible: you put the resource where the problem was, and the problem got smaller.

The trouble is that you selected the group on the same noisy measure you're now using to judge it, and you did it without a control. That single design choice is enough to manufacture a success out of nothing at all.

Regression to the mean

Francis Galton found it in 1886 and gave it the name we're stuck with. Unusually tall parents tend to have children who are tall but, on average, shorter than they are — closer to the population's middle. Unusually short parents, the reverse. He called it regression toward mediocrity, and the thing to understand is that it isn't a force pulling anyone anywhere. It is just what happens whenever two measurements are correlated but not perfectly: the extremes don't fully repeat.¹

The mechanism is plain once you say it out loud. Any score a team posts is part real and part luck — a stable level of how that team is actually doing, plus the noise of a particular quarter: a star who happened to quit that month, a reorg mid-survey, a single bad week that colored everyone's answers. When you skim off the lowest scores, you are selecting partly for teams that are genuinely struggling and partly for teams that just had an unusually bad draw of noise. The genuine struggle persists. The bad draw, by definition, does not. So next time you measure, the noise washes out and the group drifts upward — back toward where it really lives. No coach required.

Kahneman tells the cleanest version of how this fools us, from flight instructors he once lectured. They had noticed that when they praised a cadet for a beautiful landing, the next one was usually worse, and when they screamed at a cadet for a bad one, the next was usually better. They had concluded, reasonably and completely wrongly, that praise hurts performance and punishment helps it. What they were actually watching was regression to the mean: an exceptional landing is followed by a more ordinary one, and an awful landing by a more ordinary one, no matter what the instructor says in between. The praise and the rebuke were riding on a pattern they didn't cause and couldn't change.² Every program that targets the extreme and re-measures is the flight instructor, handed a result the world was going to produce anyway.

What the honest version looks like

The fix is not to stop helping the teams that need it. It is to build the one comparison that lets you see your own effect through the regression.

Where you can, randomize within the group you selected: take the bottom decile and give the program to a random half, hold the other half (or wait-list them, which is often easier to sell as fair). Both halves are equally extreme, so regression lifts both by the same amount; the gap that's left is what you actually did. Where randomizing inside the bottom is impossible, reach for a comparison group of similarly-low teams that didn't get the program and run the difference-in-differences — the same move the causal-inference piece describes, for the same reason. And where you can't manage even that, you can at least model the regression you should expect from the correlation between the two surveys, and check whether your gain clears it. Campbell and Stanley named "statistical regression" a threat to a study's validity sixty years ago precisely for designs that select on extreme scores, and the defense they prescribed is the one that still works: a control group exposed to the same selection.³

What you lose is the clean two-bar slide. What you gain is the ability to answer the only question that matters — did we cause the improvement, or did we schedule a measurement at the bottom of a bounce? — and sometimes the honest answer is the program did little and the money should go elsewhere. That answer is worth more than a celebration built on arithmetic that was always going to come out your way.

Why it's worth raising your voice about

Because the slide doesn't just flatter a program; it spends on it. The development initiative gets rolled out company-wide on the strength of a gain the bottom decile would have posted regardless, and a year later nobody can explain why the company-wide version didn't replicate — because the original "win" was never an effect to begin with. Worse, the same physics runs in reverse at the top: the highest-scoring managers, also selected on an extreme, drift down next year, and someone gets a hard conversation about a decline that is pure regression. A vendor's "proven 18-point lift" is, often as not, this exact picture sold back to you in a deck.

And it's getting easier to fall for, because the targeting is getting automated. A model flags the highest-risk employees; an intervention fires; their risk scores fall next quarter; the dashboard credits the model and the program. But the flag selected the extreme — a chunk of that high risk was transient, and it was always going to subside. The faster and more confidently a system picks out the worst cases, the more efficiently it sets up a regression it will then take the credit for.

So before you scale the turnaround, ask the question that separates a real effect from a bounce: did we pick these units for being extreme on the same measure we're now using to score them — and was there a group just as bad that got nothing? If the answer is yes and no, hold the applause. You may be funding a comeback that was always coming back.

---

Measurement-first, in the principal-issues register: a method piece that stands on its own whether or not you ever work with us. If you want intervention studies designed so the effect survives the regression — controls, comparison groups, honest pre-registration of what counts as a win — that discipline is the Principia measurement program's whole posture, and its causal-inference companion Correlation Isn't a Driver takes up the neighboring trap. Every footnote names a real, checkable work.

Footnotes

1. Francis Galton, "Regression towards Mediocrity in Hereditary Stature," Journal of the Anthropological Institute of Great Britain and Ireland 15 (1886): 246–263 — the origin of the term "regression": children of extreme-statured parents fall, on average, back toward the population mean. The effect is a property of imperfect correlation between two measurements, not a force acting on anyone. ↩

2. Amos Tversky & Daniel Kahneman, "Judgment under Uncertainty: Heuristics and Biases," Science 185, no. 4157 (1974): 1124–1131 — discusses regression to the mean and the systematic failure to anticipate it; the flight-instructor case (praise followed by worse performance, rebuke by better) is developed at length in Kahneman, Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011), ch. 17, "Regression to the Mean." ↩

3. Donald T. Campbell & Julian C. Stanley, Experimental and Quasi-Experimental Designs for Research (Rand McNally, 1963) — names "statistical regression" among the threats to internal validity, arising whenever groups are selected on the basis of extreme scores; the prescribed defense is a comparison group subject to the same selection. ↩