Decision support under uncertainty

Twenty bottom-decile managers go through a development program and post the largest year-over-year engagement gains anywhere; the offsite calls it a win and scales it. But they were chosen for scoring at the bottom, and a group chosen for an extreme drifts back toward the mean on re-measure regardless — because part of any extreme score is transient noise that doesn't repeat. Galton named the effect in 1886; Kahneman's flight instructors misread it as praise hurting and rebuke helping. The honest design builds the one comparison that sees through it: randomize within the bottom decile, or run difference-in-differences against equally-low teams that got nothing — the regression hits both, so the gap left is your effect. It costs you the clean two-bar slide and answers the only question that matters: did we cause the improvement, or schedule a measurement at the bottom of a bounce? Getting easier to miss as targeting automates — a model flags the high-risk, the scores subside, and the dashboard takes the credit. Before you scale the turnaround: did we pick these units for being extreme on the same measure we now use to score them, and was there a group just as bad that got nothing?

"Our assessment is 85% accurate at identifying top performers." True, validated, and mostly wrong in deployment. If top performers are 10% of applicants, work the arithmetic on a thousand: 85 of 100 real ones flagged, plus 135 false positives from the 900 who aren't — so of 220 names, only 85 pan out. The test is wrong about 61% of who it tells you to hire. Nothing about the test changed; the base rate did the damage. When the target is rare, the negatives swamp the positives and even a small error rate floods you with false alarms — which is why the question is never "how accurate" but "of everyone it flags, how many are real, at my base rate." Kahneman & Tversky (base-rate neglect), the physicians who answered 95% where the truth was 2%, and Meehl & Rosen (a cutting score can do worse than betting the base rate) all name it. Work the confusion matrix at your prevalence, not the vendor's. How rare is the thing it's looking for?

In WWII the military wanted to armor returning bombers where the bullet holes clustered; Abraham Wald said armor the engines, where there were none — the planes hit there didn't come back. The exit survey is that damage map. It captures only people who already left, who chose to respond, and who felt safe being honest with an employer who controls their reference — three layers of selection that filter out the regretted leavers who ghosted and, more importantly, everyone still here and saveable. You read "compensation" off the chart and spend a cycle on pay, while the real engine of regretted attrition — the thing the best people never write on an exit form — goes unmeasured. The fix is to measure the living: protected quit-intention signal from current employees, upstream of the exit, with the selection modeled rather than ignored. Who isn't in this data, and would they have said something different?

The survey team flags it in red: young remote Westerners are significantly less engaged, p < .05 — and a working group and a budget line follow. What the slide omits: finding it took hundreds of comparisons across items × demographics × regions, and at a 5% threshold, one in twenty pure-noise tests clears the bar. Run five hundred and you harvest ~25 "significant" findings from nothing. The p-value's promise holds for one pre-specified test; report the winners of a big search and significance stops carrying information (Ioannidis: the more relationships a field tests, the lower the odds any significant one is real) — worse when the subgroup was drawn around the noise after the fact. The fix: decide your hypotheses before you look; treat the rest as hypotheses to confirm on fresh data; correct for multiple comparisons; report effect sizes, not asterisks. How many cuts did we check to find this one?

Maria's team of seven "dropped" eighteen engagement points — the biggest fall in the org — so there's a meeting, and she's in it. The whole drop is one bad mood and one empty seat divided by seven: sampling noise, painted red, with an arrow. Tversky and Kahneman documented why we fall for it (the belief in the law of small numbers — we expect tiny samples to behave like big ones; they don't), and the math is blunt: a proportion from n=7 carries an error bar that can span half the scale, so most quarter-to-quarter movement is a wobble in the estimate, not a change in the team. Regression to the mean finishes the trap: intervene on the worst cells and most improve next quarter regardless, and you credit the fix. Small-N honesty is the cure — show the error bar, set a reporting floor, don't rank tiny teams, expect regression. The honest output for a small team is often a shrug with a confidence interval. Before you act on the reddest cell: how many people are actually in it?

In 1973 Berkeley looked like it admitted men over women — until you broke it out by department, where the gap reversed: women had applied more to the most selective departments, and the composition flipped the aggregate. That's Simpson's paradox, and it's the trap inside nearly every HR benchmark. "You pay at the 45th percentile," "your turnover is above industry," "your engagement is below the norm" — each compares your population to one that differs on role mix, geography, tenure, and demographics, so the gap you see is composition, not your practice. Comparing apples to a fruit basket and reporting the difference as a verdict. The fix isn't to stop benchmarking; it's to compare like with like — stratify, or run a multivariate adjustment — so the gap that *remains* is interpretable. The adjusted answer is messier and less tidy, but it's the one that tells you where to actually spend. Before you act on a percentile: a percentile of what, exactly?

A key-driver chart ranks survey items by how much variance they explain and calls them levers; a quarter later there's a reorg aimed at the top bar. But every number on it is a correlation, and "driver" is a causal claim — the chart promoted one into the other, and the company spent a budget on the promotion. Pearl's ladder makes it plain: observational regression lives on the bottom rung (association); a driver is a middle-rung (intervention) claim, and no statistical upgrade climbs the ladder for you — only a design does. The honest version says "associated with," uses the correlation to generate hypotheses, and earns the causal word with an experiment, a quasi-experiment, or at least confounder-aware longitudinal measurement. Fewer confident answers, more "here's the pilot that would tell us" — which is the only kind worth betting a budget on. Before you act on the drivers slide: did anything change, or did we only watch?

A quarterly review is a wall of green — usage, adoption, satisfaction, all up — until an outsider asks the question the dashboard was built to avoid: did any of this make your decisions better? Silence, because every number was chosen for one property (it can go up) and none for the only one that matters (it can go down and tell the truth). Goodhart's and Campbell's laws name the failure; vanity metrics are the genre. The honest move is the uncomfortable one: pick the number that could make you look bad — one before/after delta on the customer's actual decision error, native, monotonic, posted — and let it grade you. Performix → variance in team performance explained, start vs end; AnyComp → the stated-vs-revealed pay-alignment gap. The number worth tracking is the one you'd be a little afraid to put on the wall. Everything else is decoration with a decimal point.

A slide shows a single confident effect; the data only said "about" so, on a sample thin enough to cover both transformative and noise — and the decision gets made on a number a good week of data from reversing. The answer was never the product; the error bar is. Decision theory settled it sixty years ago: information is worth only what it changes about the decision in front of you (value of information), so the question isn't "what's the answer" but "how much certainty does this decision deserve, and what would it cost to get it." Certainty is a dial with a posted price — and telling a customer when more analysis isn't worth buying is what proves it isn't a sales tactic.

Ask three comp pros whether one company's Senior Engineer equals another's Staff Engineer and you get three defended, uncheckable answers — leveling is an argument, every time, because work has no shared reference. Pantone fixed exactly this for color in 1963: own the reference everyone maps into, not the ink, and give it coordinates so difference becomes a computed number. Jobs can carry coordinates too — kept in separate spaces, honest enough to say “no confident match,” frozen in editions. The irony that opens the market: the survey houses copyright and enforce their structures, which fences them permanently out of the neutral position none of them can occupy — and universal math locates their jobs without ever storing their codes. We own the universal structure; licensed crosswalks stay client-side. Never pull an Adobe.

The job is the primary key of the economy of work: you compare pay, plan hiring, audit representation, and map careers by job — yet in nearly every dataset it's free text two companies fill in differently and no one reconciles. The market chased skills (the attribute) and skipped the job (the entity they attach to). Governments built the codes coarse-but-open; the survey houses built them rich-but-locked; nobody built the canonical architecture that's both. Get the job right — function × level, synthesized to map every source — and pay, skills, supply/demand, representation, and careers all compose onto it. The hilltop went unclaimed because the path up was labeled boring.