The Number That Can Sink You

The quarterly review is a wall of green. Usage is up. Adoption is up. The satisfaction score ticked up two points. Logins per week, up. Every chart slopes the right way, and the room feels good about it — until someone who doesn't work there asks the question the dashboard was built to avoid: did any of this make your decisions better?

Silence. Not because the answer is no. Because the question was never being measured. Every number on the wall was chosen for one property — that it could be made to go up — and none of them for the only property that matters, which is that it could go down and tell you the truth.

They say show traction

The metric culture rewards the rising line. Pick your KPIs, build the dashboard, show traction to the board, watch the numbers climb. It is the water everyone swims in, and it has a failure mode so well documented it has two laws named after it. Goodhart's: once a measure becomes a target, it stops being a good measure.¹ Campbell's, the social-science twin: the more any indicator is used to make decisions, the more it gets gamed and the more it corrupts the thing it was supposed to track.² A number chosen because it goes up will, reliably, be made to go up — by improving the thing, if you're lucky, or by optimizing the number, which is easier.

This is what vanity metrics are, the term Eric Ries gave the genre: totals that rise no matter what, flatter the team, and answer no decision.³ Cumulative signups. Page views. The score that only moves one direction because the instrument was designed so it couldn't embarrass anyone. Even the famous one number — Net Promoter — earns its keep only when it is hard to game and tied to something real; treated as a target to lift, it becomes another line that goes up while the business goes sideways.⁴

A metric that can't sink you isn't measuring you

Here is the principal issue. The test of a real metric is not whether it can rise. It is whether it can fall and indict you. A number that can only go up, or that you can always nudge up with effort unrelated to the customer's outcome, is not measuring your product. It's decorating it.

The honest move is the uncomfortable one: pick the number that could make you look bad. Commit, per product, to a single before-and-after delta on the thing the customer actually hired you to improve — and specifically on their decision error, the gap between the call they'd make without you and the better call your work makes possible. One number. Native to what the product does. Pointed in a direction where worse is a real, reportable outcome. That is the number you grade yourself by, precisely because it can sink you.

What that looks like when you mean it

It is easy to nod at this and hard to do it, so make it concrete.

A team-performance diagnostic could report adoption, or satisfaction, or how many cards it generated — all green, all vanity. The number that can sink it is different: how much of the variance in team performance it can actually account for, measured at the start and again later, start versus end. If that explained variance doesn't move, the product didn't work, and the metric says so out loud. A compensation engine could report how many pay structures it built. The number that can sink it is the gap between what the organization says it pays for and what it actually pays for — and whether that gap closed. In both, the metric is chosen so that failure has somewhere to show up. That is the whole discipline.

It pairs with an older idea about what information is even worth: a number is only valuable if it would change a decision in front of you.⁵ So the North Star is not just any honest number — it's the honest number attached to the decision the customer is actually making. Move that, and you've earned the engagement. Fail to move it, and no amount of green on the rest of the wall should save you.

The system, graded the same way

This is how the whole system of products here is meant to be judged — not by a shared dashboard, but by a shared rule: every product declares one before/after delta on the customer's decision error, monotonic, posted, the number we let grade us. It's the same posture as the rest of the philosophy on this site. The proof graph makes every claim checkable; this makes every product falsifiable. Both are bets that the durable position is the one willing to be caught wrong.

And it costs something, which is why most companies don't do it. A number that can sink you can, in fact, sink you — in a board meeting, in a renewal conversation, in public. Vanity metrics exist precisely to spare everyone that risk; they are chosen for comfort, and comfort is their entire function. But a metric that cannot deliver bad news cannot deliver news at all. The number worth tracking is the one you'd be a little afraid to put on the wall. Everything else is decoration with a decimal point.

---

The North Star doctrine for the portfolio: each product commits to one honest before/after number on the customer's decision error — the number that can indict it — rather than a wall of metrics chosen to rise. A companion to The Error Bar Is the Product (how much to trust a number) and Measured Against Reality (judging by state, not image); the falsifiability sibling of the proof graph. No claim here is unsourced.

Footnotes

1. Charles A. E. Goodhart, "Problems of Monetary Management: The U.K. Experience" (1975); the now-canonical compression — "when a measure becomes a target, it ceases to be a good measure" — is due to Marilyn Strathern, "'Improving Ratings': Audit in the British University System," European Review 5, no. 3 (1997): 305–321. ↩

2. Donald T. Campbell, "Assessing the Impact of Planned Social Change" (1976) — "the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." ↩

3. Eric Ries, The Lean Startup (Crown Business, 2011) — the distinction between vanity metrics (totals that always rise and inform no decision) and actionable metrics (tied to a specific cause and a decision). ↩

4. Frederick F. Reichheld, "The One Number You Need to Grow," Harvard Business Review, December 2003 — the Net Promoter idea; included here as the canonical "single metric," whose usefulness depends on its being hard to game and tied to real growth rather than lifted as a target. ↩

5. The value-of-information principle from decision analysis: information has value only insofar as it can change the decision at hand. Ronald A. Howard, "Information Value Theory," IEEE Transactions on Systems Science and Cybernetics 2, no. 1 (1966): 22–26. ↩