The Error Bar Is the Product
The number was wrong, and the worse problem was that it arrived without a way to know that.
A team had run the analysis, and the slide said the effect was large — invest here, the data says so. The data did not say so, exactly; it said about so, on a sample thin enough that "about" covered both "transformative" and "noise." But the slide showed a single confident figure, because that's what a slide shows, and the room treated the figure as the finding. The decision got made on a point estimate that was one good week of data away from pointing the other direction. Nobody was lied to. The uncertainty was simply left off, the way it almost always is — and the most important thing about the number went unsaid.
This essay is about what was actually missing in that room, and what it implies for how analytics should be sold and bought. The claim is uncomfortable for an industry that sells answers: the answer is not the product. The error bar is the product.
They say the deliverable is the answer
The default contract of analytics is "we give you the number." The dashboard resolves to a figure; the model outputs a prediction; the report concludes. The implicit promise is certainty — you came with a question, you leave with an answer — and the error bar, if it survives at all, is a footnote in a smaller font, because uncertainty reads as weakness and a confident slide closes the meeting.
But a decision-maker doesn't actually need the number. They need to make a better decision, and a number changes a decision only to the degree it's trustworthy enough to act on. A point estimate with a credible interval wide enough to flip the recommendation is not a smaller version of an answer — it's a different object, and treating it as an answer is how confident, well-meaning teams walk off cliffs. The deliverable that matters isn't the dot on the chart. It's how wide the band around the dot is, and which way the decision tips across that band.
A century-old idea the slide forgets
Decision theory worked this out long before the dashboard existed, and gave it a name: the value of information. The worth of a measurement is not how interesting it is — it's how much it changes the expected outcome of the decision in front of you.1 A perfectly precise number about something that wouldn't change your choice is worth nothing. A rough number that flips a million-dollar call is worth a great deal. Information has value only relative to a decision, and only to the extent it moves you across the line where your action would change.
The practical engine built on that idea is blunt about what to measure: not what's easy, not what's impressive, but what reduces the uncertainty that's actually holding the decision hostage.2 Most measurements an organization commissions fail this test — they sharpen a number that was already precise enough, or that wouldn't change the call no matter how it came out. The error bar is what tells you which measurement is worth buying, because the error bar is what's standing between you and the decision.
There's a hard ceiling under all of this, and it deserves to be a slogan: you cannot validate your way past an unreliable instrument — a measure correlates with the real thing no better than the square root of its own reliability.3 Certainty isn't free, and it isn't infinite. It has a price and a cap, and pretending the point estimate is the whole story hides both.
Certainty is a dial, not a switch
Here is the reframe that changes the transaction. If the real product is the error bar, then the question stops being "what's the answer" and becomes "how much certainty does this decision deserve, and what would it cost to get it." Certainty is a dial with a posted price: this much confidence costs this many more observations, this much more instrumentation, this much more of your time — and here, honestly, is whether the decision in front of you is worth turning the dial.
That makes the honest version of an analytics relationship look very different from "here's your number." It looks like: here's the read we can give you now, here's the band around it, here's which way your decision tips across that band — and here's what tightening it would cost versus what it's worth to you. Sometimes the answer is "the band is already tight enough; act, don't spend more." Sometimes it's "this is too close to call on what you have; here's the cheapest measurement that would settle it." Both are the product. The first one — telling a customer not to buy more analysis because it wouldn't change their decision — is the one that proves the second isn't a sales tactic.
This is also the discipline that keeps a tool honest about itself. A prediction engine that hands over a single confident number invites the reader to mistake the dot for the truth. One that hands over the dot and the band and the value of narrowing it is telling the reader what they're actually holding: one read, often wrong at the edges, improvable at a knowable cost. That's not hedging. It's the difference between a measurement and a guess with good production values.
What you can finally do with it
Go back to the room with the large-looking effect. In the answer-is-the-product world, they invest on a figure that was a coin-flip away from reversing, and they find out which side it really landed on after the money's spent. In the error-bar-is-the-product world, they see the band, see that it spans "transformative" and "noise," and get a straight choice: act now and accept the risk the band describes, or spend a defined, priced amount to narrow it before committing — with an honest read on whether that spend is worth it for this decision. The point estimate didn't change. What changed is that they could see how much to trust it, and what trust would cost.
The industry sells answers because answers are easy to put on a slide and uncertainty is not. But the answer was never the scarce thing — the confidence was, and confidence is exactly what gets dropped to make the slide look clean. Put the error bar back at the center, price the dial, and tell customers honestly when more certainty isn't worth buying, and you've stopped selling numbers and started selling the only thing a decision-maker actually needs: a measured sense of how much to believe.
This is a piece in the People Analyst program — the doctrine under the value-of-information bridge that closes every estimate we produce, and the certainty ladder in the story-card system (each finding pre-bound to its model, its instrument, and the cost of tightening it). It's the commercial honesty behind the client-insight readout's "honest rider" and a sibling in posture to You Can't Compute With a PDF, which is about evidence that arrives with its uncertainty attached. No figures are invented here.
Footnotes
-
Ronald A. Howard, "Information Value Theory," IEEE Transactions on Systems Science and Cybernetics, 2 (1966): 22–26 — the value of information is defined by how much it improves the expected outcome of a decision; information about something that wouldn't change the decision has zero value. Builds on the decision-analytic tradition (Raiffa & Schlaifer, Applied Statistical Decision Theory, 1961). ↩
-
Douglas W. Hubbard, How to Measure Anything: Finding the Value of Intangibles in Business (Applied Information Economics) — measure what reduces the uncertainty bearing on a real decision; most commissioned measurements fail this test by sharpening already-precise or decision-irrelevant quantities. ↩
-
The attenuation ceiling from classical test theory: a measure's correlation with any criterion is bounded above by the square root of its reliability (Lord & Novick, Statistical Theories of Mental Test Scores, 1968). Certainty has both a price and a cap; see also The Reliability Problem. ↩