How Sure Are You?

The Illusion of Validity and Why the Decisions We Feel Best About Deserve the Most Questions

Nov 05, 2026

Issue 289, November 5, 2026

If you have ever been part of an annual budget cycle, you know how it goes. Sometime in the fall, next year’s forecast gets built. Someone pulls last year’s numbers, someone else adds a growth rate that feels reasonable, each department makes its case, and by the time it reaches the leadership meeting, it has become a slide with one number at the bottom. Everyone looks at it, a couple of people ask questions, and it gets approved. The assumptions line up. The logic holds together. The room is confident.

What rarely comes up in that meeting is last year’s forecast. It was built the same way, by many of the same people, with the same confidence, and odds are it missed the mark. Maybe by a little, maybe by a lot. Nobody gets promoted for pointing that out in November. And next fall the process will start all over again, and the room will feel just as sure as it did the year before.

I have been reading Noise, the book Daniel Kahneman wrote with Olivier Sibony and Cass Sunstein. It is a challenging read, but their ideas apply to so much of personal and professional life. In this instance, it gave a name to something we see happen all too often in organizations, the illusion of validity: the confidence we feel when the information in front of us fits together into a good story, whether or not that story actually predicts anything real or accurate. It lands squarely on things I have been writing about for years: how we make decisions, what information we trust, how poorly we predict, and why the consequences we never saw coming keep showing up anyway.

Our feelings of being sure are real. They just aren’t measuring what we think they are.

The Log and the Wall

Kahneman told the story of where he first noticed it many times, including in a 2011 essay for The New York Times. Many decades ago, during his national service in the Israeli army, he helped evaluate soldiers for officer training. One of the tools was something called the leaderless group challenge. Eight soldiers who didn’t know each other, with numbered tags so the observers could keep track, were given a long log and told to get the whole group over a wall about six feet high without the log or any of them touching the wall.

The observers watched who took charge, who gave up, who helped, who got in the way, and they came away with strong impressions. This one is a natural leader. That one folds under pressure. They wrote up their predictions with real conviction, because the behavior they had just watched seemed to reveal each person so clearly.

Every few months, though, the team got feedback from the officer training school on how their candidates had actually done. The news wasn’t good. Their ability to predict who would succeed was, in Kahneman’s words, negligible. Better than a blind guess, but not by much.

The bad news had no effect at all on how they evaluated the next group. The following week there was a new set of soldiers, a new log, the same wall, and the same feeling of certainty about who was who. They knew, as a general fact, that their predictions were close to worthless. In the moment, watching a particular soldier, they felt sure anyway. Kahneman named it the illusion of validity, and he called it the first cognitive fallacy he ever discovered.

His explanation was simple. Confidence, he wrote, is a feeling, determined mostly by how coherent the story is, not a reasoned estimate of the chances of being right. If the facts line up and point in one direction, we feel confident. Whether those facts have ever predicted the outcome we care about is a separate question, and the feeling doesn’t ask that question.

Rewarding Luck as Skill

It would be comforting to think this is a problem for young army psychologists and not for seasoned professionals with years of experience and results behind them. Kahneman tested that too.

Years later he was invited to speak to a group of investment advisers at a wealth management firm, and beforehand he asked for the data on how each adviser had performed. He had eight consecutive years of results for about twenty-five advisers, the numbers the firm used to decide who got the year-end bonus. If the rankings reflected real skill, an adviser who did well one year should tend to do well the next. So he compared every pair of years, twenty-eight comparisons in all, to see how consistent the rankings were.

The average correlation was .01. In his own words, zero. The firm was rewarding luck as if it were the result of skill.

When he presented this to the executives, there was no sign they disbelieved him. It should have been shocking news, and it simply wasn’t. He came away quite sure the finding was swept under the rug, because the people in that room had spent their careers feeling the judgment they exercised every day. One of them told him that he had done very well for the firm and that no one could take that away from him. And from where he stood, of course, it felt exactly like skill, and the confidence that came with it felt earned.

Philip Tetlock found the same pattern on a much larger scale. Over roughly twenty years he gathered tens of thousands of predictions from 284 people whose job included commenting on or advising about political and economic trends. On the whole, the experts did little better than chance, or as one summary of his work put it, no better than monkeys throwing darts. And the experts in greatest demand by the media tended to do worse than their less famous colleagues. The more confident and compelling the story, the more airtime it earned, and the less likely it was to come true. But they kept getting invited to share their “expertise.”

The illusion of validity isn’t a failure of intelligence or effort. If anything, more knowledge gives us more material to build a convincing story out of. And because we like to create stories, building one becomes our default.

What the Story Leaves Out

So why does the feeling stay so strong when the track record is so weak?

Part of the answer is in how we put stories together. In the original 1973 paper where Kahneman and Amos Tversky first described the illusion of validity, they pointed out that confidence rises with how consistent the information looks. The trouble is that consistent information is very often redundant information. Five indicators that all come from the same underlying source feel like five confirmations, when they are really one fact counted five times. Think of the strategy deck with eight slides of supporting data that, traced back far enough, all start with the same customer survey. The fact that it is all one survey disappears by slide two.

The other part of the answer is what the story winds up leaving out. In Noise, Kahneman, Sibony and Sunstein give the gap between how sure we feel and how accurate we are a blunt name: objective ignorance. Some of it is what could be known but wasn’t gathered. Much of it is what simply cannot be known in advance, because the future hasn’t happened yet. The new hire’s manager quits in month two. A competitor gets acquired. A key customer’s budget is frozen by a decision made three levels above anyone you have ever met. No interview, no forecast, and no business case could have contained those things, and yet each of them can influence and decide the outcome.

Everyone is ignorant in this sense. The problem is that we routinely act as if we aren’t, and the confident story we tell ourselves leaves no room on the page for everything it can’t see. The authors describe an internal signal of completion, the sense that the pieces fit, and that signal is what tells us a judgment is done. The evidence running out has nothing to do with it.

This is where unintended consequences come from. I have written about them for years in this newsletter, in the mid-year piece on unintended consequences in 2022, in how economic interconnectedness turns one decision into many, and in randomness and flukes that change the course of things nobody was watching. The illusion of validity is the other half of that picture. The consequences were always out there. A coherent story made it easy not to look for them because nothing in the story pointed that way.

Where It Lives at Work

The illusion of validity is everywhere in organizational life, and once you see it, it is hard to unsee. It’s in the vendor selection where one demo went beautifully, and the reference calls were all with clients the vendor chose, in the performance review where a single memorable project colors the whole year, and in the reorganization chart that looks so logical on the slide that no one asks how it will feel to the people living inside the boxes.

Kahneman wasn’t dismissive of expertise. He argued that intuition can be trusted under two conditions. The environment has to be regular enough that the future actually resembles the past, and the person has to have had lots of practice with quick, clear feedback on their mistakes. An anesthesiologist gets that feedback in minutes. A veteran fire commander, in the research Gary Klein did and later wrote up with Kahneman, has read hundreds of burning buildings. Those are real experts, and their gut deserves respect given the years of experience they have.

Now put most organizational decisions up against those two conditions. Strategy plays out over years, in markets that change underneath it. A hire’s success shows up long after the interview, mixed in with a dozen other influences. A reorganization’s results arrive slowly, and by the time they do, the people who made the original call have often moved on, so the feedback never reaches them. These are close to the worst possible conditions for learning whether our confidence is deserved, which makes them close to the best possible conditions for the illusion to thrive.

Why Change Efforts Are So Exposed

Nowhere in an organization are all of these conditions present at once the way they are in a change effort.

Start with the business case. A case for transformation is, almost by definition, a coherent story about a future no one has seen yet. The current state is described, the future state is described, and the path between them is laid out in phases with milestones and owners. It has to sound confident, because a case that admits how much it doesn’t know rarely gets funded. So the people writing it smooth the edges, and the people approving it read the smoothness as evidence. The more polished the story becomes, the more sure the room feels, and the fewer people ask what the plan can’t know.

Look next at where the confidence comes from. The readiness assessment, the steering committee, the pilot group and the sponsor’s own read of the organization often draw on the same small circle of people, the ones closest to the work and those most invested in it. Four sources that agree look like strong evidence. Four sources drawn from the same well are closer to one source repeated four times, which is exactly the trap Kahneman and Tversky described in 1973, the strategy deck echo with a budget attached.

There is also the feedback problem. Most organizations don’t run the same transformation twice. A new system, a merger or a restructuring is usually a first for the people leading it, so there is no record of similar calls to check their confidence against. Results arrive slowly and tangled up with everything else going on in the market. By the time anyone can say whether the original judgment was sound, the sponsor may have moved on, and the story of what happened gets written by whoever remains and holds the reins. That is about as far from the anesthesiologist’s world as it gets.

And what routinely gets overlooked is the people. A plan can schedule the change. It can’t schedule the transition people go through in response, which runs on a different clock entirely. The illusion does its quietest damage here, because the plan assumes adoption will follow the timeline simply because the timeline is logical. If you have ever lived through a software rollout that was “complete” on the project plan while half the office kept a spreadsheet on the side, you have seen this up close. It happens nearly all the time.

Once the effort is underway, the illusion changes its costume. Early signs that don’t fit the story get filed as resistance or a communications problem rather than treated as information, the same misread that happens whenever friction gets mistaken for pushback. Status reports stay green because each workstream is on track by its own measure, even when the people inside the work can feel the whole thing drifting. Tina Berger and I talked about a version of this on The Human Factor Podcast: a project can be on scope, on schedule, and on budget and still miss the reason it existed. The plan was valid. It just wasn’t predictive at all.

None of this means change leaders set out or intend to be careless. They are working inside a system that, at the very moment change and transformation most need honest questions, rewards and embraces the most confident-sounding answers. And the people with the earliest and clearest signals that something is off are often the furthest from the room where the story is being told.

The Most Certain Person in the Room

There is one more reason the illusion survives, and it has less to do with how our minds work than with what organizations embrace and reward.

Most organizations prize conviction. The leader who says “I’m confident this is the right call” sounds stronger than the one who says “I think this is our best option, and here is what would change my mind.” Boards want certainty. Teams want direction. Investors want a clear story. I looked at that pull in The Risk of Certainty back in 2024, and earlier this year I wrote about the confidence calibration problem, how the person who knows least can speak with the most authority. The illusion of validity is a related problem, and in some ways a harder one. It doesn’t need anyone to be uninformed. It catches the well-informed, the experienced and the careful, all nodding together, because the story is good.

And the more senior the room, the harder it is to be the one who asks how often this kind of decision actually works out. Last week I wrote about the costumes we wear at work, and how leaders in particular cover their doubts. Put those two things together and you get a meeting where everyone is privately a little less sure than they look, and the decision is made at the level of confidence everyone is performing. That is where decision theater comes from.

None of this means we should stop deciding. Decisions have to be made with the information available, and waiting for certainty is its own kind of mistake. But the point is narrower than that. The feeling of being sure isn’t evidence. It’s a report on how tidy, believable and defensible the story is.

Back to the Budget Meeting

So, go back to that budget meeting. Nobody in that room intended to be careless. They had numbers that fit together, assumptions that agreed with each other and a story about next year that was believable, reasonable and made sense to everyone. That is exactly what the soldiers carrying the log gave Kahneman’s team, and exactly what eight years of returns gave the wealth managers. Each time, the story was good, the feeling was strong and the feeling had very little to do with what happened next.

The one question that almost never gets asked in that meeting is how close we came last year. It is the simplest track record any organization has, and it sits in a file that nobody but the CFO opens while the new forecast is up on the screen.

What the illusion of validity asks of us isn’t less confidence. It asks us to notice what our confidence consists of. Is it made of a track record, of the kind of quick and honest feedback that actually teaches us something? Or is it made of a story that simply hangs together well, built from information that mostly agrees with itself, with no room on the page for what can’t be known?

Think about the decisions you felt surest about this past year. The hire, the forecast, the vendor, the reorganization, the plan you defended in the meeting. How did they turn out? And if you are honest, how much of that certainty came from evidence, and how much came from how good the story sounded?

Research Sources

1. Kahneman, Daniel, Sibony, Olivier, and Sunstein, Cass R. (2021). Noise: A Flaw in Human Judgment. Little, Brown Spark. Source of objective ignorance, its two parts (what could be known but was not gathered, and what cannot be known), and the argument that people making predictions deny it.

2. Kahneman, Daniel (2011). Don’t Blink! The Hazards of Confidence. The New York Times, October 19, 2011, adapted from Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011). Source of the leaderless group challenge, the “negligible” predictive ability, the unchanged confidence with each new group, the name and “first cognitive fallacy,” confidence as a feeling determined by the coherence of the story, the wealth adviser study (about 25 advisers, eight years, 28 correlations averaging .01), and the two conditions for trustworthy intuition.

3. Kahneman, Daniel, and Tversky, Amos (1973). On the Psychology of Prediction. Psychological Review, volume 80, number 4, pages 237 to 251. The paper that first described the illusion of validity, and the point that consistent information is often redundant information.

4. Kahneman, Daniel, and Klein, Gary (2009). Conditions for Intuitive Expertise: A Failure to Disagree. American Psychologist, volume 64, number 6, pages 515 to 526. Source for the firefighter example of trustworthy intuition.

5. Tetlock, Philip E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press. The 284 experts, roughly twenty years of predictions, the comparison with chance, and the finding that media-favored experts did worse, as summarized in Stanley Zarowin, Beware Expert Predictions, Journal of Accountancy, March 2006.

Discussion about this post

Ready for more?