If I ask you to tell me the likelihood of a particular major league batter getting a hit in a particular at-bat next week, against a pitcher who has yet to be named, you’d probably say the odds are 25-35%. And you would be about right.
If I asked you how well a particular investor is likely to perform compared to the market averages, long term, if he actively manages his own money, you’d say “worse than the average,” and you’d probably be right.
It’s important to know the odds for things. We make most of our major decisions based on probability. And that’s why I find it so interesting that we (the general public) don’t know the odds for science. Put another way, how often is a peer-reviewed scientific study later discovered to be wrong? How often is a hypothesis by a credible scientist proved right? And how often does a major theory get revised or even discarded? If the general public doesn’t know the answers to those questions, how screwed are we?
I started thinking about this when I saw a news story (that is now behind a pay wall) about the scary number of peer-reviewed articles in scientific journals that turned out to be fraudulent or simply wrong. I say “scary” because the news left out the probability. Was it 1% of the studies that were bad or 50%? I don’t know. And that makes a big difference for how much trust I put in the next peer-reviewed study. If scientists someday claim they have studies showing that eating carrots will double my life expectancy, do I buy a bushel and start gnawing on them or do I shrug?
What about a typical hypothesis? I assume a typical hypothesis formulated by a credible scientist is shown to be wrong most of the time. But are we talking about 55% of the time or 99%? Or am I wrong that it’s wrong most of the time? It matters to me because I might want to start eating carrots before the studies are done, based on the hypothesis alone, just in case.
And how about a typical theory? How often are theories wrong, or at least wrongish?
I pause here to remind readers that the word “theory” is often used in casual conversation to mean something like an educated guess. But in the scientific context, it means something closer to “proven true by experiments.” The technical definition is unwieldy (sorry, bearded taint who is angrily reading this) and would be a distraction from my point today.
The great thing about science is that it allows today’s truth to be revised to a new and better truth if new information says so. But how often does that happen for something as well-tested as a major theory? Not often, I assume. But does not often mean 1% of theories, or does it mean 20% if you wait a few generations? Beats me.
Judging scientific probability is a question of life and death. Consider climate change science. If a reasonable person judged the likelihood of climate change science being correct, using nothing but a lifetime of observing science stories in the media, he would probably conclude that the claimed risk is overblown or non-existent. The media concentrates on new and speculative stories about science, and stories about frauds and mistakes, so one can easily get the impression that science has a batting average on par with astrology. You don’t see many stories about theories that are rock solid. That isn’t news, so our sense of the track record for science is skewed.
I use climate change as my example because it fits the question so well. Personally, I’m spring-loaded to agree with the vast majority of scientists who say it’s a huge problem and that humans are a big part of it. On the other hand, the climate models remind me of economic models because of the many variables, and we know how accurate those are. For me, the tie-breaker would be some useful information about the track record of scientists when a complicated model is involved.
So my questions for today are these:
1. How often is a peer reviewed study wrong?
2. How often is a credible scientist’s hypothesis wrong?
3. How often is a scientific theory later proven wrong?
4. How often does a complicated scientific model make correct predictions?
If you don’t know the answers to those questions, your opinions on our most important science questions are worthless at best, and deadly at worst. And I’m right there with you.