Tortured Data

“Beware of testing too many hypotheses; the more you torture the data, the more likely they are to confesss, but confession obtained under duress may not be admissible in the court of scientific opinion. ”
—Stephen M. Stigler, “Testing Hypotheses or Fitting Models?” (1987)

That is useful advice for everyone, but even moreso a warning to those seeking to massage cherrypicked data to tell just-so stories. In particular, a few HBDers (human biodversity advocates) can be quite brilliant in their ability to speculate and gather data to support their speculations, while ignoring data that contradicts them. This is seen in the defense of race realism, a popular ideology among HBDers.

Some HBDers and other race realists are so talented at speculating that they come to treat their ideologically-driven interpretations as factual statements of truth, even when they deny this is the case. Just as they deny the consequences of such ideologies being enforced for centuries through social control, political oppression, and economic inequality. A result can be misinterpreted as cause, an easy error to make when evidence for direction of causation is lacking. It leaves the field open to self-serving bias.

When one starts with a hypothesis that one assumes is true, it’s easy to look for evidence to support what one already wants to believe. There are few people in the world who couldn’t offer what they consider evidence in support of their beliefs, no matter how weak and grasping it might appear to others. This is even easier to accomplish when looking for correlations, as anything can be correlated with many other things without ever having to prove a causal connection, and it’s easy to ignore the fact that most correlations are spurious.

None of that matters to the true believer, though. Torturing the data until it confesses is the whole point. As in real world incidents of torture, the validity of the confession is irrelevant.

2 thoughts on “Tortured Data

  1. The other day a friend pestered me into trying to play Donkey Kong. I sucked when it came out, and still suck. When I was killed immediately after he handed me the controller, I tried to play again, but I kept dying in the exact same way, three or four times in a row. Only eventually did it become clear to me that the demo was playing at the beginning, imitating a player; it wasn’t me playing. I’d been ‘playing’ the demo, getting ‘killed’ at the same place each time, while I whined and remonstrated about the game and my skills.

    The scientific method is an inoculation against our inherent and untiring prediction engine, our cognitive biases, and our brute stupidity. The proper way to employ the scientific method is to prove the null hypothesis, i.e., prove an idea wrong– not prove one’s little favored idea or hypothesis. What that generally means is that the task of the scientist is to disprove one’s hypothesis; what results after successful experimentation is then a much healthier and broader statement, that “we weren’t yet able to prove my idea was wrong.” With the attitude that statement implies, the requirement for replication seems a given, thereby encouraging, at best, a healthy meander toward truth.

    If we took the spirit of this perspective to heart in our normal lives, we’d have much less trust of our senses, our logic, and our abstract belief scaffolds within politics, religion, romance, and ethics. It would help with bias, yes, but it would also underscore for us repeatedly in our lives how complex the real world is– how beautiful it is, really– and how impossible it is for us to grasp well anything other than bits and pieces.

    • That is an amusing anecdote. Thanks for sharing. It made me laugh. I’m not the most talented person when it comes to video games and I can easily imagine the scenario you describe, although I enjoyed Donkey Kong as a child.

      I agree with you about proving ideas wrong. That is more in line with how I think. I prefer to clear away what is impossible or implausible. But that still leaves many alternative explanations. I love speculation. The trick is to realize you are speculating. It’s easy to disprove simple notions of race realism. What is much harder to accomplish is testing and proving a hypothesis that makes sense of the complex reality that defies such simple notions.

      That is why I often end on a note of intellectual humility, just stating what isn’t known and maybe isn’t even knowable. I’m more motivated by a sense of curiosity and wonder than anything else. I doubt we are close to understanding most things about the world we live in, including our own humanity. More important than seeking the right answer is asking a useful or at least interesting question.

      This has come up recently in my criticisms of certain debates. An example is the nature vs nurture disagreement. The entire frame seems wrong to me, pointless and irrelevant. The very dualistic view doesn’t make any sense. There is no nature that is opposite a nurture. It’s bullshit. I have no interest in debating it, as there is so many more interesting discussions to have. I don’t mind conflict. But if we must argue, then make sure it is something worthy of our time and effort.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s