One morning in August, the social science reporter for National Public Radio, a man named Shankar Vedantam, sounded a little shellshocked. You couldn’t blame him.
Like so many science writers in the popular press, he is charged with reporting provocative findings from the world of behavioral science: “. . . and researchers were very surprised at what they found. The peer-reviewed study suggests that [dog lovers, redheads, Tea Party members] are much more likely to [wear short sleeves, participate in hockey fights, …] than …
I’m just making these up, obviously, but as we shall see, there’s a lot of that going around.
On this August morning Science magazine had published a scandalous article. The subject was the practice of behavioral psychology. Behavioral psychology is a wellspring of modern journalism. It is the source for most of those thrilling studies that keep reporters like Vedantam in business.
Over 270 researchers, working as the Reproducibility Project, had gathered 100 studies from three of the most prestigious journals in the field of social psychology. Then they set about to redo the experiments and see if they could get the same results. …
These 100 studies had cleared the highest hurdles that social science puts up. They had been edited, revised, reviewed by panels of peers, revised again, published, widely read, and taken by other social scientists as the starting point for further experiments. Except . . .
The researchers, Vedantam glumly told his NPR audience, “found something very disappointing. Nearly two-thirds of the experiments did not replicate, meaning that scientists repeated these studies but could not obtain the results that were found by the original research team.”
Peer review is supposed to be the imprimatur that this is, to use the current bit of triteness, “settled science”. Sadly, it’s become a hollow idol. And while the “social sciences” seem particularly susceptible to error, fraud-for-funding, fame-seeking, axe-grinding, etc., other sciences are also troubled by like fraud and error. Were this just a tempest in an academic teapot, it would not matter too much, but politicians are committing nations’ economies to “settled science” (= “Global Climate Change”) that may be nothing more than a casserole of fraudulent analysis of selected & cooked “data”.