This “peer review” stuff is NOT all it’s trumped up to be. An excerpt from my Front Page blog titled “Studies show . . .”
( Republican Operative / Studies show . . . ):
This is often seen as one of the two essential elements of a foundation that renders “proof” that a paper is factual. The other element of course is “p-value”. If a paper both passes the threshold of p-value AND peer review, it is often presented as “fact”.
Again, not so fast.
“Peer review” is seen as the Gold Standard (or sacred cow, if you will) of publishing. I wouldn’t disagree that it is something to be considered, and it is something that I always want to see (more on that in a bit), just like I want to see what p-value was calculated (AND what the null hypothesis was specifically), but neither peer review nor p-value represents the WHOLE picture.
Pseudo-intellectuals often hang their hat on p-value and peer review, as though that was all that was necessary to validate the study. Not so fast, again.
Let’s look at this “peer review” business a little closer than just the buzz phrase that the unwashed masses view as this sacred cow. I mean, when they hear that phrase . . . that’s it, period . . . no more scrutiny necessary. After all, if it’s been reviewed and “approved” by scientists that are much more knowledgeable in that field than me . . . who am I to question their judgment?
You guessed it . . . NOT SO FAST.
That is exactly why the pseudo-intellectuals shroud these things in mystery. THEY DON’T WANT YOU TO THINK YOU HAVE THE ABILITY TO SCRUTINIZE THESE THINGS.
First of all, the phrase “peer review” as it applies to studies is actually a misnomer. Those that examine studies for the accuracy and quality necessary for publishing in journals are known as “REFEREES”, and the study/paper is REFEREED, NOT PEER REVIEWED. The phrase “peer review” applies to the scrutiny necessary for FUNDING OF GRANTS.
If a scientist submits an application for a GRANT, to say, the National Science Foundation, or the National Institutes of Health, then that application is PEER REVIEWED. If a scientist submits a study/paper for publication in, say, the journal Nature, then that article submission is REFEREED.
But the phrase “peer review” has become recognized by the unwashed masses as what’s done to studies/papers in journals, so we’ll use “peer review” in that context (even though it’s not accurate . . . but I’m not going to stand on nitpicking technicalities).
Peer review became popular only in the past few decades, although it was used going all the way back to the mid 1600′s. That recent popularity is NOT based on any new notion that peer review enhances the credibility of a paper, but rather relieves overworked journal editors of the burden of reviewing thousands of papers (see my fourth and last “TRIVIA” item below.)
Now let’s take a look at some of the trivia, flaws, and criticisms of peer review.
TRIVIA: Watson and Crick’s breakthrough on DNA was NEVER subjected to peer review.
TRIVIA: Many papers that have been cited in work that won Nobel Prizes were originally rejected by peer review.
TRIVIA: Edward Jenner’s paper on vaccination for smallpox was rejected by some peer review people.
TRIVIA: In 2013, some 10,952 papers were submitted to the journal Nature. In 1997, there were only 7, 680 submissions.
Reviewers seem biased in favor of authors from prestigious institutions (the “halo effect”). In a study in which papers that had been published in journals by authors from prestigious institutions were retyped and resubmitted with a non-prestigious affiliation indicated for the author, not only did peer reviewers mostly fail to recognize these previously published papers in their field, they recommended rejection.
The chairman of the investigating committee of the Royal Society told a British newspaper in 2003, “We are all aware that some referees’ reports are not worth the paper they are written on. It’s also hard for a journal editor when reports come back that are contradictory, and it’s often down to a question of a value judgment whether something is published or not.”
He also pointed out that peer review has been criticized for being used by the scientific establishment “to prevent unorthodox ideas, methods, and views, regardless of their merit“.
In one study, researchers deliberately inserted errors into a manuscript, and referees did NOT detect some of them.
The deputy editor of the Journal of the American Medical Association once said, “There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.”
The editor of the British medical journal The Lancet once said: “The mistake, of course, is to have thought that peer review was any more than just a crude means of discovering the acceptability . . . not the validity . . . of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”
Competitors are often chosen as peer reviewers. Might a competitor be inclined to unfavorably review a submission and then steal the idea for him/her self? The irresistible opportunity to put a spoke in a rival’s wheel?
Peer review in journals assumes that the article reviewed has been honestly prepared and the process is not designed to detect fraud. It assumes ALL scientists are integral, IOW not subject to human flaws. A peer reviewer must preserve scholarly integrity by rising above the three deadly sins of intellectual life: envy, favoritism, and the temptation to plagiarize.
Peer review is under reconsideration even within the heart of establishment scientific publishing.
But the most damaging criticism of peer review may be that which is exemplified by the cloning hoax of Hwang Woo Suk. (BBC NEWS | UK | Magazine | The pressure to hoax)
Hwang submitted a paper to the journal Science which was later found to be hugely fraudulent. Of course, it passed peer review. It could NOT have been duplicated simply because the results were totally fabricated. In this case, DUPLICATION, NOT peer review, would have uncovered the hoax. To paraphrase what I said earlier, one disadvantage to the modern obsessive attention on PEER REVIEW is the emphasis it places on that peer review TO THE EXCLUSION OF CONFIRMATION BY REPEATED EXPERIMENTS. Often, a paper will be perceived as not needing duplication if it “passes” the PEER REVIEW threshold.
Of course, no one should expect a perfect system, or condemn peer review as a whole for its occasional failures. and I’m not doing that. Peer review is like democracy was to Churchill: “the worst form of government except all those other forms that have been tried from time to time”
What I’m pointing out is, much like p-values, it ain’t all it’s trumped up to be. It forms ONLY part of the picture, and to hang your hat on it shows me that your analysis of a study is extremely flawed.
Oh . . . I almost forgot. I had said earlier that peer review was something I would like to see. Most journals maintain peer reviewers in anonymity, and the identity of a peer reviewer is a closely guarded secret, generally held ONLY by the journal chief editor. Peer reviewer identities are not normally published (there are exceptions). Consequently, one CANNOT normally see who a peer reviewer was, whether or not he/she is a competitor, and perhaps more importantly, WHERE he/she draws financial support from. The only thing you DO see is that the article got published, which means it passed peer review.
The anonymity of peer reviewers contributes to the “Oz-behind-the-curtain” effect: Reviewers that work anonymously have a greater opportunity to act arbitrarily. The REVIEWEE has no comparable curtain to stand behind. Basically, the REVIEWER can take potshots at the REVIEWEE with NO accountability.