Since csbrown brought up “peer review” in his post #15, and since his charge that @Pappadave doesn’t understand “peer review” would be laughable if it weren’t so pathetic, let’s take a close look at the peer review sacred cow:
First of all, the phrase “peer review” as it applies to studies is actually a misnomer. Those that examine studies for the accuracy and quality necessary for publishing in journals are known as “REFEREES”, and the study/paper is REFEREED, NOT PEER REVIEWED. The phrase “peer review” applies to the scrutiny necessary for FUNDING OF GRANTS.
If a scientist submits an application for a GRANT, to say, the National Science Foundation, or the National Institutes of Health, then that application is PEER REVIEWED. If a scientist submits a study/paper for publication in, say, the journal Nature, then that article submission is REFEREED.
But the phrase “peer review” has become recognized by the unwashed masses . . . and csbrown . . . as what’s done to studies/papers in journals, so many editors use the phrase “peer review” and we’ll use “peer review” for purposes of this conversation (even though it’s not accurate).
And csbrown “understands” peer review? Give me a break . . .
Peer review became popular only in the past few decades, although it was used going all the way back to the mid 1600′s. That recent popularity is NOT based on any new notion that peer review enhances the credibility of a paper, but rather relieves overworked journal editors of the burden of reviewing thousands of papers (see my fourth and last FACT below.)
Now let’s take a look at some of the less well known facts, flaws, and criticisms of peer review. (And I suspect some of these are what @Sunsettommy was referring to when he said “peer review is a vastly overrated process”)
FACT: Watson and Crick’s breakthrough on DNA was NEVER subjected to peer review.
FACT: Many papers that have been cited in work that won Nobel Prizes were originally rejected by peer review.
FACT: Edward Jenner’s paper on vaccination for smallpox was rejected by some peer review people.
FACT: In 2013, some 10,952 papers were submitted to the journal Nature. In 1997, there were only 7,680 submissions.
Reviewers seem biased in favor of authors from prestigious institutions (the “halo effect”). In a study in which papers that had been published in journals by authors from prestigious institutions were retyped and resubmitted with a non-prestigious affiliation indicated for the author, not only did peer reviewers mostly fail to recognize these previously published papers in their field, they recommended rejection.
The chairman of the investigating committee of the Royal Society told a British newspaper in 2003, “We are all aware that some referees’ reports are not worth the paper they are written on. It’s also hard for a journal editor when reports come back that are contradictory, and it’s often down to a question of a value judgment whether something is published or not.”
He also pointed out that peer review has been criticized for being used by the scientific establishment “to prevent unorthodox ideas, methods, and views, regardless of their merit“.
In one study, researchers deliberately inserted errors into a manuscript, and referees did NOT detect some of them. (Oooops . . . “peer reviewers” in csbrown’s usage.)
The deputy editor of the Journal of the American Medical Association once said, “There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.”
The editor of the British medical journal The Lancet once said: “The mistake, of course, is to have thought that peer review was any more than just a crude means of discovering the acceptability . . . not the validity . . . of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”
Competitors are often chosen as peer reviewers. Might a competitor be inclined to unfavorably review a submission and then steal the idea for him/her self? The irresistible opportunity to put a spoke in a rival’s wheel?
Peer review in journals assumes that the article reviewed has been honestly prepared and the process is not designed to detect fraud. It assumes ALL scientists are integral, IOW not subject to human flaws. A peer reviewer must preserve scholarly integrity by rising above the three deadly sins of intellectual life: envy, favoritism, and the temptation to plagiarize. That is NOT some kind of a “global conspiracy theory”, it is just a simple account of human nature. Are not scientists part of the human race, or are they angelic?
Peer review is under reconsideration even within the heart of establishment scientific publishing. And that is NOT because of the “internet”:
NONE of the items (criticisms of peer review) I listed above have ANYTHING to do with the Internet.
But the most damaging criticism of peer review may be that which is exemplified by the cloning hoax of Hwang Woo Suk.
Hwang submitted a paper to the journal Science which was later found to be hugely fraudulent. Of course, it passed peer review. It could NOT have been duplicated simply because the results were totally fabricated. In this case, DUPLICATION, NOT peer review, would have uncovered the hoax. One of the many disadvantages to the modern obsessive attention on PEER REVIEW (and csbrown seems to have that obsession . . . more on his denial of that in a second) is the emphasis it places on that peer review TO THE EXCLUSION OF CONFIRMATION BY REPEATED EXPERIMENTS. Often, a paper will be perceived as not needing duplication if it “passes” the PEER REVIEW threshold.
Finally, on this notion of peer review, most journals maintain peer reviewers in anonymity, and the identity of a peer reviewer is a closely guarded secret, generally held ONLY by the journal chief editor (who is the one who decides what gets published, NOT the scientific community). Peer reviewer identities are not normally published (there are exceptions). Consequently, one CANNOT normally see who a peer reviewer was, whether or not he/she is a competitor, and perhaps more importantly, WHERE he/she draws financial support from. The only thing you DO see is that the article got published, which means it passed peer review.
The anonymity of peer reviewers contributes to the “Oz-behind-the-curtain” effect: Reviewers that work anonymously have a greater opportunity to act arbitrarily. The REVIEWEE has no comparable curtain to stand behind. Basically, the REVIEWER can take potshots at the REVIEWEE with NO accountability.
So much for hanging your hat on peer review.
Now let’s get ready for the “I never said peer review was weighted heavily in my evaluations . . . or that it was perfect . . .” dance.
No, it’s just the FIRST thing you look at:
I give “peer review” little, if any, weight. It is definitely not the “first” thing I look at.
@csbrown28 gives it more weight than it warrants:
Wait a minute . . . wait a minute . . .
You mean you’re hanging your hat on the education system we all know is heavily leftist, especially at institutions of higher “learning”?
Lemme’ get this straight.
When Trump makes a mistake, it’s a lie . . . but when your AGW guys make a mistake, it’s “genuine”?
Oh, before I forget, I was hoping Sunsettommy would show up. His rebuttals of the AGW . . . people . . . are good reading. The guy knows his stuff, and can run circles around the AGW . . . people.