Peer Review is Not What You Think

Scientific Peer-Review is a Lightweight Process
11/30/09 - ChicagoBoyz by Shannon Love

[edited] "Peer Review" says nothing about conclusions. It is the fate of most scientific papers to be proven completely wrong.

Peer review protects a journal’s reputation. The journal hires experts to check for basic errors in math or methodology, along with grammar and spelling. It offloads responsibility for publishing bad papers onto anonymous scientists. It is a form of blame-passing that everyone would like to use. It does not confirm or refute experimental or theoretical conclusions.

The anonymous and secret peer review process is not part of actual science. Science demands that that all observers of a phenomenon can agree they see the same thing. Ruthless transparency is critical. Secrecy hinders the functioning of science, and peer review is a secret process. Science is not settled by the secret complaints of the anonymous.

Some people will say that a scientific result is true because it appears in a peer reviewed journal. That is the weakest defense possible. It means only that some editor and his reviewers found it to meet their minimum quality standards for publishing. It meets no standards if the editors and peer reviewers are corrupt.

When people see "peer review", they usually think of "scientific review", which is the detailed investigation of data and the replication of results by independent scientists. Scientific review gives some confidence that the claimed results are correct. Even then, conclusions about what the results "mean" can be wrong.

- -
ClimateGate: The Fix is In
11/24/09 - Real Clear Politics By Robert Tracinski
Via SmallDeadAnimals

[edited] Global warming "skeptics" had unearthed evidence that scientists at the Hadley Climatic Research Unit (CRU) at Britain's University of East Anglia had cherry-picked data to manufacture a "hockey stick" graph. This graph showed a dramatic, but illusory, runaway warming trend in the late 20th century.

Much broader evidence has emerged that will break that scandal wide open. Pundits have named it "Climategate." Thousands of e-mails and data from the CRU are now available on the Web.

The following stood out for me. There is extensive evidence of the hijacking of the "peer review" process to enforce global warming dogma. Peer review is the practice of subjecting scientific papers to review by other scientists with relevant expertise before they can be published in professional journals. The idea is to weed out research with obvious flaws or weak arguments, but there is a clear danger that such a process will simply reinforce groupthink.

Peer review has been corrupted, becoming a mechanism for an entrenched establishment to exclude legitimate challenges by simply refusing to give critics a hearing.

The noted climate researcher Michael Mann emailed about pressuring the journal Climate Research, which published a paper critical of global warming.

I think we have to stop considering "Climate Research" as a legitimate peer-reviewed journal. Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal.

This is the scandal of the century. It needs to be thoroughly investigated, and the culprits need to be brought to justice.

Read more at the link:

• The emails involve numerous leading British and American climate scientists outside of the CRU.
• Private admissions of doubt or scientific weakness in the global warming theory.
• A prominent global warming alarmist admits to using a statistical "trick" to "hide the decline" in temperatures.
• Cherry-picked data
• Evasion of legal requests for data, under the Freedom of Information Act.

The Intergovernmental Panel on Climate Change (IPCC) is a project of the United Nations. Its reports are taken as gospel by governments and scientists pushing for global control of industry, to avoid catastrophic global warming.

Many of the scientists contributing to the IPCC, especially at the Hadley CRU, were committed to avoiding scientific review. Despicably, they used their powers of peer review to exclude criticism of their papers, and they refused to release underlying data to any independent scientific review. Fortunately, a few scientific bloggers were able to make many of the faults public.

Amazingly, the IPCC didn't even restrict itself to peer reviewed results. The IPCC is a political institution, not a scientific one. Being half a scientist is like being half a truth.

1/3rd of IPCC claims were not peer-reviewed
04/20/10 - by Don Surber

Citizens Audit of the UN's Climate Report
04/07/10 - by Noconsensus.Org

- -
The IPCC is Political, Not Scientific
04/20/10 - PrisonPlanet by economist Richard Tol

Working Groups 2 and 3 of the AR4 (Assessment Report 4) violated all IPCC procedures. The conclusions are scientifically unfounded in part, and some are even copied from the environmental movement. Valid comments were ignored. AR4 contains crude errors as a result, only some known publicly.

- -
Galilean Peer Review
12/06/09 - Throckmorton's Other Signs

A doctor teaches his residents how to read published, peer-reviewed papers in medical journals. They must carefully examine the evidence and methodology. Most of the papers are not convincing after a hard look.

Think about that, the next time a newspaper breathlessly reports a finding in a newly published scientific paper.

- -
An array of errors
09/10/11 - The Economist

Summary:  Researchers Anil Potti and Joseph Nevins at Duke University published that they could predict the course of lung cancer using expression arrays, colorful activity patterns of thousands of genes in a tissue sample. The research was sloppy and wrong, despite initial peer review and repeated publication in respected medical journals. Those journals refused most critical comment.

AMG:  This is similar to the controversy surrounding the data, statistics, and computer code used to construct climate models. Prominent climate scientists have refused to release this supporting information for independent confirmation. The climate journals seem to be a much tighter and more defensive group than the medical journals.

When researchers refuse to supply their source data and methods, they are not scientists. A researcher earns our trust through open disclosure. He does not deserve any trust from calling himself a scientist or from working for a prestigious institution.

[edited]:  Investigations into alleged scientific misconduct have revealed numerous holes in the oversight of science and scientific publishing.

Bio-statisticians Keith Baggerly and Kevin Coombes work at the MD Anderson Cancer Centre in Houston. They found serious flaws in the work at Duke.

Dr. Baggerly noted that he did not have full access to the computer code and consistent raw data on which the work was based.

Journals that had readily published Dr. Potti’s papers were reluctant to publish Dr. Baggerly's criticism of Potti's work. Nature Medicine published one critical letter, and a rebuttal from the team at Duke, but rejected further comments as more problems arose. Other journals behaved similarly.

Eventually, Baggerly and Coombes resorted to publishing their criticisms in a statistical journal, unlikely to reach the same audience as a medical journal.

Dr. Califf is vice-chancellor in charge of clinical research at Duke University. He and other senior administrators acknowledged they gave too much weight to Dr. Nevins's judgment. That led them to withhold Dr. Baggerly’s criticisms from the external-review committee in 2009. The internal committees responsible for overseeing clinical trials lacked the expertise to review the complex statistical methods used in experiments on gene expression.

The process of peer review relies (as it always has) on the goodwill of workers in the field, who have jobs of their own and frequently cannot spend the time needed to check other people’s papers in a thorough manner. (amg: Despite the fact that they have agreed to peer review those papers.)

Dr. McShane estimates she spent about 350 hours reviewing the Duke work. Drs. Baggerly and Coombes estimate they spent nearly 2,000 hours. The methods sections of papers are supposed to provide enough information for others to replicate the work, but often do not.

Dodgy work will be revealed eventually, as it is found not to fit in with other, more reliable discoveries. But that all takes time and money.