Abstract
An error in the Fourth Assessment Report (2007) of the Intergovernmental Panel on Climate Change (IPCC), which wrongly predicted the disappearance of Himalayan glaciers by 2035, fueled doubts about the authority, honesty and rigor of the IPCC as a leading institution in climate science and, correspondingly, raised questions about whether global warming is anything more than a hoax put forward by environmentalists. The late and confusing reaction of the IPCC to these allegations only worsened the matter. By comparing assessment reports issued by the IPCC, this paper asks the question: ‘Why, despite extensive peer review, did the Himalayan glacier melting rate error get published?’ I chronicle exactly what happened (Sect. 3), consider why it happened as it did (Sect. 4), and show how these answers require us to rethink the concept of ‘peer review’ in scientific practice. I also identify several future directions for peer review (Sect. 5) if it wants to stay ahead of the game, bearing in mind the forthcoming IPCC Assessment Reports.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
For a detailed description of the tasks and responsibilities for Lead Authors, Coordinating Lead Authors, Contributing Authors, Expert Reviewers, Review Editors and Government Focal Points see (IPCC 2008b).
Note that at the time of publication, only the Working Group I report of the Fifth Assessment Report was released.
The most well-known dispute here was the unauthorized release of e-mail exchanges, which were seen to discredit the University of East Anglia and its scientists. This leaked e-mail traffic gave rise to serious skeptical doubts concerning the honesty of climate scientists. The question of whether (some of) the doubts raised were grounded is beyond our scope here. In any case, the credibility of the overall conclusion of the Fourth Assessment Report (2007), namely that global warming is happening and is due mainly to anthropogenic factors, was not affected by this incident in any respect.
Since the Indian government pointed out the melting error in early November 2010, the IPCC took more than two full months to release a statement, made public on 20 January 2010, containing the IPCC board’s apologies. The apology read as follows: “It has, however, recently come to our attention that a paragraph in the 938-page Working Group II contribution to the underlying assessment refers to poorly substantiated estimates of rate of recession and date for the disappearance of Himalayan glaciers. In drafting the paragraph in question, the clear and well-established standards of evidence, required by the IPCC procedures were not applied properly. The Chair, Vice-Chairs, and Co-Chairs of the IPCC regret the poor application of well-established IPCC procedures in this instance. This episode demonstrates that the quality of the assessment depends on absolute adherence to the IPCC standards, including thorough review of the quality and validity of each source before incorporating results from the source into an IPCC Report. We reaffirm our strong commitment to ensuring the level of performance” (IPCC 2010a). Follow-up letters explaining the principles, procedures, role and key elements of the IPCC were released on 2 and 4 February 2010 (IPCC 2010b, c). The Chairman of the IPCC, Rajendra Pachauri, made no personal comment at the time, but in response to criticism concerning his position as Chairman he responded as follows to the Guardian "They can't attack the science so they attack the chairman. But they won't sink me. I am the unsinkable Molly Brown. In fact, I will float much higher." (Carrington 2010) Even more striking is the first (and emotional) response made by the Chairman in response to the geologist Vijay Kumar Raina, who issued a report in name of the Indian government pointing out the error. Pachauri dismissed the report as not peer-reviewed and said: “With the greatest of respect this guy retired years ago and I find it totally baffling that he comes out and throws out everything that has been established years ago.” (Carrington 2010).
I recognize this to be an ill-supported slippery slope and should hereby acknowledge that the overall conclusion, that “Widespread mass losses from glaciers and reductions in snow cover over recent decades are projected to accelerate throughout the twenty-first century, reducing water availability and hydropower potential” (IPCC 2007: 49), still holds. Regardless, however, of whether one finds this claim and critique convincing, one has to recognize the impact that a situation like this could have upon overall public opinion and media reporting, bearing in mind the halting effect of propagating doubt on public policy making and how even a very small hunch can successfully merchandise doubt (Oreskes and Conway 2010).
It is important to keep in mind, as suggested by a reviewer of this paper, that since the claim made is only hypothetical, no deliberate falsification of data is involved here.
The standpoint of the IAC-report on the matter reads as follows: “In this example, IPCC’s report review process failed in two ways: (1) Failure of the authors to carefully consider thoughtful review comments (E10-466 and E10-468), which would have improved the quality of the report, and (2) Failure of the Review Editors to ensure that reviewer comments were adequately addressed and that controversies are reflected adequately in the text of the report (E10-468)” (IAC 2010, 24).
The error mentioned here is ‘that the likelihood of Himalayan glaciers disappearing by the year 2035 and perhaps sooner is very high if the Earth keeps warming at the current rate’.
It is not my intention here to scapegoat the Review Editors and/or Writing Team members. My aim is simply to address what happened.
As the IAC-study suggests, most of the review comments were related to problems with grey literature.
In our example, as the Table shows, the reviewers’ names were made public and yet the error was published anyway. I return to this point later in the paper.
A possible suggestion is to install Review Editors with more power to ensure that grey literature is selected appropriately and used consistently (IAC 2010, 20).
Although the IAC does not suggest any concrete way for tackling this problem, it proposes that the IPCC adopt a more targeted and effective process for responding to reviewer comments: “In such a process, Review Editors would prepare a written summary of the most significant issues raised by reviewers shortly after review comments have been received. Authors would be required to provide detailed written responses to the most significant review issues identified by the Review Editors, abbreviated responses to all non-editorial comments, and no written responses to editorial comments” (IAC 2010, 21).
Such as seen on the academic blog http://www.newappsblog.com/.
We do not address this question here, as our example merely engages with reviewers checking the reliability of information, as can be seen in the Table and in the stated values of the IPCC (Sect. 1). For further information on how this distinction relates to peer review in academic journals, where both ways of assessing a paper come into play, see Frans and Kosolosky (2014).
One could raise the objection that ‘If comments are overlooked, having a name attached to them might make no difference.’ Whereas it is often true that if no one responds to remarks, nothing will happen (as was the case here), attaching a name to comments does make a difference in another respect. As the example shows, it becomes easier for insiders and outsiders to see what exactly happened, to allocate credit to those who managed to see the flaws, and assign responsibility accordingly. Not only do the reviewers benefit from this (in terms of the due credit they receive for checking and questioning the reliability/correctness of information), but also the organization as a whole benefits (as its transparency is increased and responsible factors can be located, allowing us to deal with the matter more quickly).
One can easily imagine cases in which it would be best to conceal the name of the reviewer, such as when a paper is rejected on grounds that it lacks originality. However, this does not exclude the possibility of installing independent experts to assess whether the decision to reject was grounded or not. We will go no further into this matter here, as the question of how to judge the originality and publication-worthiness of a paper is not at issue in our example. As suggested, however, I do think that these two ways of understanding peer review can and should be separated.
Recall, here, the guidelines of the IPCC, noted in Sect. 1.
The statement of Georg Kaser, an expert in tropical glaciology and lead author for the Fourth Assessment Report of the IPCC, in response to the 2035 prediction illustrates my claim nicely: “All the responsible people are aware of this weakness in the fourth assessment. All are aware of the mistakes made. If it had not been the focus of so much public opinion, we would have said ‘we will do better next time’” (Carrington 2010). At the boundary between science and policy, there seem to be requirements and relations at stake other than those to which scientists are accustomed in their ‘laboratory’ work.
In this paper, I have argued for this only in the first sense of reviewing, i.e. of checking the reliability of information.
References
Archer, D. R., & Fowler, H. J. (2004). Spatial and temporal variations in precipitation in the Upper Indus Basin, global teleconnections and hydrological implications. Hydrology and Earth System Sciences, 8(1), 47–61.
Barnett, T. P., & Adam, J. C., & Lettenmaier, D. P. (2005). Potential impacts of a warming climate on water availability in snow-dominated regions. Nature, 438, 303–309. doi:10.1038/nature04141.
Bjurström, A., & Polk, M. (2011). Physical and economic bias in climate change research: A scientometric study of IPCC Third Assessment Report. Climatic Change, 108, 1–22.
Carrington, D. (2010). IPCC officials admit mistake over melting Himalayan glaciers, the guardian. http://www.guardian.co.uk/environment/2010/jan/20/ipcc-himalayan-glaciers-mistake.
Dalton, R. (2001). Peers under pressure. Nature, 413, 102–104.
Fitzpatrick, K. (2010). Peer-to-peer review and the future of scholarly authority. Social Epistemology, 24(3), 161–179.
Fowler, H. J., & Archer, D. R. (2006). Conflicting signals of climate change in the Upper Indus Basin. Journal of Climate, 19, 4276.
Frans, J., & Kosolosky, L. (2014). Mathematical proofs in practice: Revisiting the reliability of published mathematical proofs, Theoria, 81, 345–360.
Fuller, S. (2002). Knowledge management foundations. Woburn, MA: Butterworth-Heinemann.
Hewitt, K. (2005). The Karakoram anomaly? Glacier expansion and the ‘elevation effect’, Karakoram Himalaya. Mountain Research and Development, 25, 332–340.
IAC (2010). Climate change assessments, review of the processes and procedures of the IPCC. Retrieved October 24, 2011 from http://reviewipcc.interacademycouncil.net/report.html.
IPCC, official website. http://www.ipcc.ch/index.htm.
IPCC (1995). IPCC Second assessment report. http://www.ipcc.ch/pdf/climate-changes-1995/ipcc-2nd-assessment/2nd-assessment-en.pdf.
IPCC (2007). Climate change 2007: Synthesis report. Contribution of working groups I, II and III to the fourth assessment report of the intergovernmental panel on climate change In: Core Writing Team, R. K. Pachauri & A. Reisinger (Eds.) (p. 104) Geneva, Switzerland: IPCC.
IPCC (2008a). Procedures for the preparation, review, acceptance, adoption, approval, and publication of IPCC reports. Retrieved October 24, 2011 from IPCC: Official website. http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf.
IPCC (2008b). Annex 1: Tasks and responsibilities for lead authors, coordinating lead authors, contributing authors, expert reviewers and review editors of IPCC reports and government focal points. In IPCC, Procedures for the preparation, review, acceptance, adoption, approval, and publication of IPCC reports. Retrieved October 24, 2011 from IPCC: Official website. http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf.
IPCC (2008c). Annex 2: Procedure for using non-published/non-peer-reviewed sources in IPCC reports. In IPCC, Procedures for the preparation, review, acceptance, adoption, approval, and publication of IPCC reports. Retrieved October 24, 2011, from IPCC: Official website. http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf.
IPCC (2010a). IPCC statement on the melting of Himalayan glaciers. Letter put online on the official IPCC website on January 20th, 2010.
IPCC (2010b). Statement on IPCC principles and procedures. Letter put online on the official IPCC website on February 2nd, 2010.
IPCC (2010c). The role of the IPCC and key elements of the IPCC assessment process. Letter put online on the official IPCC website on February 4th, 2010.
Kihara, H. (2003). The extension of peer review, How should it or should not be done? Social Epistemology, 17(1), 65–77.
Oreskes, N. (2004). Science and public policy: What’s proof got to do with it? Environmental Science and Policy, 7, 369–383.
Oreskes, N., & Conway, E. M. (2010). Merchants of doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. New York: Bloomsbury Press.
Pearce, F. (2010). Debate heats up over IPCC melting glaciers claim, The Guardian. http://www.newscientist.com/article/dn18363-debate-heats-up-over-ipcc-melting-glaciers-claim.html?DCMP=OTC-rss&nsref=online-news.
Roy, R., & Ashburn, J. R. (2001). The perils of peer review. Nature, 414, 393–394.
Shirky, C. (2008). Here comes everybody: The power of organizing without organizations. New York: Penguin Press.
Turner, L. (2003). Promoting FAITH in peer review: Five core attributes of effective peer review. Journal of Academic Ethics, 1, 181–188.
WWF Nepal Program (March, 2005) An overview of glaciers, glacier retreat, and subsequent impacts in Nepal, India and China. https://www.wwf.or.jp/activities/lib/pdf_climate/environment/Overview_of_Glaciers.pdf.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kosolosky, L. “Peer Review is Melting Our Glaciers”: What Led the Intergovernmental Panel on Climate Change (IPCC) to Go Astray?. J Gen Philos Sci 46, 351–366 (2015). https://doi.org/10.1007/s10838-015-9303-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10838-015-9303-y