A common approach for conducting peer review processes is by either (i) panel review; (ii) mail review; or (iii) both. Both review processes are based on insights and recommendation of well-informed experts on various quality dimensions of research, as guided by a scorecard (Ruegg & Feller, 2003). The following section provides a detailed discussion and comparison of the panel and mail review processes.
3.2.1 Panel Review
In a panel meeting, reviewers are co-opted by the funding agency and a formal meeting is convened. There is usually an appointed chairperson who ensures that all applications, as logged and pre-screened by the funding agency, are reviewed with clear recommendations provided by the panel. The role of the chairperson is to facilitate the discussion on an application and guide the panel towards a consensus decision to either “fund” or “not fund” a specific application. The chairperson will also ensure that an appropriate length of time is allowed for the evaluation of each proposal. In addition to having an appointed chairperson, there is also an appointed assessor who ensures that personal biases from any appointed reviewer is minimised. The assessor’s role is also to ensure that the processes adopted during the meeting are fair and transparent and that the same criteria are applied consistently by all the panel members for the evaluation and scoring of all applications. In essence, the role of the assessor is to ensure procedural consistencies are applied when evaluating proposals. At the end of the panel meeting, both the chairperson and the assessor will submit a jointly written report which will be used by the funding agency to either improve or retain specific review processes. Supporting the chairperson and assessor in a panel meeting is a rapporteur whose role is to capture the proceedings of the meeting on a verbatim basis. This is an important process as it ensures transparency as well as provides a reference point for contestations that may arise from time to time, especially if researchers were unsuccessful in their application to obtain funding and require detailed feedback.
The role of reviewers is to make recommendations to the funding agency on whether each application, when considered in their entirety, should be funded or not. The panel is required to use the prescribed scorecard from the funding agency as a guide for evaluating the applications. The panel reviewers are required to submit a completed reviewer evaluation form at the end of the meeting that can also be used by funding agency staff to provide feed back to the applicant. This report must outline the successes, challenges and areas for improvement in the submitted application. During the panel review, usually two reviewers present a research proposal to the rest of the participants of the peer review group (Braun, 1998). This opens the floor to dialogue and opposing views by the other panel participants. There is a tendency in this review method for those reviewers evaluating a proposal to have the prerogative in the decision on whether or not a project is successful (Lee & Harley, 1998). Although a peer review can gain consensus on proposals that are either outstanding or poor, it is difficult to reach a consensus on proposals that score in the middle range which is a major limitation associated with the peer review system (Kostoff, 1994). At this stage, the role of both the assessor and chairperson becomes of paramount importance, especially in terms of ensuring that the key purpose of a peer review is to support outstanding proposals and reject those proposals that are deemed poor.
The drawbacks associated with the panel review method are cost implications and an inherently subjective decision making process that depends on the interests, experience and knowledge of the evaluators (Lee & Harley, 1998). Furthermore, the quality of the review can never go beyond the competence of the reviewers (Kostoff, 1994). It is, therefore, essential that the reviewer profile of the panel includes a combination from different countries and research backgrounds that span the spectrum of disciplines shortlisted in the pre-screening process, e.g. physical sciences and biological sciences. The use of international reviewers that host and manage mega-RIs should be identified as potential reviewers. These reviewers not only provide an independent and objective expert perspective but also guide the funding agency on best practices, risks, opportunities and challenges relating to the investment in RIs. A drawback to the use of international reviewers is their lack of understanding of local or national imperatives and context.
3.2.2 Mail Review
Funding agencies also employ a mail or postal review system where referees or reviewers decide on the credibility of the proposal and the research applicant in accordance with the guidelines and a scorecard prescribed by the funding agency. In the mail review system, the referee or reviewer makes an independent decision without being exposed to the opinion(s) of other reviewers (Lee & Harley, 1998). Usually two or three mail reviewers are requested on the same project proposal in order to balance the views of proposals. One of two processes can unfold post the submission of mail review reports.
Firstly, the reports can be anonymised and subsequently fed as source documents into the panel review meeting. These mail review reports provide an alternate perspective on the proposals to be evaluated at a panel meeting. If this process is undertaken, the panel reviewers have the final decision relating to whether or not a project is successful. Secondly, the reports are used by the funding agency staff to make the final decision on the outcomes of the application (Braun, 1998).
The general experience in the South African context is that the poor quality of the postal review reports do not provide adequate information for a decision to be made by either the funding agency or panel reviewers on whether or not an application should be funded. Hence the consensus is that the panel review be exclusively employed which aids in reducing (i) the complexity related to awarding RI grants; and (ii) the conflict(s) of interest that may emerge due to the small pool of reviewers available in the country.