In the following, we examine U.S. federal government audit standards that might pertain or be applied to assessment of the performance of the institutional research misconduct process and ORI’s (or an OIG’s) oversight of that process. Of primary note are the Government Accountability Office’s (GAO’s) “generally accepted government auditing standards” or “GAGAS” (GAO 2011). These standards, also referred to as the Yellow Book, underlie the more investigation-specific standards found in the government’s “Quality Standards for Federal Offices of Inspector General”, known as the Silver Book (CIGIE 2012).
The federal government’s audit standards provide some important direction on issues pertaining to assessing and managing potential conflicts of interest which might affect regulatory bodies, such as those discussed above for ORI. For example, GAGAS states:
“Maintaining objectivity includes a continuing assessment of relationships with audited entities and other stakeholders in the context of the auditors’ responsibility to the public. The concepts of objectivity and independence are closely related. Independence impairments impact objectivity.”
Of particular importance with respect to ORI and its dual roles of institutional support and oversight, it is not sufficient to simply declare there is independence because functions such as support and investigation are in separate “divisions”, as is currently the case for ORI:
3.10 “for the purposes of independence evaluation using the conceptual framework, an audit organization that includes multiple offices or units, or includes multiple entities related or affiliated through common control, is considered to be one audit organization.”
More specifically, GAGAS provides additional guidance as to actions that true auditors should take to address threats to their independence:
“3.14 Threats to independence may be created by a wide range of relationships and circumstances. Auditors should evaluate the following broad categories of threats to independence when threats are being identified and evaluated:
a. Self-interest threat - the threat that a financial or other interest will inappropriately influence an auditor’s judgment or behavior;
b. Self-review threat - the threat that an auditor or audit organization that has provided nonaudit services will not appropriately evaluate the results of previous judgments made or services performed as part of the nonaudit services when forming a judgment significant to an audit;
c. Bias threat - the threat that an auditor will, as a result of political, ideological, social, or other convictions, take a position that is not objective;
d. Familiarity threat - the threat that aspects of a relationship with management or personnel of an audited entity, such as a close or long relationship, or that of an immediate or close family member, will lead an auditor to take a position that is not objective;
e. Undue influence threat - the threat that external influences or pressures will impact an auditor’s ability to make independent and objective judgments;
f. Management participation threat - the threat that results from an auditor’s taking on the role of management or otherwise performing management functions on behalf of the entity undergoing an audit; and
g. Structural threat - the threat that an audit organization’s placement within a government entity, in combination with the structure of the government entity being audited, will impact the audit organization’s ability to perform work and report results objectively.”
Self-review threat could encompass the conflict of interest arising from the services and duties of ORI’s Division of Education and Integrity and its Division of Investigative Oversight (DIO). For example, if ORI’s education and support division believes that it has established that an institution is doing a good job of training and taking other steps against research misconduct, then there might be a bias against the investigative division making findings that suggest otherwise. We previously compared this situation to the one that led to passage of the Sarbanes–Oxley Act, which restricted audit firms from providing non-audit services to the same client.
Structural threat was suggested when former ORI Director Wright questioned whether the placement of ORI in the highly politicized environment of OASH at HHS was appropriate. (Kaiser 2014).
Also of note, self-interest threats are raised by the potential conflict that can arise when an institution and its deans or other officials have professional or financial interests in the grants being obtained by faculty who have been accused of research misconduct. The NSF case referenced above (Bauchwitz 2016) involved evidence of a similarly conflicted process. In that case, the NSF OIG criticized the investigation performed by faculty investigative committees at two universities concerning a faculty member who had been hired by one of the universities (the University of Colorado, Boulder) from the other (North Carolina State University). From an article on the NSF OIG report by a journalist:
“The [NSF] inspector general’s report also pointed to lapses in the internal investigations. The N.C. State investigative committee, for example, did not examine [graduate student] Gugliotti’s lab notebooks, which were the basis of the Science article.
The University of Colorado investigation had more problems, according to the report. For one, it inaccurately said that N.C. State had “exonerated” Feldheim and Eaton. N.C. State had found that Feldheim acted negligently and that all three published false statements.
The inspector general said Colorado failed to contact key witnesses and accepted the promises of Feldheim and Eaton to correct the record, which they have not done.
Joseph Rosse, head of research integrity and compliance at Colorado, declined to be interviewed.” (Neff 2016).
The unwillingness of a research integrity officer to respond to a well-known journalist who covers scientific misconduct matters illustrates the benefits of using professional auditors such as those in an OIG who can require responsiveness under law.
This NSF OIG case even more starkly shows just how inappropriate the results of such conflicts of interest can become. One of the UC Boulder investigative committee members apparently made derogatory comments to the press about a professor who had originally acted as a whistleblower in bringing the research misconduct concerns to light:
“The investigation was led by Colorado physics professor Alan Franklin, who told The Daily Camera of Boulder in 2014 that [NC State Professor Stefan] Franzen had ‘made a career’ of complaining about Feldheim and Eaton.
‘There’s been great damage done to people, mainly Eaton and Feldheim, for no good reason,’ Franklin told the newspaper.
Contacted this week, Franklin declined to discuss the final NSF reprimand, saying it was a confidential personnel matter.
Franzen welcomed the NSF ruling as vindication of his struggle to correct the record but said he remains bitter at how he says Feldheim, Eaton and their allies have sullied his reputation for years. Instead of addressing the underlying science, he said, his adversaries hired lawyers to gum up and prolong the investigations and resorted to personal insults to slander him.” (Neff 2016).
Even the NSF OIG is apparently willing to risk self-review threat by returning the handling of allegations of research misconduct back to an affected university:
“Whenever possible, OIG also relies on the relevant professional community to evaluate the seriousness of alleged misconduct based on its accepted standards and practices. To achieve this, we often refer investigations to the institution managing the award for evaluation.”
It is not clear that allegations referred back to an institution by NSF OIG would necessarily proceed to the level of faculty inquiry.
Institutional self-investigation also falls into what is more commonly referred to as “self-policing” or, more formally, the conflict of interest known as group or industrial self-regulation. With respect to actual law, most of those at the federal level deal with financial conflicts of interest of individual government employees. (Maskell 2014). There is limited law that addresses organizational conflicts of interest (COI), such as 48 CFR subpart 9.5, federal “Organizational and Consultant Conflicts of Interest” law, which is cited by the Centers for Medicare and Medicaid Services (CMS) and the Department of Defense. While this law might be used to address conflicts of interest from a research institution providing grant services and also investigating misconduct associated with those services, it does not deal with potential intra-agency or inter-agency COI that could be relevant to the structure of ORI. For this reason, we believe that the standards set by GAGAS against threats to independence should be codified into federal research misconduct oversight laws.
Ultimately, we recommend that university faculty should no more be relied upon to investigate their colleagues than they would be allowed to vote on their federal grant funding. At a minimum, HHS OIG could convene investigative review committees, just as NIH runs grant review committees. Alternatively, or in combination, independent, investigative and audit firms could be employed, as in the SOX model.
There are a number of federal audit guidelines for addressing the aforementioned threats to independence and conflict of interest. Some of those involve the important concept of performance audits:
“2.10 Performance audits are defined as audits that provide findings or conclusions based on an evaluation of sufficient, appropriate evidence against criteria.”
Performance audits are also more broadly of value in determining whether an oversight system is actually functioning adequately.
One example of a performance audit involving the handling of research misconduct that can be obtained by online search was produced by the Department of Energy’s OIG in 2014 (DOE 2014). That audit, however, stated that it “did not review the actual allegations”. As part of this study, DOE’s OIG was asked whether they would “have been able to review all allegations of research misconduct within its purview had it chosen to do so”, i.e. “would sufficient documentation per GAGAS 6.56 and 6.57 be retained by responsible entities in order for DOE OIG to assess all allegations, including in particular those dismissed prior to any inquiry”. However, DOE’s OIG declined to answer, writing that, “The posed questions exceed the scope of our audit”. They also did not specify, in response to a question, any law that would have required such documentation of allegations. (Written response to questions from F. Jones, DOE OIG, April 29, 2016).
An example of a performance audit that included delving into actual allegations was produced by the Colorado Office of the State Auditor (Colorado OSA 2016). In this case, the allegations were ethics complaints made against state judges, but the concept and process would apply equally to scientists. Interestingly, legislation to create a federal OIG for the U.S. judiciary has now been introduced into Congress (S. 1418 2015). Thus, the proposals made in this article are part of a more generalized approach to improve government function using IGs and audits.
Audits do not need to be conducted by governmental bodies, but can involve third party reviews in order to independently confirm performance. GAGAS discusses what it terms “safeguards, which are used to mitigate threats to auditor/investigator independence”; these include reviews by external entities:
“3.16 Safeguards are controls designed to eliminate or reduce to an acceptable level threats to independence.
3.17 Examples of safeguards include:
consulting an independent third party, such as a professional organization, a professional regulatory body, or another auditor;
involving another audit organization to perform or reperform part of the audit.”
The preceding “examples”, while potentially very sensible, are not obviously mandated. However, there is some required quality control and assurance in GAGAS 3.82. Of note:
“Each audit organization performing audits in accordance with GAGAS must:
b. have an external peer review performed by reviewers independent of the audit organization being reviewed at least once every 3 years.”
Characteristics of an external peer review are specified in GAGAS 3.105:
“An external audit organization should make its most recent peer review report publicly available. For example, an audit organization may satisfy this requirement by posting the peer review report on a publicly available web site or to a publicly available file designed for public transparency of peer review results. Alternatively, if neither of these options is available to the audit organization, then it should use the same transparency mechanism it uses to make other information public. The audit organization should provide the peer review report to others upon request.”
Such peer reviews of government IG audit are sometimes published as a short letter, and frequently are performed by another federal audit IG. What we propose here is a much more detailed, non-governmental third party review that completely conforms to GAGAS and is published, to the extent possible (see next), in its entirety.
A very common objection made against the publication of raw audit information is that it will undermine confidentiality. GAGAS also addresses this issue:
“5.40 Certain information may be classified or may be otherwise prohibited from general disclosure by federal, state, or local laws or regulations. In such circumstances, auditors may issue a separate classified or limited use report containing such information and distribute the report only to persons authorized by law or regulation to receive it.”
Therefore, at a minimum, independent auditors should see all confidential information, even if laws prevent the public from seeing the same. External peer review auditors may have to publish such details in a restricted manner. Thus, ORI, or any other oversight entity such as HHS OIG, should be provided with confidential information regarding allegations when auditing the performance of institutions and ORI in handling them, and the same information should be made available to independent third party reviewers who engage in performance audits of ORI and its handling of biomedical research misconduct allegations.
A related and very major issue is the retention of adequate data for performance audits:
“6.01 This chapter contains field work requirements and guidance for performance audits conducted in accordance with generally accepted government auditing standards (GAGAS). The purpose of field work requirements is to establish an overall approach for auditors to apply in obtaining reasonable assurance that the evidence is sufficient and appropriate to support the auditors’ findings and conclusions.”
As noted previously, we strongly question whether the use of aggregate data alone to assess allegations of biomedical research misconduct could provide sufficient evidence for any sort of audit. More definitively, GAGAS contains the following requirement:
“6.56 Auditors must obtain sufficient, appropriate evidence to provide a reasonable basis for their findings and conclusions.
6.57 The concept of sufficient, appropriate evidence is integral to an audit. Appropriateness is the measure of the quality of evidence that encompasses its relevance, validity, and reliability in providing support for findings and conclusions related to the audit objectives. In assessing the overall appropriateness of evidence, auditors should assess whether the evidence is relevant, valid, and reliable. Sufficiency is a measure of the quantity of evidence used to support the findings and conclusions related to the audit objectives. In assessing the sufficiency of evidence, auditors should determine whether enough evidence has been obtained to persuade a knowledgeable person that the findings are reasonable.”
If such a standard were specified in U.S. federal research misconduct laws, we believe that the resulting evidence could permit a valid assessment of current procedures to handle allegations of research misconduct.