Medical guidelines, since their conception as an aid for health care, have increased exponentially and encompass an ever wider spectrum of conditions [1]. A prime example is critical illness, with a growing number of guidelines being rapidly proposed and developed [2]. Despite their potential usefulness, however, the actual application and influence of guidelines on care has been inconsistent [3]. The basis for this inconsistency relates to both weaknesses in guideline developmental methodology and the sometimes poor quality of the recommendations themselves [4–6]. In this regard, Dr. Gorman and his co-investigators should be commended for their effort in assessing 24 critical care medicine guidelines [7]. They focused their analysis on guidelines that include recommendations for pharmacotherapy. This was an ambitious project that could provide insights into the applicability of guidelines for critically ill patients. However, without additional presentation of data from their analysis, it is not possible to fully assess the validity of the investigator’s conclusions about the guidelines and recommendations judged weak, but which are widely applied. This is also true for guidelines and recommendations judged strong, but which have engendered controversy.

The investigators employed the Appraisal of Guidelines, Research and Evaluation (AGREE) Collaboration tool to evaluate the methodologic quality of each of the 24 guidelines reviewed [5]. This tool includes six broad areas of assessment (domains) including: scope and purpose, stakeholder involvement, rigor of development, clarity, applicability and editorial independence. Each of these domains in turn contains from two to seven individual elements. Each element was evaluated and scored on a 4-point Likert scale by four investigators [7]. Domain scores were calculated by summing the individual items and by standardizing the total as a percentage (0–100%) of the maximum possible score for the domain. With this analysis, Gorman et al. present a standardized AGREE domain score showing how the 24 critical care guidelines performed as a group (Fig. 2) [7]. They also provide individual standardized domain scores (average without variability) for each guideline (Appendix 1, erratum [8]), and an overall assessment of each individual guideline (i.e., strongly recommended, recommended with alterations or not recommended) based on analysis with the AGREE instrument (Appendix 1) [7].

However, element scores, the basis for the overall assessment of each domain, are not provided. This more detailed data would allow the groups that formulated each guideline to either effectively revise them or to potentially rebut the present authors’ assessment. These data are also necessary information for the reader or user to accurately judge the potential strengths and weaknesses of a particular guideline. For example, the AGREE domain encompassing rigor of development contains elements to be evaluated that are very different; including methods used to initially search for recommendations versus the process employed in the final review of the guideline before publication. Understanding which of these elements most contributed to either a weak or strong domain score is essential for assessing the guideline process. It is also a critical component needed to evaluate why the investigators’ judgment of a particular guideline might conflict with others. For example, the Surviving Sepsis Campaign (SSC): International Guidelines for the Management of Severe Sepsis and Septic Shock published in 2008, which the investigators “strongly recommend,” while receiving endorsement from some professional societies, was denied endorsement after evaluation by the Infectious Disease of America (IDSA) and the American Thoracic Society (ATS) [9]. On the other hand, the investigators judged the Guidelines for the Prevention of Intravascular Catheter Related Infections as “recommended with alterations” [10]. However, this guideline was endorsed by all participating professional societies including the IDSA and ATS and is widely employed by clinicians. At present it is not clear what factors influenced the investigators assessment of these guidelines. Inclusion of all supporting data would greatly illuminate how they came to their conclusions.

Well-formulated guidelines and their recommendations may indeed aid health care workers in the effective management of critically ill patients. Employing tools to assess the rigor and applicability of both published and proposed guidelines is likely important. However, to be of greatest value both for those considering revising a guideline and for potential users, presentation of data from such assessments should be comprehensive and clearly presented. This may require methods including participation by those actually writing the guidelines themselves to fully capture all relevant data and to preserve balance.