Introduction The Nominal Group Technique (NGT) and Delphi Technique are consensus methods used in research that is directed at problem-solving, idea-generation, or determining priorities. While consensus methods are commonly used in health services literature, few studies in pharmacy practice use these methods. This paper provides an overview of the NGT and Delphi technique, including the steps involved and the types of research questions best suited to each method, with examples from the pharmacy literature. Methodology The NGT entails face-to-face discussion in small groups, and provides a prompt result for researchers. The classic NGT involves four key stages: silent generation, round robin, clarification and voting (ranking). Variations have occurred in relation to generating ideas, and how ‘consensus’ is obtained from participants. The Delphi technique uses a multistage self-completed questionnaire with individual feedback, to determine consensus from a larger group of ‘experts.’ Questionnaires have been mailed, or more recently, e-mailed to participants. When to use The NGT has been used to explore consumer and stakeholder views, while the Delphi technique is commonly used to develop guidelines with health professionals. Method choice is influenced by various factors, including the research question, the perception of consensus required, and associated practicalities such as time and geography. Limitations The NGT requires participants to personally attend a meeting. This may prove difficult to organise and geography may limit attendance. The Delphi technique can take weeks or months to conclude, especially if multiple rounds are required, and may be complex for lay people to complete.
The Nominal Group Technique (NGT) and the Delphi Technique are commonly referred to as consensus methods . They aim to achieve a general agreement or convergence of opinion around a particular topic. Consensus methods are used in research that is directed at problem-solving, idea-generation, or determining priorities . How consensus is defined and operationalised will vary from study to study, depending on the research objectives .
Consensus techniques such as the NGT and Delphi Technique are superficially similar to focus groups, a commonly used method in pharmacy practice research. All methods involve interaction within a group of participants, yet they can provide different outcomes. Focus groups are useful for investigating an issue in-depth, including the identification of problems, questions or significant issues. Consensus methods, however, raise potential solutions or answers to a question, which can then be prioritised or agreed upon. A key strength of consensus methods is the balanced participation from group members, unlike a focus group, whereby the facilitator must control for, and minimise the risk of, a dominant participant influencing the discussion. The structured format of consensus methods avoids this issue.
The aim of this paper is to provide an overview of the NGT and Delphi technique, including the steps involved and the types of research questions best suited to each method, with examples from the pharmacy literature. Therefore, it provides a useful starting point for pharmacy practice researchers new to consensus methods. Initially it describes how to conduct the NGT and Delphi Technique and provides examples of their use within the pharmacy context. Then, it considers the choice of experts for the panels and which types of research questions are best suited to which method.
Nominal group technique
The NGT is a highly structured face-to-face group interaction, which empowers participants by providing an opportunity to have their voices heard and opinions considered by other members . It was designed by Delbecq and Van de Ven and comprises four key stages: silent generation, round robin, clarification and voting (ranking or rating) . These stages are briefly explained below.
How to run the nominal group technique
While groups of between two and fourteen participants have been used in nominal group research (Table 1), a maximum of seven has been recommended . A nominal group generally involves one to two questions which are sent to participants in advance. At the beginning of the meeting, participants are given up to twenty minutes to silently reflect or record their individual ideas in response to a question, i.e. silent generation . The facilitator then asks one participant at a time to state a single idea to the group in a ‘round robin’ fashion. Participants are able to think of new ideas during this process, but must wait their turn before they can share with the group. This stage takes as much time as needed until no new ideas are forthcoming. It is recommended that there be no discussion at this stage and ideas are merely recorded verbatim on, for example, a flipchart or white board .
The third stage is clarification of the ideas, which also provides the opportunity for a grouping step, where similar ideas are grouped together with agreement from all participants. Participants may also exclude, include or alter ideas, as well as generate grouping themes . All ideas should be discussed to ensure participant understanding , thus enabling them to make an informed decision when they come to voting on ideas. Facilitators should emphasise that participants do not have to agree with all ideas listed as, at the end of the clarification stage participants are able to ignore ideas by voting on personal preferences. The round robin  and clarification phase  can take up to 30 min each. Facilitators should not direct participants during the clarification process, which may make this stage particularly difficult.
Participants are then provided with a ranking sheet, where they are asked to select their top preferences from the generated ideas. The number of items chosen by participants depends on the topic, but the ranking of five ideas is common in the literature [2, 5, 10]. The facilitator should specify that a number should be allocated to each selected item, with larger numbers reflecting greater importance [2, 5]. For example, for five ideas, the most important idea is scored five points. Although there is no anonymity for participants during nominal group discussions, individual scoring on a ranking sheet is confidential. Finally, the scores for each idea are summed and presented to the group for discussion. The timing for this stage is likely to depend on a number of factors, including the complexity of the topic and how many items need to be prioritised (the more items to rank, the harder the process and more time consuming it can become). Dening et al.  noted that voting could take up to 10 min to complete.
Ultimately, the time to complete one nominal group is variable, and depends on group size, how many questions are asked, and the type of participants involved. For example, Bradley et al.  documented a 2-h time limit to conduct a NGT for one question, whereas Hutchings and colleagues allocated half a day to conduct a NGT for two questions, followed by another half-day for a forum event  (see “Variations on the nominal group technique”).
Variations on the nominal group technique
The NGT is a highly adaptable method, and can be used in addition to  or to inform, other methods, e.g. a discrete choice experiment . NGT variations may be influenced by the available research and participant time, or the level of clarification, consensus or generalisability required for the topic. Ultimately, researchers need to ensure that the NGT is working for each participant group; it may be that stages need to be adapted. For example, for indigenous or culturally and linguistically diverse populations, it may be the cultural norm to discuss ideas as a group. Thus, a more appropriate variation to the process for generating ideas has been to combine the round robin and clarification stages . Other variations could be in direct response to participant ability. If it is too difficult for participants to group similar ideas at the clarification stage then this grouping step could be avoided altogether . While this may make it harder for participants to vote, i.e. there is a longer list of ideas to consider, it may cause less frustration for participants.
Generally, variations are seen in relation to generating ideas, and how ‘consensus’ is obtained from participants, i.e. the ranking process (Fig. 1):
Generating ideas instead of silent generation followed by a round robin, ideas are obtained from a literature review, or exploratory surveys are used which could be viewed as a way to achieve greater consultation [14, 15];
Re-ranking allowing participants to revise their original ranking, i.e. re-ranking, either in the original NGT meeting , via a secondary survey , or obtaining validation by sending a survey of nominal group results to other participants . Alternatively, the re-ranking process could continue until no further changes are seen with the most important ideas .
Where separate nominal groups are held for similar participants, e.g. consumer groups, health professional groups or stakeholder groups, a mixed-forum event can provide the opportunity for consensus to be achieved by forming new groups with different participant types . In a study that exemplified the use of a mixed forum event, Hutchings and colleagues asked previous participants to individually review the overall NGT results (overarching themes), and to rank the themes (pre-forum responses). At the forum, participants were asked to discuss the pre-forum responses in their newly allocated groups, which consisted of participants from differing backgrounds. Individuals were then asked to re-rank themes for a third time.
Other researchers have provided valuable information on important nominal group design considerations , its practical application  and method of analysis . Black et al.  reviewed the literature to identify the evidence for certain ‘best practice recommendations’ for consensus methods. While that review provides some important considerations for researchers wishing to use these methods, the articles included are, at a minimum, over 15 years old. Using specific examples, Tully and Cantrill  discuss the steps involved in a nominal group, and guidance for researchers with respect to group composition. While a discussion of qualitative and quantitative analysis is also included, McMillan et al.  take this one step further in their paper by detailing the entire analysis process for researchers who undertake more than ten nominal groups.
Applications to pharmacy research
The NGT has been applied in numerous healthcare settings, to develop guidelines  or explore opinions of different health professionals , lay people and carers [10, 19, 20], or to compare views of both parties [9, 21]. It is gradually building traction within the pharmacy setting, as seen in Table 1. Researchers have generated evidence based guidelines or criteria for pharmacy practice situations [7, 22, 23], informed practice change [11, 24] and the profession [12, 16] about particular topics, and identified attributes to be included when interviewing pharmacy students .
The Delphi technique
Like the NGT, the Delphi Technique is a highly structured group interaction. However, the Delphi Technique uses interactions between group (called panel) members via questionnaires rather than face-to-face communication. This means that it preserves participant anonymity, if that is relevant. The Delphi Technique was developed by the Rand Corporation in 1953  and uses a multistage self-completed questionnaire with individual feedback.
How to run the Delphi technique
There is no standard method to calculate a panel size for the Delphi Technique; however, the aim of the study and available resources are important . A sample of about fifteen has been suggested  but larger panels have also been used (Table 2). Inviting more participants increases the variety of expertise, but eventually leads to diminishing returns .
The first-round questionnaire will present a series of statements that the respondent is asked to rate on a clearly defined Likert scale. The content of the statements may come from a variety of sources, singly or in combination, including the literature [28–30], clinical practice [31, 32], or from previous research findings, including NGT studies [30, 33]. Respondents are asked both to rate the item and to write free-text comments that, for example, explain their rating or express disagreement with the statement’s relevance. Reminders are sent to non-responders in the usual way.
The responses to the first-round questionnaires are collated and used to create the second-round questionnaire. The latter presents the same statements as before, together with both the individual respondent’s rating and the median rating from the entire panel. A selection of the free-text responses is given, to represent the breadth of opinion. Respondents to the previous round thus get a personalised, unique questionnaire. Figure 2 provides an example of a statement from a second-round questionnaire seeking consensus on indicators for assessing medicines reconciliation processes . After considering the group median and free-text comments, respondents re-rate the statements, by either giving the same rating as before or an amended rating. Respondents may give further comments about the statements if they wish.
The number of survey rounds is usually decided in advance and is dependent upon the level of dissension expected. In most studies, two rounds are used but occasionally, only a single round has been run . More than two rounds increases panel attrition, so this is rarely done. The minimum time for a two-round Delphi can be as long as 30 days, although it may well take longer if multiple reminders are needed. The time required for the collation of responses and the creation of personalised second-round questionnaires should not be underestimated.
Often a 9-point Likert scale is used for the rating [29–31, 34], although 3-point , 5-point [28, 37] and 7-point  scales have also been used. The decision as to when consensus will have been reached must be made at the beginning of the study. For example, if the aim is to develop assessment criteria using the RAND 9-point scale , then consensus is reached that a statement is appropriate if the median score is greater or equal to 7, and it is inappropriate if the median score is less than or equal to 3. Disagreement is defined as where at least one third of respondents rate the statement at the opposite end of the scale to their peers. Such a finding would mean that consensus had not been reached.
Variations on the Delphi technique
A common variant is for the initial study questionnaire to collect ideas in response to open questions . Only subsequent questionnaires then ask respondents to conduct the rating process described above [33, 37, 39].
Examples of other modifications include researchers including only items that had failed to reach consensus in the second questionnaire (rather than all items, regardless of the ratings they received initially [29, 36]) and asking respondents to choose between alternatives (rather than rate items) for each question . Traditionally, the questionnaire was sent by post, but more recently, e-mail has been used for the so-called e-Delphi Technique . Emailed questionnaires per se are now so commonplace, that this is probably the new norm.
The RAND appropriateness method has been described as a variant of both the Delphi technique  and the NGT , as it has features of both. It involves participants reading a detailed literature review, followed by a traditional Delphi questionnaire. However, participants discuss the first-round results at a face-to-face meeting, followed by a second-round Delphi questionnaire and re-rating of the items.
Applications to pharmacy research
An early use of the Delphi Technique in pharmacy practice research was in forecasting the future of hospital pharmacy in Australia . It has been used to gain consensus on indicators for assessing prescribing appropriateness  or quality , criteria for safety features , clinically significant interactions  or aspects of student education  including communication skills  and professional engagement , or definitions, such as prescribing error . This range of topics reflects the common use of the technique for the generation of clinical guidelines within the wider healthcare arena .
Choice of experts
Experts, in the context of consensus methods, are those people who have knowledge about the topic of concern. Understandably, this is dependent upon the research aims and objectives, but such experts may not always be healthcare professionals. Given that greater importance has been placed on involving health consumers in research, consensus methods can be used to identify what is currently important to, or valued by, these experts. McMillan and colleagues, for example, explored the views of both the public and pharmacy staff on ideal community pharmacy services . Therefore, their experts included people living with chronic conditions, their carers, and pharmacy staff who provided the relevant services. Campbell and colleagues, on the other hand, identified prescribing indicators that used data from dispensed prescriptions . Therefore, their experts were the medical and pharmaceutical advisors who would be using the resultant indicators.
The NGT appears to be used more commonly with lay people than the Delphi Technique, although the reason why is not clear. Lay people may feel more comfortable participating in a face-to-face meeting, than in a relatively complex survey. For example, the NGT can be adapted to accommodate people with poor literacy . The Delphi Technique has been used with patients in a small number of studies to prioritise outcome measures for clinical trials  and has begun to be used with members of the public (in this case, parents of children with attention deficit/hyperactivity disorder) in pharmacy practice research .
Power differentials between people in the NGT may mean that people with less power may feel unable to contribute their own views or contradict the views of someone more powerful. Therefore, it is usual for the experts in each meeting to be relatively homogeneous in status (see Table 1), such as running separate meetings for consumers and pharmacists . This power differential may be less relevant for the Delphi Technique, as the experts are anonymous. Nonetheless, in those few Delphi studies that included both lay people and healthcare professionals, only patient data from the first-round questionnaire was sent to patients in the second-round questionnaire .
Choice of consensus method
The decision whether to use the NGT or the Delphi Technique is influenced by various factors, including the research question, the perception of consensus required, and the associated practicalities and limitations such as time and geography.
If researchers are seeking to explore ideas in relation to a problem or question, this best aligns with the NGT, as idea generation is an integral part of this method. If researchers want to develop guidelines, a Delphi Technique involving experts who are likely to use the guidelines in question would be more suitable. The development of guidelines requires a more rigorous process, with consensus needed from a larger number of experts, which is easier with the Delphi Technique. This larger group may be needed to give authority to the final decision .
While some researchers have specified a numerical level of consensus when using the NGT, this is not well documented and would likely require further re-ranking beyond the initial steps. Alternatively, most researchers using the Delphi method will explicitly refer to a consensus value, i.e. a numerical level of agreement, determined by researchers in advance. Thus, it could be viewed that these two techniques sit along a spectrum of consensus, with a clearer description of the level of agreement thought to be given by the Delphi Technique .
As the NGT involves participants for only a few hours, results can be obtained quickly, suiting researchers who require a prompt result. It is particularly suited if participants are likely to only want to attend a single session compared to answering multiple questionnaires several weeks apart. The NGT requires face-to-face meetings, but this may be more culturally appropriate even if participants are at a distance. However, it may be more difficult to organise a nominal group meeting for a time that suits everyone. In contrast, the Delphi Technique is more flexible. The Delphi Technique, especially if conducted by email, is accessible to participants regardless of location, thereby avoiding travel expenses. Yet, this method can take weeks or months to conclude, especially if multiple rounds are undertaken.
The NGT and Delphi Technique are both consensus methods that involve a group of ‘experts’ to generate ideas and determine priorities. The NGT has been used to explore consumer and stakeholder views, while the Delphi technique is commonly used to develop guidelines with health professionals. The NGT requires face-to-face discussion in small groups, and provides a prompt result for researchers. Alternatively, the Delphi technique uses questionnaires to preserve participant anonymity, can involve more participants but takes place over a longer time period.
Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376–80.
Delbecq AL, van de Ven AH, Gustafson DH. group techniques for program planning, a guide to nominal group and Delphi processes. Glenview, IL: Scott, Foresman and Company; 1975.
Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, et al. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999;4(4):236–48.
Tully MP, Cantrill JA. The use of the Nominal group technique in pharmacy practice research: processes and practicalities. J Soc Admin Pharm. 1997;14:93–104.
McMillan SS, Kelly F, Sav A, Kendall E, King MA, Whitty JA, et al. Using the nominal group technique: how to analyse across multiple groups. Health Serv Outcomes Res Method. 2014;14:92–108.
Claxton JD, Ritchie JRB, Zaichkowsky J. The nominal group technique: its potential for consumer research. J Consum Res. 1980;7:308–13.
Bissell P, Ward PR, Noyce PR. Appropriateness measurement: application to advice-giving in community pharmacies. Soc Sci Med. 2000;51(3):343–59.
Potter M, Gordon S, Hamer P. The nominal group technique: a useful consensus methodology in physiotherapy research. N Z J Physiother. 2004;32:126–30.
Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: A research tool for general practice? Fam Pract. 1993;10:76–81.
Dening KH, Jones L, Sampson EL. Preferences for end-of-life care: a nominal group study of people with dementia and their family carers. Palliat Med. 2012;27(5):409–17.
Bradley F, Schafheutle EI, Willis SC, Noyce PR. Changes to supervision in community pharmacy: pharmacist and pharmacy support staff views. Health Soc Care Community. 2013;21(6):644–54.
Hutchings HA, Rapport FL, Wright S, Doel MA, Wainwright P. Obtaining consensus regarding patient-centred professionalism in community pharmacy: nominal group work activity with professionals, stakeholders and members of the public. Int J Pharm Pract. 2010;18(3):149–58.
Hiligsmann M, van Durme C, Geusens P, Dellaert BG, Dirksen CD, van der Weijden T, et al. Nominal group technique to select attributes for discrete choice experiments: an example for drug treatment choice in osteoporosis. Patient Prefer Adherence. 2013;7:133–9.
Allen J, Dyas J, Jones M. Building consensus in health care: a guide to using the nominal group technique. Br J Community Nurs. 2004;9(3):110–4.
Vella K, Goldfrad C, Rowan K, Bion J, Black N. Use of consensus development to establish national research priorities in critical care. BMJ. 2000;320(7240):976–80.
McMillan SS, Kelly F, Sav A, Kendall E, King MA, Whitty JA, et al. Consumers and carers versus pharmacy staff: Do their priorities for Australian Pharmacy Services align? Patient. 2015;8:411–22.
Duncan E. The nature and use of consensus methodology in practice. In: Kielhofner G, editor. Research in occupational therapy: methods of inquiry for enhancing practice. Philidelphia: F.A. Davis; 2006. p. 401–10.
Harvey N, Holmes CA. Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract. 2012;18(2):188–94.
Elliott T, Shewchuk R. Using the nominal group technique to identify the problems experienced by persons living with severe physical disabilities. J Clin Psychol Med S. 2002;9(2):65–76.
Miller D, Shewchuk R, Elliot TR, Richards S. Nominal group technique: a process for identifying diabetes self-care issues among patients and caregivers. Diabetes Educ 2000;26(2):305–10, 312, 314.
Drennan V, Walters K, Lenihan P, Cohen S, Myerson S, Iliffe S, et al. Priorities in identifying unmet need in older people attending general practice: a nominal group technique study. Fam Pract. 2007;24(5):454–60.
Bond CM, Watson MC. The development of evidence-based guidelines for over-the-counter treatment of vulvovaginal candidiasis. Pharm World Sci. 2003;25(4):177–81.
Tully MP, Cantrill JA. Exploring the domains of appropriateness of drug therapy, using the nominal group technique. Pharm World Sci. 2002;24:128–31.
Gastelurrutia MA, Benrimoj SI, Castrillon CC, de Amezua MJ, Fernandez-Llimos F, Faus MJ. Facilitators for practice change in Spanish community pharmacy. Pharm World Sci. 2009;31(1):32–9.
Cameron AJ, MacKeigan LD. Development and pilot testing of a multiple mini-interview for admission to a pharmacy degree program. Am J Pharm Educ 2012;76(1):10.
Linstone HA, Turoff M. The Delphi survey: method techniques and applications. Reading: Addison-Wesley; 1975.
Campbell SM, Cantrill JA. Consensus methods in prescribing research. J Clin Pharm Ther. 2001;26(1):5–14.
Chan A, Tan SH, Wong CM, Yap KY, Ko Y. Clinically significant drug–drug interactions between oral anticancer agents and nonanticancer agents: a Delphi survey of oncology pharmacists. Clin Ther. 2009;31(Pt 2):2379–86.
Dean B, Barber N, Schachter M. What is a prescribing error? Qual Health Care. 2000;9:232–7.
Mackellar A, Ashcroft DM, Bell D, James DH, Marriott J. Identifying criteria for the assessment of pharmacy students’ communication skills with patients. Am J Pharm Educ. 2007;71(3):50.
Campbell SM, Cantrill JA, Roberts R. Prescribing indicators for UK general practice: Delphi consultation study. BMJ. 2000;321:1–5.
Baysari MT, Westbrook JI, Egan B, Day RO. Identification of strategies to reduce computerized alerts in an electronic prescribing system using a Delphi approach. Stud Health Technol Inform. 2013;192:8–12.
Cantrill JA, Sibbald B, Buetow S. Indicators of the appropriateness of long term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility and reliability. Qual Health Care. 1998;7:130–5.
Aljamal M, Ashcroft DM, Tully MP. Development of indicators to assess the quality of medicines reconciliation at hospital admission: an e-Delphi study. Int J Pharm Pract 2016 (in press).
Aronson BD, Janke KK, Traynor AP. Investigating student pharmacist perceptions of professional engagement using a modified Delphi process. Am J Pharm Educ. 2012;76(7):125.
Cassar Flores A, Marshall S, Cordina M. Use of the Delphi technique to determine safety features to be included in a neonatal and paediatric prescription chart. Int J Clin Pharm. 2014;36(6):1179–89.
McDermott JH, Caiola SM, Kuhn KF, Stritter FT, Beza J. A delphi survey to identify the components of a community pharmacy clerkship. Am J Pharm Educ. 1995;59(4):334–41.
Fitch K, Bernstein SJ, Aguilar MS, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA Appropriateness Method User’s Manual. RAND corporation 2001 [cited 2015 July 11]; Available from: URL: http://www.rand.org/pubs/monograph_reports/MR1269.
McBride AJ, Pates R, Ramadan R, McGowan C. Delphi survey of experts’ opinions on strategies used by community pharmacists to reduce over-the-counter drug misuse. Addiction. 2003;98(4):487–97.
Avery AJ, Savelyich BSP, Sheikh A, Cantrill J, Morris CJ, Fernando B, et al. Identifying and establishing consensus on the most important safety features of GP computer systems: e-Delphi study. Inform Prim Care. 2005;13:3–11.
Plumridge RJ. Forecast of the future of hospital pharmacy in Australia. Am J Hosp Pharm. 1981;38(10):1469–72.
Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393.
Ahmed R, McCaffery KJ, Aslani P. Development and validation of a question prompt list for parents of children with attention-deficit/hyperactivity disorder: a Delphi study. Health Expect 2015 Jan 19. doi:10.1111/hex.12341.
Mease PJ, Arnold LM, Crofford LJ, Williams DA, Russell IJ, Humphrey L, et al. Identifying the clinical domains of fibromyalgia: contributions from clinician and patient Delphi exercises. Arthritis Rheum. 2008;59(7):952–60.
Conflicts of interest
About this article
Cite this article
McMillan, S.S., King, M. & Tully, M.P. How to use the nominal group and Delphi techniques. Int J Clin Pharm 38, 655–662 (2016). https://doi.org/10.1007/s11096-016-0257-x
- Consensus methods
- Delphi technique
- Nominal group technique