The NGT is a highly structured face-to-face group interaction, which empowers participants by providing an opportunity to have their voices heard and opinions considered by other members . It was designed by Delbecq and Van de Ven and comprises four key stages: silent generation, round robin, clarification and voting (ranking or rating) . These stages are briefly explained below.
How to run the nominal group technique
While groups of between two and fourteen participants have been used in nominal group research (Table 1), a maximum of seven has been recommended . A nominal group generally involves one to two questions which are sent to participants in advance. At the beginning of the meeting, participants are given up to twenty minutes to silently reflect or record their individual ideas in response to a question, i.e. silent generation . The facilitator then asks one participant at a time to state a single idea to the group in a ‘round robin’ fashion. Participants are able to think of new ideas during this process, but must wait their turn before they can share with the group. This stage takes as much time as needed until no new ideas are forthcoming. It is recommended that there be no discussion at this stage and ideas are merely recorded verbatim on, for example, a flipchart or white board .
The third stage is clarification of the ideas, which also provides the opportunity for a grouping step, where similar ideas are grouped together with agreement from all participants. Participants may also exclude, include or alter ideas, as well as generate grouping themes . All ideas should be discussed to ensure participant understanding , thus enabling them to make an informed decision when they come to voting on ideas. Facilitators should emphasise that participants do not have to agree with all ideas listed as, at the end of the clarification stage participants are able to ignore ideas by voting on personal preferences. The round robin  and clarification phase  can take up to 30 min each. Facilitators should not direct participants during the clarification process, which may make this stage particularly difficult.
Participants are then provided with a ranking sheet, where they are asked to select their top preferences from the generated ideas. The number of items chosen by participants depends on the topic, but the ranking of five ideas is common in the literature [2, 5, 10]. The facilitator should specify that a number should be allocated to each selected item, with larger numbers reflecting greater importance [2, 5]. For example, for five ideas, the most important idea is scored five points. Although there is no anonymity for participants during nominal group discussions, individual scoring on a ranking sheet is confidential. Finally, the scores for each idea are summed and presented to the group for discussion. The timing for this stage is likely to depend on a number of factors, including the complexity of the topic and how many items need to be prioritised (the more items to rank, the harder the process and more time consuming it can become). Dening et al.  noted that voting could take up to 10 min to complete.
Ultimately, the time to complete one nominal group is variable, and depends on group size, how many questions are asked, and the type of participants involved. For example, Bradley et al.  documented a 2-h time limit to conduct a NGT for one question, whereas Hutchings and colleagues allocated half a day to conduct a NGT for two questions, followed by another half-day for a forum event  (see “Variations on the nominal group technique”).
Variations on the nominal group technique
The NGT is a highly adaptable method, and can be used in addition to  or to inform, other methods, e.g. a discrete choice experiment . NGT variations may be influenced by the available research and participant time, or the level of clarification, consensus or generalisability required for the topic. Ultimately, researchers need to ensure that the NGT is working for each participant group; it may be that stages need to be adapted. For example, for indigenous or culturally and linguistically diverse populations, it may be the cultural norm to discuss ideas as a group. Thus, a more appropriate variation to the process for generating ideas has been to combine the round robin and clarification stages . Other variations could be in direct response to participant ability. If it is too difficult for participants to group similar ideas at the clarification stage then this grouping step could be avoided altogether . While this may make it harder for participants to vote, i.e. there is a longer list of ideas to consider, it may cause less frustration for participants.
Generally, variations are seen in relation to generating ideas, and how ‘consensus’ is obtained from participants, i.e. the ranking process (Fig. 1):
Generating ideas instead of silent generation followed by a round robin, ideas are obtained from a literature review, or exploratory surveys are used which could be viewed as a way to achieve greater consultation [14, 15];
Ranking this may be completed by either allocating a score or by a rating on a Likert scale ;
Re-ranking allowing participants to revise their original ranking, i.e. re-ranking, either in the original NGT meeting , via a secondary survey , or obtaining validation by sending a survey of nominal group results to other participants . Alternatively, the re-ranking process could continue until no further changes are seen with the most important ideas .
Where separate nominal groups are held for similar participants, e.g. consumer groups, health professional groups or stakeholder groups, a mixed-forum event can provide the opportunity for consensus to be achieved by forming new groups with different participant types . In a study that exemplified the use of a mixed forum event, Hutchings and colleagues asked previous participants to individually review the overall NGT results (overarching themes), and to rank the themes (pre-forum responses). At the forum, participants were asked to discuss the pre-forum responses in their newly allocated groups, which consisted of participants from differing backgrounds. Individuals were then asked to re-rank themes for a third time.
Other researchers have provided valuable information on important nominal group design considerations , its practical application  and method of analysis . Black et al.  reviewed the literature to identify the evidence for certain ‘best practice recommendations’ for consensus methods. While that review provides some important considerations for researchers wishing to use these methods, the articles included are, at a minimum, over 15 years old. Using specific examples, Tully and Cantrill  discuss the steps involved in a nominal group, and guidance for researchers with respect to group composition. While a discussion of qualitative and quantitative analysis is also included, McMillan et al.  take this one step further in their paper by detailing the entire analysis process for researchers who undertake more than ten nominal groups.
Applications to pharmacy research
The NGT has been applied in numerous healthcare settings, to develop guidelines  or explore opinions of different health professionals , lay people and carers [10, 19, 20], or to compare views of both parties [9, 21]. It is gradually building traction within the pharmacy setting, as seen in Table 1. Researchers have generated evidence based guidelines or criteria for pharmacy practice situations [7, 22, 23], informed practice change [11, 24] and the profession [12, 16] about particular topics, and identified attributes to be included when interviewing pharmacy students .