An online international comparison of thresholds for triggering a negative response to the “Surprise Question”: a study protocol
The Surprise Question (SQ) “would I be surprised if this patient were to die in the next 12 months?” has been suggested to help clinicians, and especially General Practitioners (GPs), identify people who might benefit from palliative care. The prognostic accuracy of this approach is unclear and little is known about how GPs use this tool in practice. Are GPs consistent, individually and as a group? Are there international differences in the use of the tool? Does including the alternative Surprise Question (“Would I be surprised if the patient were still alive after 12 months?”) alter the response? What is the impact on the treatment plan in response to the SQ? This study aims to address these questions.
An online study will be completed by 600 (100 per country) registered GPs. They will be asked to review 20 hypothetical patient vignettes. For each vignette they will be asked to provide a response to the following four questions: (1) the SQ [Yes/No]; (2) the alternative SQ [Yes/No]; (3) the percentage probability of dying [0% no chance – 100% certain death]; and (4) the proposed treatment plan [multiple choice]. A “surprise threshold” for each participant will be calculated by comparing the responses to the SQ with the probability estimates of death. We will use linear regression to explore any differences in thresholds between countries and other clinician-related factors, such as years of experience. We will describe the actions taken by the clinicians and explore the differences between groups. We will also investigate the relationship between the alternative SQ and the other responses. Participants will receive a certificate of completion and the option to receive feedback on their performance.
This study explores the extent to which the SQ is consistently used at an individual, group, and national level. The findings of this study will help to understand the clinical value of using the SQ in routine practice.
Clinicaltrials.gov NCT03697213 (05/10/2018). Prospectively registered.
KeywordsPrognosis Palliative care Surprise question Survival Death
British Medical Association
Clinical Commissioning Groups
General Data Protection Regulation
Local Medical Committees
Participant Information Sheet
Royal College of General Practitioners
University College London
Predicting how long a patient has left to live is an inaccurate and complicated clinical skill [1, 2, 3, 4, 5]. Yet it is a clinical skill expected of all medical doctors . In the UK, early identification of people approaching the last year of life can facilitate referral to specialist services and funding, which in turn improves the patient’s quality of life [7, 8, 9, 10, 11, 12]. The task of early identification of these patients often falls on General Practitioners (GPs) in the community. The timely identification of patients in the last 12 months of life has been described by GPs as one of the main challenges they face . It has been reported that GPs often wait until very close to death before discussing end of life issues  which can often prevent the individual experiencing a “good death” [15, 16]. The “Surprise Question” (SQ) is one such tool available to identify those who might benefit from palliative care. The SQ describes the process whereby clinicians are encouraged to ask themselves whether or not they would “be surprised” if a patient were to die within a specific time (e.g. 12 months, 6 months, 1 month, 7 days). It is suggested that if a clinician “would not be surprised” if a patient were to die within, for example, the next 12 months then this should act as a trigger to adopting a more palliative care approach to their care (e.g. placing the patient on an end-of-life communication register or referring them to specialist palliative care services). Although the original aim of the SQ was not simply to predict survival but to identify people with palliative care needs [17, 18], it is often used as a means to identify people who might be in the last 12 months of life. The prognostic capability of the SQ has been reported to be variable [19, 20, 21].
One problem with the SQ is that it is not clear to what extent a death needs to be expected before a clinician would be “surprised” if it did or did not occur. It has previously been highlighted by GPs how subjective and difficult the SQ can be in clinical practice . If there are large variations in these “trigger” values, or “threshold levels”, between clinicians then this would have important implications for how responses to the SQ should be interpreted. The variation between individuals’ threshold levels, and the consistency of the responses to the SQ, have not previously been evaluated. It is also not known which other factors may affect trigger values (e.g. nationality, age, gender, palliative care experience, or years of seniority). A better understanding about when clinicians would, or would not, be “surprised” if a patient were to die would help to standardise and calibrate the use of the SQ. Being aware of the threshold probabilities which trigger “surprise” in different groups may also lead to strategies to improve the accuracy and the consistency with which the SQ is used in clinical practice.
Research by Carel Veldhoven and colleagues [23, 24] reported that the use of an additional alternative Surprise Question “would I be surprised if this patient is still alive after 12 months?”, following a negative response (“No”) to the original Surprise Question, improves the predictive value of the question. Incorporating this question, alongside the original SQ could help improve the prognostic capability of the tool in addition to focusing care where it is needed most. Further research is needed to validate this finding.
In addition to this, there is little evidence about the relationship between the SQ response and the course of action decided upon. What evidence is available suggests that the treatment decisions made at the end of life vary considerable depending on clinician and patient factors . Do GPs consistently pursue the same course of action for patients with similar symptoms, or for those with the same SQ response?
To determine how consistent General Practitioners are in their response to the SQ, and what their “threshold” level is in order to provide a negative response (“No, I would not be surprised”) to the Surprise Question
To determine the relationship between the original Surprise Question, the alternative Surprise Question (“Would I be surprised if this person were still alive after 12 months”), and the estimated probability of death occurring in the next 12 months.
To understand the course of action decided upon in relation to the prognostic estimates provided.
Methods: participants, interventions, and outcomes
The Surprise Study is a non-randomised multicentre online study with a single group of GPs who will complete 20 hypothetical patient summaries, or “vignettes”.
The online study will be distributed to GPs in the following participating countries: Belgium, Germany, Italy, The Netherlands, Switzerland, and the United Kingdom.
A convenience sample of GPs with a variety of years of experience and seniority across six European countries will be approached to participate.
Registered GP in one of the six participating countries
Able to read and understand the language in which the study is presented to them
Decline to participate
Participants will receive a Participant Information Sheet (PIS) if they receive an email to participate and/or they will be able to download a PIS directly from the study website. Informed consent will be embedded in the study website and will be obtained via four checkboxes before starting the study. The first will be to consent to participate. The second will be to consent to the results being used in future publications, research and educational packages. The third box will be consent to provide an email address in order to enable participants to log out and return, and for the research team to send reminder emails. The fourth consent box is optional and will ask participants if they wish to receive feedback on their performance. The participant will be reminded that they are free to withdraw at any time. On completion, a debrief page will be provided to remind the participant what the results will be used for. The contact details for the study team will also be displayed should they have any concerns or issues they wish to raise.
Once consented, participants will be asked to read through 20 vignettes of patients with advanced diseases but who have not yet been referred to specialist palliative care services.
To ensure an equal distribution of symptoms and disease groups, the vignettes were developed follow a guidance template, informed from the Gold Standards Framework Proactive Indicator Guidance , as well as disease specific grading measures (Additional file 2). Each country participating developed a 5 vignettes following this template, from which the final set of vignettes were selected.
We will adhere to the European Organisation for Research and Treatment of Cancer (EORTC) guidelines  for the translation process as much as possible within the confines of the resources available for this study. All study documents (the website content, the PIS, the certificate of completion, and the vignettes) will firstly be written in English, verified by the UCL study team. The forward translation into the languages of all countries participating in this study (English, German, Italian, and Dutch) will be completed independently by two translators per language. The translators will compare and form one translated set of documents. The documents will then be back translated to English by a third translator who was not involved in the forward translation. The UCL study team will then review the original version with the back translated version and any differences will be discussed between the translators until the differences have been resolved.
The primary outcomes will be the continuous estimates of probability of dying within the next 12 months for the SQ (0–100%).
Dichotomous response to the SQ (Yes/No)
Dichotomous response to the alternative SQ (Yes/No)
The options for the course of action selected by the participants in each vignette
(only if not completed at time point 1)
Reminder email (7 days)a
Reminder email (21 days)a
The Surprise Question
The alternative Surprise Question
EXCLUSION (if incomplete)
Twenty vignettes have been developed. This number of vignettes was estimated in order to keep the task burden to a minimum, to reduce attrition, and to collect enough data points to establish the threshold score for each participant.
We aim to recruit the target sample size of 600 doctors (100 per country) within 8 months of the study opening. This timeframe should be sufficient to allow GPs, with busy working schedules, to complete the study. The sample size is based on calculating the trigger probability to an acceptable level of precision. As there is no current evidence about the trigger probability level for individuals, we assumed a trigger probability of 50%. Using this conservative estimate, aiming at a 4% margin of error (equivalently a precision of 8%), with a level of confidence of 95%, we will need to recruit 600 participants. From previous research that approached a membership body of doctors to participate , we anticipate a response rate of approximately 10–15% and therefore will aim to approach 1000–1500 doctors per country.
An email or receipt of study information within a newsletter from a membership affiliation email
An email or word of mouth from a colleague who has participated/heard of the study (snow-balling)
In the UK for example, we could use Local Medical Committees (LMCs), Clinical Commissioning Groups (CCGs) administrators, and the newsletters of the British Medical Association (BMA) and the Royal College of General Practitioners (RCGP) to disseminate a link to the website to participants. In addition to this, we will ask any GP who participates to share the study link with eligible colleagues.
Methods: data collection, management, and analysis
Outcome data collection
Demographic and biographical information about the participants will be collected (location, job title, specialty, setting, age, gender, years of experience since qualifying), and information about their use of the SQ (frequency of use, confidence).
Would you be surprised if this patient were to die in the next 12 months? (Y/N) (The Surprise Question)
Would you be surprised if this patient were to remain alive after 12 months? (Y/N) (The alternative surprise question)
What do you think the probability is of this patient dying within the next 12 months? 0% (Certain survival) - 100% (Certain death)
What do you think this patient needs? (select more than one if appropriate)
This final question (Q4) will contain a list of potential treatment or care options. The precise wording of these questions will be developed and agreed with the European collaborators to make sure that the questions are culturally and linguistically meaningful.
At the completion of the study, participants will be able to download a certificate of participation. Once the study has closed, they will receive feedback on their performance (if they requested this on the consent page). It is hoped that these options, as well as the option to log out of the website and return at a later date, will promote participant recruitment and retention.
The database will be designed by a database specialist (CT). The database will be rigorously tested by the UCL study team (NW, LO, VV, & PS) prior to the study starting.
Data collection tools
The study is accessed by the participants online and as such, the data will be collected directly from participants.
Each participant will be able to complete a practice vignette prior to completing the main set of vignettes in order to familiarise themselves with the online environment. There will be no time limit for completion so that each participant will be able to take their own time. Participants will not be able to move on to the next page if required information is missing or if responses have been submitted in an incorrect format (e.g. incorrect email format). The UCL study team and the statistician will review the data being collected before, during, and at the start of the recruitment phase to ensure data integrity.
A detailed statistical analysis plan will be drawn up in a separate document prior to analysis.
Primary outcome analysis
This is an exploratory study to determine how the SQ is utilised by different GPs in different countries. For this reason, we will employ a per-protocol analysis where those participants who do not complete more than 15 vignettes or violate the protocol (e.g. putting the same answer for every vignette) will be removed from the analysis.
We will examine each participant’s responses to the 20 vignettes to calculate the ‘trigger probability’ (or ‘threshold’) for each individual. To calculate this threshold we will examine the participant’s responses to the SQ and the responses to the probability of dying in the next 12 months. We define the threshold as the probability level after which all responses to the SQ are ‘no’. We will also describe the level of uncertainty about this decision. This uncertainty will be calculated by looking at the difference between the lowest probability level with which the participant responded ‘no’ to the SQ and the threshold level.
Three example GP responses are described below to explain this further.
GP 1 responded to 20 vignettes. For each vignette the GP stated the probability of dying and a corresponding answer to the SQ (yes/no). The probability results ranged from 0 to 100%. With each probability the GP also stated ‘yes/no’ to the questions (for ease of display, the individual probability estimates have not been provided but the ‘Y’/‘N’ responses have been placed over the corresponding % to show the point at which the responses changed from Y to N):
GP 2 also responded to 20 vignettes.
GP 3 also responded to 20 vignettes.
Once the thresholds are calculated for all participants, we will use linear regression to explore the differences, if any, in thresholds between countries and other demographic detail collected.
Secondary outcome analysis
We will explore the relationship between the alternative SQ and the other responses. We will also describe the actions taken by the clinicians and explore the differences between groups.
Throughout the study, the research team will review recruitment figures and the study will close once each country has recruited 100 participants. Researchers at UCL will check the responses given by the participants to assess for compliance with the protocol. Participants may be excluded from the analysis if their response record strongly suggests that they did not comply with the study protocol (e.g. all items answered with the same response). The UCL Study Management Group (PS, LO, NW, CT, VV) will be responsible for overseeing the trial and will meet regularly (at least four times per year) to review recruitment figures.
This is a very low risk study. There are no expected side effects of our intervention and this study will not have a Data Monitoring Committee.
Data management & dissemination
All data will be handled in accordance with the General Data Protection Regulation (GDPR) 2018 .
Enable the participant to log out and back in to the series of vignettes at the same place
To contact participants who do not complete the study with a reminder email at the one week and 3 week time points after the last website visit.
The participants will not be asked for any other personal identifiable information. During the study, all data will be kept securely on a web-based database, which is encrypted and password protected. The database will be accessible to approved members of the UCL research team only. Once all data have been reviewed, the names and email addresses will be retained for 12 months after publication, at which point all personal information will be destroyed and only the participant ID will be referenced. This will be explained clearly in the PIS and consent will be sought.
Access to the data
In the study, demographic data and outcome data will be collected from participants in accordance with the PIS and this protocol.
The final study database will be downloaded from the website by the research team (Marie Curie Palliative Care Research Department, Division of Psychiatry, UCL) for statistical analysis and UCL will act as the data controller of such data for the study.
The research team (Marie Curie Palliative Care Research Department, Division of Psychiatry, and UCL) will process, store and dispose of the final study database in accordance with all applicable legal and regulatory requirements, including the General Data Protection Regulation (GDPR) and any amendments thereto. Data will be stored electronically, on an encrypted hard drive, with restricted access. The hard drive is maintained by UCL and routinely backed up.
The anonymous dataset will be shared with the other research groups participating in the study.
Record keeping and archiving
At the end of the study, all essential documentation will be archived securely by the CI for a minimum of 20 years from the declaration of end of study.
Essential documents are those which enable both the conduct of the study and the quality of the data produced to be evaluated and show whether the site complied with all applicable regulatory requirements.
The sponsor will notify the site when study documentation can be archived. All archived documents will be continued to be available for inspection by appropriate authorities upon request.
Study results will be published in peer-reviewed, indexed, journals using an open access format, and the results will be presented at national and international conferences. Authorship eligibility will be in accordance with The International Committee of Medical Journal Editors. All proposed publications will adhere to UCL publication policy.
Marie Curie supports this study and we will work with them to disseminate findings through blogs, newsletters and social media.
Four members of the Marie Curie Expert Research Voices Group have reviewed a summary of this study to ensure its aims, objectives, study design and outputs are in line with patient and family needs. All respondents felt it was important to identify patients who may require palliative care early, and they felt the Surprise Question could be a useful way to get doctors to think about the sorts of patients who may or may not require palliative care. Other comments included respondents’ own experiences when a loved one died. These comments were taken in to consideration when developing the study documents.
It is expected that the recruitment of GPs will be challenging due to the nature of their work. We further anticipate that attrition may be an issue due to time constraints on the participants. We have added a function for the doctor to “log out” of the study and return at a more convenient time. We have also endeavoured to minimise the number of vignettes. In addition to this, we will keep the study open as long as possible to try and maximise recruitment. We have adopted recruitment strategies that have proven to be successful in another study which also recruited GPs from the UK .
We would like to thank the members of the PPI group at Marie Curie Expert Voices for their invaluable feedback on the study documents.
This study is partially funded by the Marie Curie I-CAN-CARE Program grant (MCCC-FPO-18-U). Nicola White, Linda Oostendorp, Victoria Vickerstaff, and Patrick Stone are partly supported by the UCLH NIHR Biomedical Research Centre. There are no financial interests by the CI, PI, or study management members. University College London (UCL) is the sponsor. The sponsor will have no role in the design, collection, analysis, interpretation, or writing of this data.
Availability of data and materials
The study will be led by the group based in the UK. There will be a group of lead researchers in each country who will manage the project within each country. United Kingdom: NW will create the vignettes and the online study, recruitment, analyse the data, and write up the results. LO will assist in the creation of the online study, recruitment, analyse the data, and write up the results. PS will oversee all actions of the study team. PS will oversee the analysis and will contribute to the final report. VV will provide support regarding the statistical analysis of the data and will review the statistical content within the final report. CT will build the online website and database which will hold the data gathered from the participants. CA will assist in the study design and will assist in the write up of the results. The researchers in all participating countries (CG, YE, MM, JW, BL, GB, SZ, SE, RC, GO, CV) helped to create the vignettes and the online study, the translation of study materials into each of the applicable languages, support with recruitment, and interpretation, write up and dissemination of the results. All authors have read and approved the manuscript.
Ethics approval and consent to participate
This study gained ethics approval from University College London REC on the 9th August 2018 (ref 8675/003), University of Antwerp REC on the 7th January 2019 (ref 18/50/589), University of Mainz REC on the 26th November 2018, Radboud University Medical Center REC on the 20th December 2018 (ref 2018–4949), Bern Ethics Committee on the 29th August 2018 (ref 2018–00710) and University of Bologna REC on the 25th January 2019 (ref 12,590). In case any protocol amendments are required, these will be reviewed by the study team before submission to the relevant committees, and the trial registry will be updated where necessary. Consent is obtained electronically through a series of check boxes, see section “Methods, Methods: Participants, interventions, and outcomes” for further detail.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 3.Maltoni M, Caraceni A, Brunelli C, Broeckaert B, Christakis N, Eychmueller S, Glare P, Nabal M, Viganò A, Larkin P, et al. Prognostic factors in advanced Cancer patients: evidence-based clinical recommendations—a study by the steering Committee of the European Association for palliative care. J Clin Oncol. 2005;23(25):6240–8.CrossRefGoogle Scholar
- 8.Seow H, Brazil K, Sussman J, Pereira J, Marshall D, Austin PC, Husain A, Rangrej J, Barbera L. Impact of community based, specialist palliative care teams on hospitalisations and emergency department visits late in life and hospital deaths: a pooled analysis. BMJ. 2014;348:3496.CrossRefGoogle Scholar
- 13.Mitchell S, Loew J, Millington-Sanders C, Dale J: Providing end-of-life care in general practice: findings of a national GP questionnaire survey. Br J Gen Pract 2016:bjgpsep-2016-2066-2650-mitchell-fl-p.Google Scholar
- 15.The End of Life Care Strategy. In. Edited by Department of Health; 2008.Google Scholar
- 21.Mitchell GK, Senior HE, Rhee JJ, Ware RS, Young S, Teo PC, Murray S, Boyd K, Clayton JM. Using intuition or a formal palliative care needs assessment screening process in general practice to predict death within 12 months: a randomised controlled trial. Palliat Med. 2018;32(2):384–94.CrossRefGoogle Scholar
- 22.Elliott M, Nicholson C. A qualitative study exploring use of the surprise question in the care of older people: perceptions of general practitioners and challenges for practice. BMJ Support Palliat Care. 2014:bmjspcare–2014-000679.Google Scholar
- 23.Veldhoven C, Weijers F, Verhagen S, Vissers KYE. Combining the original and alternative surprise question for the early identification of palliative patients: a Dutch pilot study. In: European Association of Palliative Care. Madrid. 2017:189. http://www.eapc-2017.org/files/EAPC17/dl/EJPC-Abstract-Book-2017.pdf.
- 25.Hinkka H, Kosunen E, Lammi U-K, Metsänoja R, Puustelli A, Kellokumpu-Lehtinen P. Decision making in terminal care: a survey of Finnish doctors' treatment decisions in end-of-life scenarios involving a terminal cancer and a terminal dementia patient. Palliat Med. 2002;16(3):195–204.CrossRefGoogle Scholar
- 26.Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT). http://www.spirit-statement.org/. Accessed 1 Apr 2019.
- 27.The Gold Standards Framework Proactive Identification Guidance (PIG). http://www.goldstandardsframework.org.uk. Accessed 1 Apr 2019.
- 28.European Organisation for Research and Treatment of Cancer (EORTC): EORTC quality of life group translation procedure. In., 4th edn; 2017.Google Scholar
- 30.White N, Harries P, Harris AJL, Vickerstaff V, Lodge P, McGowan C, Minton O, Tomlinson C, Tookman A, Reid F, et al. How do palliative care doctors recognise imminently dying patients? A judgement analysis. BMJ Open. (in press).Google Scholar
- 31.White N, Harries P, Harris AJ, Vickerstaff V, Lodge P, McGowan C, Minton O, Tomlinson C, Tookman A, Reid F et al: 44 an evidenced-based heuristics model (or rule of thumb) to improve doctors’ intuition about when patients are imminently dying. BMJ Support Palliat Care 2018, 8:376–376.Google Scholar
- 32.General Data Protection Regulation (GDPR). https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/. Accessed 1 Apr 2019.
- 33.Curtis P, Taylor G, Harris M. How preferred learning approaches change with time: a survey of GPs and GP specialist trainees. Education for Primary Care. 2018:1–6.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.