Abstract
Background
The concept of the regulatory sandbox—a safe space for testing new regulatory processes—was first used within the financial technologies (FinTech) sector, but has since expanded into other sectors, including healthcare.
Objectives
This review aims to describe the extent of use of sandboxes in healthcare and assess the potential for the sandbox approach to be used to test and develop emerging health technology assessment (HTA) methods, policies and processes for innovative technologies.
Methods
A systematic literature review was undertaken to identify published papers and reports that described and/or assessed the use of sandboxes in the healthcare sector. Searches were conducted in Medline, Embase, Econlit, Social Policy and Practice, and Health Management Information Consortium databases from inception to March 2020. Free-text Google search was also conducted to identify relevant grey literature. Only papers and reports discussing or evaluating the use of sandboxes in healthcare settings and published in English were included. Included studies were qualitatively summarised using a thematic analysis approach.
Results
Overall, 46 papers and reports were included. The topics covered were classified into 4 major themes: history of the regulatory sandbox, the sandbox as a testing environment, the sandbox as a regulatory approach, examples of using sandboxes in healthcare. Findings show that the use of regulatory sandboxes in healthcare is relatively new and primarily used in high-income countries to support the adoption of new technologies, particularly those related to digital health. Recommendations are made based on these findings to guide its use in HTA policy and methods development.
Conclusions
Sandboxes are increasingly used within healthcare regulation. Despite its potential, this approach has not been used in HTA policy and methodological developments to date. HTA agencies should consider this approach to facilitate developing policies, methods and processes for innovative and disruptive health technologies. Transferability to low- and middle-income countries’ settings, however, should be assessed.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
The sandbox approach originated in the financial technologies sector and since then has started to be used within healthcare for developing regulatory policies relating to innovative technologies. |
This approach has considerable potential to be used in accelerating health technology assessment (HTA) methods and policy developments, particularly in response to new innovations. |
The success of applying this approach in HTA should be facilitated by consortium working, clarity regarding scope, focus on innovative technologies with no clear appraisal pathway and the use of an anticipatory model. |
1 Introduction
The development of innovative products, services and business models is often delayed or prevented due to regulatory and governance restrictions; however, innovation is crucial in ensuring economic growth and societal benefit [1]. As technological advancements, such as the use of big data and artificial intelligence (AI), start to emerge, regulatory and health technology assessment (HTA) agencies are required to evolve to allow these potentially disruptive innovations to be implemented for the benefit of society [2,3,4].
A range of mechanisms are being adopted by regulators to support innovation while also delivering appropriate regulatory oversight, through what is referred to as “anticipatory regulation”, where regulation is seen as a support tool for safeguarding responsible innovation, rather than a barrier [5]. One such mechanism is live-testing environments such as ‘sandboxes’ [1]. These allow innovators to trial products, services, and business models in a safe space, to confirm their compliance with existing regulation before implementing them within the wider sector. The UK’s Financial Conduct Authority (FCA) first used the sandbox environment for exploring regulatory issues in 2016 [6]. This regulatory sandbox was described ‘a safe space in which businesses can test innovative products, services, business models and delivery mechanisms without immediately incurring all the normal regulatory consequences.
The sandbox environment also allows exploration of processes that may violate current rules and regulations but have the potential to reap a large benefit if introduced into standard practice [7]. These testing environments have also been referred to as ‘testbeds’ or ‘living labs’, which can lead to confusion around their definition within the literature [8].
This review aims to provide an overview of sandboxes and provide examples of their current use and applications within the healthcare sector. It also aims to discuss the potential of using these environments by HTA agencies for trialling innovative ways of assessment and funding new technologies, including AI-based digital technologies and advanced therapy medicinal products, before their widespread adoption and implementation [9].
2 Methods
A scoping review was undertaken to identify published papers and reports that described or assessed the use of sandboxes in the healthcare sector. The search strategy is presented in the Supplementary Material. It was run across the following databases simultaneously via Ovid Interface: Medline (1946–March 2020), Embase (1974–March 2020), Econlit (1886–March 2020), Social Policy and Practice (1890s–March 2020) and Health Management Information Consortium (1979–March 2020).
A similar search strategy was employed for search engine (Google/Google Scholar) use, mainly focused around “policy sandbox”, “regulatory sandbox”, “sandbox” and the various healthcare search terms listed in the search strategy presented in the Supplementary Material. The search was limited to articles published in English. We did not apply any publication date limits.
The search was conducted on 11th March 2020. The records identified were exported into an Excel Database and checked for inclusion by two reviewers (EL and DD) independently. The main inclusion criterion was that the article should focus on the definition or application of policy or regulatory sandboxes in the healthcare sector. These articles could be scientifically published papers or grey literature (government reports, organisational reports). Articles were excluded if they were published in a language other than English, related to sectors than healthcare or used the term sandbox in an unrelated context (e.g., children’s playgrounds). There was no time cut-off in the exclusion criteria. Any disagreements between the two reviewers were resolved through discussion and consensus building. Extracted information included country, sector, main stakeholders involved, aim of the sandbox, technologies/interventions tested, the year it was reported and its active status. The themes extracted from the included articles were mapped by a single reviewer (EL). These themes were then discussed with the second reviewer (DD) and a final list was agreed upon between the two. The extracted themes were then described and discussed narratively. The reviewers followed PRISMA guidelines for conducting and reporting of reviews.
3 Results
The database search identified 141 records. After deduplication, including articles identified from additional sources and the grey literature and applying the inclusion/exclusion criteria, a total of 46 articles and reports were finally included [1, 5,6,7,8, 10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33, 36,37,38,39, 44,45,46,47,48,49,50,51,52, 55,56,57,58]. The included articles and reports are described in more detail in the Supplementary Material. Figure 1 presents a flow chart of the search process. A diagram presenting the main themes identified in the included articles and reports is also presented (see Fig. 2). These themes are discussed in turn below.
3.1 History of the Regulatory Sandbox
The concept of “the regulatory sandbox”—was first considered in 2015 by Mark Walport, former chief scientific advisor of the UK Government, in the review ‘FinTech Futures’ [10]. The financial technologies (FinTech) sector is concerned with the ‘development and commercialisation of new financial business models and disruptive innovation’. However, as the sector also exposes new methods for business and consumer fraud and exploitation, it is key that both the government and regulators maintain an oversight of activity within the sector to mitigate any potential risk.
Walport recommended that the financial sector develop a similar system to that used in the pharmaceutical sector, where new biomedical approaches are assessed using clinical trials, allowing testing, and monitoring to be done in a safe, controlled environment. The review recommended that the financial sector develop a similar system, allowing new technologies and consumer propositions to be tested in a safe environment without the threat of destabilising current financial systems [10]. This system was referred to as a ‘sandbox’, where regulators, institutions and FinTech companies could collaborate within virtual environments, and even with real data.
Following these recommendations, the UK’s Financial Conduct Authority (FCA) published a report referencing the feasibility of introducing a ‘regulatory sandbox’ as part of its Project Innovate programme, which aims to challenge regulatory barriers to allow innovation within the financial sector [6]. The regulatory sandbox was described as ‘a safe space in which businesses can test innovative products, services, business models and delivery mechanisms without immediately incurring all the normal regulatory consequences’.
The FCA regulatory sandbox was launched in June 2016, with the framework piloted using a reduced number of businesses in the first year. Since then, the regulatory sandbox concept has moved beyond the financial sector and is now being utilised across multiple sectors across the globe. One area of interest that has emerged is how these live-testing environments can be used within health and social care settings. This potential use was first realised through the set-up of the Care Quality Commission’s (CQC) regulatory sandbox—the first regulatory sandbox for healthcare innovation in the UK [11].
3.2 The Sandbox as a Testing Environment
Arntzen et al [8] categorised testing environments into different groups depending on two factors: how controlled the environment is, and at which point the approach is used in the product development process. The diagram in Fig. 3, adapted from Arntzen et al [8], explains the key differences between common testing environments used in product innovation in terms of the level of control in the environment and the stage in development.
“Living labs” can be defined as “user-centred open innovation ecosystems based on a systematic co-creation approach” [8]. They operate in simulated or real-world environments during the earliest stages of product development, encouraging all stakeholders to collaborate early in the innovation process.
“Sandboxes” allow innovators to trial these products, services and business models in a safe space, to confirm their compliance with existing regulation before implementing them within the wider sector. These environments also allow exploration of processes that may violate current rules and regulations but have the potential to reap a large benefit if introduced into standard practice [7]. They operate further along in the product development process.
Finally, “real-world test beds” are controlled environments for testing innovative products in real-world conditions in the environments in which they will be used or operated. These are useful in the latest stages of product development when the product is close to implementation. Laboratories and simulated/constructed environments can also be classified as test beds but have varying degrees of real-world components compared to real-world test beds.
3.3 The Sandbox as a Regulatory Approach
According to Armstrong and Rae, Regulation can be split into three categories of approaches: advisory, adaptive and anticipatory [12]. A sandbox’s position within these categories is determined by several factors; its goal, its desired outcome and the participants involved in it (Fig. 4). Armstrong and Rae [12] described the sandbox approach as an example of “anticipatory regulation”, where regulation is seen as a support tool rather than a barrier to innovation. Anticipatory regulation is designed to anticipate future challenges, and support innovators when faced with them.
Advisory approaches aim to aid new products and services in meeting existing regulations. The outcome of these approaches is a change to the new product or service, and usually only involves the regulators and innovators.
Adaptive approaches are more flexible and aim to support innovation through the adaptation of current regulations to suit the new product or service. These approaches result in not only product changes, but also changes to policy and regulation.
Finally, anticipatory approaches are the most flexible, as these aim to develop regulation in an iterative process alongside the advancement of new products and services. The outcome of these approaches is often a more comprehensive understanding of regulatory requirements, and therefore requires input of a much more varied selection of stakeholders. The risks and uncertainty increase with each approach; however, anticipatory approaches are the most proactive and flexible, so may lead to increased improvement in the long term.
Anticipatory regulatory approaches can also be described according to six key principles: inclusive and collaborative, future-facing, proactive, iterative, outcomes-based and experimental [5]. For a sandbox to truly be anticipatory in its approach, it must consider all these principles within its intended activity and outcomes.
3.4 Examples of Using the Sandbox Approach in Healthcare
Sandboxes have the potential to be beneficial in driving forward improvement of health and social care services. In this context, the regulatory sandbox aims to not only improve healthcare experience and outcomes, but also to improve the experiences of healthcare providers, commissioners, and regulators. The sandbox participants collaborate to determine ‘what good looks like’, and to design and drive innovation without the associated risks [13]. The examples identified from the published papers and reports have been categorised as “outcomes-focused” or “data-focused”, depending on their aims and outputs. These sandboxes are summarised below and in Table 1 and described in detail in the Supplementary Material. These detailed discussions include examples of how the sandboxes have led to changes in related policy or processes.
3.4.1 Outcomes-Focused
Sandboxes included under this category focus on optimising patient outcomes achieved from the adoption or roll-out of interventions or service models. The majority of the interventions tested were digital health-related such as digital triage tools, the use of machine learning (ML) applications within diagnostics, interventions related to tele-medicine and mobile health, and the use of blockchain technology. Additionally, sandboxes have been set up to test service models such as those focused on the restructure of social care services and determining the appropriate staff skill-mix to support care home services.
3.4.2 Data-Focused
While electronic health records (EHRs) are becoming more common within healthcare as healthcare systems adopt new technology, issues persist relating to access to EHRs for biomedical and clinical research. However, the real-world data within these records are invaluable in progressing research and development, and therefore solutions to these issues are paramount in ensuring these sources are utilised effectively [33]. As a result, a related sandbox solution has become popular with organisations using electronic health data to facilitate testing of new technologies. These organisations either offer a service whereby they facilitate access to health data or are involved in the design of software tools to facilitate access and processing of health data for product testing.
4 Discussion
4.1 Summary of Findings
This review identified 46 articles and reports that covered the principles, development and use of the sandbox approach in healthcare. A thematic analysis of these articles and reports identified four key themes covering the history and development of this approach, examples of its use in healthcare and applying it to facilitate regulating and access to innovations. The examples identified from the literature highlight the potential usefulness of this approach for developing and implementing innovations in HTA methods, processes and policies in response to new and disruptive technology development.
4.2 Benefits of the Sandbox Approach
The main benefits of the sandboxes identified in this review include reduction in time and cost of getting products to market as delays due to regulatory uncertainty have a larger impact on first-time innovators [6]. Increased access to finances for innovators is also reported, as innovation is reliant on sufficient investment, and companies can struggle to raise funds due to regulatory uncertainty. Increased throughput of tested and introduced products to market is another important benefit, as regulatory uncertainty prevents the most innovative products from reaching the market, as they are often abandoned at early stages due to associated risks.
Improved collaboration between the regulator and innovators to ensure protection of consumers is also an important reported benefit. The sandbox approach has also been applied in numerous other sectors aside from FinTech and health, including energy and transport [34, 35]. The aim of these sandboxes differs between countries; for example the Singapore Energy Market Authority’s sandbox seeks innovations with the gas and electricity space, whereas the equivalent energy sandbox in Germany is focusing on increasing use of renewable energy sources [1]. These sectors similarly commented that the sandbox approach helped innovators and businesses to trial novel products, services and business models, to ensure viability before marketing. These benefits are equally applicable to HTA, given its pivotal role as a pre-requisite for access to innovative technologies in many countries, where early engagement with innovators can facilitate faster adoption of these technologies [3].
4.3 Approaches and Requirements for Developing a Policy Sandbox
Previous and ongoing sandboxes have yielded useful information on what approaches work when facilitating a sandbox programme [36]. One of the key measures identified for the success of sandboxes in general is stakeholders’ ability to collaborate effectively. This collaboration can be assisted using directed consortia, a partnership framework that provides collaborative environments for multiple, and sometime competitive, organisations with differing priorities [37]. Providing a neutral consortium environment facilitates this collaborative working. The problem of stakeholder engagement can also be seen in healthcare, whereby innovative technologies or products are often met with scepticism. Clinicians and patients may favour what is familiar to them rather than choosing to test new technologies. Thus, stakeholder engagement, including public and patient involvement, will be key in developing these innovations to ensure its uptake and diffusion within the healthcare system.
Consortia are often used during biomedical product translation. Research by consortium is also fast becoming a permanent fixture within biomedical R&D, with potential to be adapted for use by other areas of the healthcare sector [38].
The use of clear mission statements, targeted deliverables and timeframes, workstreams defined by participants involved, and expected milestones are key success factors [37]. Governance structures are used to define strategic decision making, with transparency being key to ensuring good collaboration and dissemination of results. Use of these features would provide clarity to those involved regarding their responsibilities and expectations during the testing process, as well as facilitating effective communication internally and externally. The use of eligibility criteria for access to the sandbox is also very important [36].
4.4 Potential Application of Sandboxes in HTA
While there is a selection of sandboxes focused on regulatory and data access challenges, there is limited sandbox activity that relates directly to HTA. Some projects currently being explored within the identified regulatory sandboxes are directly relevant to the HTA perspective, as they are considering advancements in innovative health technologies that do not currently have an established appraisal pathway within HTA agencies. Many of these are related to the advancement of digital technologies for diagnostics, disease management and prognostics such as the UK CQC sandbox, the UK NHS Test Beds Programme and the Health Data Research sandbox [11, 18, 26].
As digital health technology advances, HTA agencies must adapt to successfully appraise such technologies. NICE has published evidence standards relating to digital health technologies as part of a project funded by NHS UK [39]. The aim of the standards is to improve innovator and commissioner understanding of the levels of evidence required for digital healthcare technologies. While a sandbox has been used to explore regulation of these products in the UK, sandboxes have not been used to determine the best methods for assessing these products from an HTA perspective or the policies and processes that underpin these assessments. Developing these evidence standards to accommodate the new advances in digital technologies, such as AI- and ML-based innovations, can be accomplished within the context of HTA policy sandbox programme to facilitate early and continuous engagement with these technologies’ developers and other stakeholders with the aim of mutual learning and adaptation. Closer working with stakeholders including regulators, digital health developers, clinicians, patients, academics and guideline developers will facilitate an iterative process of product development and ensure faster access to and uptake of innovation in the healthcare system [4].
The appraisal of other innovative technologies such as advanced therapy medicinal products including gene and cell therapies and histology-independent cancer drugs also represent a challenge to HTA agencies given the lack of experience in appraising these technologies [40]. Horizon Scanning and Scientific Advice activities implemented by some HTA agencies have been useful in this context [2, 41]. However, these do not go a long way towards achieving this anticipatory environment that allow development of HTA policy and methods in response to new innovations. Using an anticipatory sandbox approach to facilitate the development of appraisal methods, policies, and processes and to adapt to the unique features of these disruptive innovations would enable HTA agencies and all stakeholders involved to have a constructive discussion that would facilitate faster uptake and patient access to these technologies while supporting innovation and developing HTA methods, processes and policies.
Thus, based on the findings of this review we recommend the following as the key elements to focus on when developing a HTA policy sandbox (Fig. 5):
-
Focusing on HTA policy and methods: there is limited activity within this area, as many sandboxes currently focus on regulatory issues only, and do not include HTA within the product development pathway. A HTA sandbox would therefore complement already established regulatory sandbox programmes.
-
Focus on future advances and innovations in healthcare that do not fit within an established appraisal pathway: there are many products that do not fit within current processes that have the potential to provide a large population benefit, particularly those making use of AI and ML approaches.
-
Using an anticipatory approach, as it is important that HTA prepares for future innovations rather than being reactive. Through an anticipatory approach, it is possible for HTA agencies to directly influence innovation, minimise risks, optimise outcomes and prevent population harm.
-
Using consortium working, to ensure successful collaboration between multidisciplinary stakeholders, creating outputs that are transferable across a variety of settings.
Our review supports establishing such sandboxes by HTA agencies, however, research is needed to understand how to embed these sandboxes in the organisational structure of current HTA agencies, managing the changes that accompany their implementation, and evaluate their benefits. It has to be pointed out as well that the examples identified in this review have been mainly from high-income countries, and most from the UK. This might be a result of language limitation. However, we have identified literature on regulatory sandboxes in sectors other than healthcare in the newer EU countries (the EU13) [42]. Despite previously being identified as having an ‘innovation gap’ compared to other older member states, some countries have chosen to adopt innovation hubs which are similar to regulatory sandboxes in their aims of providing regulatory guidance [43]. This supports the transferability of this concept to HTA agencies in low- and middle-income countries as well.
5 Conclusion
The sandbox approach has been used across a variety of sectors since its implementation in the finance sector. Healthcare regulatory agencies have developed sandboxes focused on the regulation of new healthcare innovations, primarily those related to digital health with successful outputs reported. HTA sandboxes could be useful in anticipating the challenges in assessing innovative technologies and future-proofing methods and processes.
References
Armstrong H, Bárd I, Engström E. Regulator approaches to faciliate, support and enable innovation: final report [Internet]. 2020. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/861078/regulator-approaches-facilitate-support-enable-innovation.pdf. Accessed 25 Mar 2020.
Euroscan International Network. A toolkit for the identification and assessment of new and emerging health technologies [Internet]. 2014. http://epapers.bham.ac.uk/2120/1/EuroScan_Methods_Toolkit_October_2014_FINAL_CC_added.pdf. Accessed 25 Mar 2020.
Tummers M, Kvaerner K, Sampietro-Colom L, Siebert M, Krahn M, Melien O, et al. On the integration of early health technology assessment in the innovation process: reflections from five stakeholders. Int J Technol Assess Healthc. 2020;36:1–5.
Vollmer S, Mateen B, Bohner G, Király F, Ghani R, Jonsson P, et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ. 2020;368:I6927.
Armstrong H, Gorst C, Rae J. Renewing regulation: anticipatory regulation in an age of disruption [Internet]. 2019. https://media.nesta.org.uk/documents/Renewing_regulation_v3.pdf. Accessed 25 Mar 2020.
Financial Conduct Authority. Regulatory sandbox [Internet]. 2015. https://www.fca.org.uk/publication/research/regulatory-sandbox.pdf. Accessed 25 Mar 2020.
Pratt A. The benefits of a regulatory sandbox. Vet Rec. 2017;181:573.
Arntzen S, Wilcox Z, Lee N, Hadfield C. Testing innovation in the real world: real-world testbeds [Internet]. 2019. https://media.nesta.org.uk/documents/Testing_innovation_in_the_real_world.pdf. Accessed 25 Mar 2020.
Accelerated Access Collaborative. Advanced Therapy Medicinal Products (ATMPs) [Internet]. 2020. https://www.england.nhs.uk/aac/wp-content/uploads/sites/50/2020/01/aac005-atmps.pdf. Accessed 25 Mar 2020.
Walport M. FinTech Futures: the UK as a world leader in financial technologies [Internet]. 2015. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/413095/gs-15-3-fintech-futures.pdf. Accessed 25 Mar 2020.
Care Quality Commission. Regulatory sandbox [Internet]. 2020. https://www.cqc.org.uk/what-we-do/how-we-work-people/regulatory-sandbox. Accessed 25 Mar 2020.
Armstrong H, Rae J. A working model for anticipatory regulation [Internet]. 2017. https://media.nesta.org.uk/documents/working_model_for_anticipatory_regulation_0.pdf. Accessed 25 Mar 2020.
Reid K. From enforcer to enabler: how regulatory sandboxes and adaptive approaches support to move from compliance to collaboration in health and social care [Internet]. 2018. https://www.careinspectorate.com/images/From_enforcer_to_enabler.pdf. Accessed 25 Mar 2020.
Department of Health and Social Care. The future of healthcare: our vision for digital, data and technology in health and care [Internet]. 2018. https://www.gov.uk/government/publications/the-future-of-healthcare-our-vision-for-digital-data-and-technology-in-health-and-care/the-future-of-healthcare-our-vision-for-digital-data-and-technology-in-health-and-care. Accessed 25 Mar 2020.
Care Quality Commission. Digital triage in health services: a report with recommendations from CQC’s first regulatory sandbox [Internet]. 2020. https://www.cqc.org.uk/sites/default/files/20200122_Digital-triage-in-health-services_sandbox-report.pdf. Accessed 25 Mar 2020.
Care Quality Commission. Regulatory sandbox round 3-community care at home teams [Internet]. 2020. https://www.cqc.org.uk/what-we-do/how-we-work-people/regulatory-sandbox-round-3-community-care-home-teams. Accessed 25 Mar 2020.
Care Quality Commission. Using machine learning in diagnostic services: a report with recommendations from CQC’s regulatory sandbox [Internet]. 2020. https://www.cqc.org.uk/sites/default/files/20200324 CQC sandbox report_machine learning in diagnostic services.pdf. Accessed 25 Mar 2020.
NHS England. NHS Test Beds programme [Internet]. 2018. https://www.england.nhs.uk/aac/what-we-do/how-can-the-aac-help-me/test-beds/. Accessed 25 Mar 2020.
Galea A, Hough E, Khan I. Test beds: the story so far. London: NHS England; 2017.
Diabetes My Way. GM Diabetes My Way. 2020; https://diabetesmyway.nhs.uk/about-us/. Accessed 25 Mar 2020.
South West London Health & Care Partnership. Diabetes: You & Type Two [Internet]. 2020. https://www.swlondon.nhs.uk/diabetes-you-type-2/. Accessed 25 Mar 2020.
BEATDiabetes NHS. Who we are [Internet]. 2020. https://www.beatdiabetes.org.uk/who-we-are/. Accessed 25 Mar 2020.
Ministry of Health Singapore. Licensing experimentation and adaptation programme (LEAP)—a MOH regulatory sandbox [Internet]. 2020. https://www.moh.gov.sg/home/our-healthcare-system/licensing-experimentation-and-adaptation-programme-(leap)---a-moh-regulatory-sandbox. Accessed 25 Mar 2020.
Prime Minister’s Office of Japan. Regulatory sandbox system [Internet]. 2020. https://www.kantei.go.jp/jp/singi/keizaisaisei/regulatorysandbox.html. Accessed 25 Mar 2020.
Information Commissioner’s Office. The guide to the sandbox (beta phase) [Internet]. 2019. https://ico.org.uk/for-organisations/the-guide-to-the-sandbox-beta-phase/. Accessed 25 Mar 2020.
Health Data Research UK. Home [Internet]. 2020. https://www.hdruk.ac.uk/. Accessed 25 Mar 2020.
Cornwall Trade and Investment. Kernow Health CIC supercharges Cornish healthtech innovation with new ‘sandbox’ offering [Internet]. 2019. https://www.cornwallti.com/2019/04/02/kernow-health-cic-supercharges-cornish-healthtech-innovation-with-new-sandbox-offering/. Accessed 25 Mar 2020.
Digital Jersey. Digital Health Sandbox [Internet]. 2020. https://www.digital.je/choose-jersey/sandbox-jersey/digital-health-sandbox/. Accessed 25 Mar 2020.
Mass Digital Health. Digital Health Sandbox Network [Internet]. 2020. https://www.massdigitalhealth.org/sandbox-network. Accessed 25 Mar 2020.
Mass Digital Health. The Digital Health Sandbox Program [Internet]. 2020. https://www.massdigitalhealth.org/mass-digital-health-programs/digital-health-sandbox-program. Accessed 25 Mar 2020.
Glicksberg BS, Oskotsky B, Giangreco N, Thangaraj PM, Rudrapatna V, Datta D, et al. ROMOP: a light-weight R package for interfacing with OMOP-formatted electronic health record data. JAMA Open. 2019;2:10–4.
Glicksberg BS, Oskotsky B, Thangaraj PM, Giangreco N, Badgeley MA, Johnson KW, et al. PatientExploreR: an extensible application for dynamic visualization of patient clinical history from electronic health records in the OMOP common data model. Bioinformatics. 2019;35:4515–8.
Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med. 2011;365:1758–9.
Energy Market Authority. Framework for a regulatory sandbox for the energy sector in singapore. 2019.
Federal Ministry for Economic Affairs and Energy of Germany. SINTEG - Smart energy showcases [Internet]. 2018. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/861078/regulator-approaches-facilitate-support-enable-innovation.pdf. Accessed 25 Mar 2020.
Financial Conduct Authority. Regulatory sandbox lessons learned report [Internet]. 2017. https://www.fca.org.uk/publication/research-and-data/regulatory-sandbox-lessons-learned-report.pdf. Accessed 25 Mar 2020.
Lim MD. Consortium sandbox: building and sharing resources. Sci Transl Med. 2014;6:242cm6.
Papadaki M, Hirsch G. Curing consortium fatigue. Sci Transl Med. 2013;5(200):200fs35.
National Institute for Health and Care Excellence. Evidence standards framework for digital health technologies [Internet]. 2020. https://www.nice.org.uk/about/what-we-do/our-programmes/evidence-standards-framework-for-digital-health-technologies. Accessed 25 Mar 2020.
Cooper S, Bouvy J, Baker L, Maignen F, Jonsson P, Clark P, et al. How should we assess the clinical and cost effectiveness of histology independent cancer drugs? BMJ. 2020;368:I6435.
Gwladys N, Maignen F, Kusel J. Experience and Impact of NICE scientific advice: 10 years on. Value Health. 2019;22:S802.
European Parliament. Overcoming innovation gaps in the EU-13 member states [Internet]. 2018. Available from: https://www.europarl.europa.eu/stoa/en/document/EPRS_STU%282018%29614537. Accessed 25 Mar 2020.
European Supervisory Authorities. FinTech: Regulatory sandboxes and innovation hubs [Internet]. 2019. https://www.europarl.europa.eu/stoa/en/document/EPRS_STU%282018%29614537. Accessed 25 Mar 2020.
Financial Conduct Authority. Call for input: cross-sector sandbox [Internet]. 2019. https://www.fca.org.uk/publications/calls-input/call-input-cross-sector-sandbox. Accessed 25 Mar 2020.
Joshi I, Morley J. Artificial Intelligence: how to get it right. Putting policy into practice for safe data-driven innovation in health and care. London: NHSX; 2019.
Gould M. Regulating AI in health and care [Internet]. 2020. https://www.nhsx.nhs.uk/blogs/regulating-ai-health-and-care/. Accessed 25 Mar 2020.
Department for Business Energy and Industrial Strategy. Department for Business, Energy and Industrial Strategy single departmental plan, June 2019 [Internet]. 2019. https://www.gov.uk/government/publications/department-for-business-energy-and-industrial-strategy-single-departmental-plan/department-for-business-energy-and-industrial-strategy-single-departmental-plan-june-2019. Accessed 25 Mar 2020.
Mackey T, Bekki H, Matsuzaki T, Mizushima H. Examining the potential of blockchain technology to meet the needs of 21st-century japanese health care: viewpoint on use cases and policy. JMIR. 2019;22:e13469.
Lugo-Palacios DG, Hammond J, Allen T, Darley S, McDonald R, Blakeman T, et al. The impact of a combinatiorial digital and organisational intervention on the management of long-term conditions in UK primary care: a non-randomised evaluation. BMC Health Serv Res. 2019;19:1–10.
Varey S, Hernández P, Tom MA, Mateus C, Wilkinson J, Dixon M, Milligan C. How effective and cost-effective are innovative combinatorial technologies and practices for supporting older people with long-term conditions to remain well in the community:? An evaluation protocol for an NHS Test Bed in North West England. BMJ Open. 2018;8:e017268.
Milligan C, Mateus C, Palmer T, Varey S, Hernandez H, Dixon M. Lancashire and Cumbria innovation alliance test bed: final evaluation report for a targeted supported self-care programme. Lancaster University; 2018.
Health Data Research UK. The Hubs [Internet]. 2020. https://www.hdruk.ac.uk/infrastructure/the-hubs/. Accessed 25 Mar 2020.
UK Health Data Research Alliance. About [Internet]. 2020. https://ukhealthdata.org/about/. Accessed 25 Mar 2020.
Health Data Research Innovation Gateway. About [Internet]. 2020. https://www.healthdatagateway.org/pages/about. Accessed 25 Mar 2020.
Digital Jersey. Soulgenic App [Internet]. 2020. https://www.digital.je/choose-jersey/success-stories/digital-health-sandbox/soulgenic/. Accessed 25 Mar 2020.
Digital Jersey. TSG - MediBooks [Internet]. 2020. Available from: https://www.digital.je/choose-jersey/success-stories/digital-health-sandbox/tsg-medibooks/. Accessed 25 Mar 2020.
eIT Health. The Digital Sandbox Accelerator [Internet]. 2020. https://eithealth.eu/project/digital-sandbox/. Accessed 25 Mar 2020.
eIT Health. Biobanks and Health Data [Internet]. 2020. https://eithealth.eu/project/biobanks/. Accessed 25 Mar 2020.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Funding
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant agreement No 82516
Conflicts of interest
All authors declare that they have no conflict of interest.
Availability of data and material
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
Code availability
Not Applicable.
Consent for publication
Not Applicable.
Authors' contributions
All authors contributed to the conception and design of the study, EL and DD acquired the data, interpreted the findings, and drafted the manuscript. JB and PJ contributed to interpretation of findings and revised the draft manuscript. All authors contributed to the final manuscript and approved its submission.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
About this article
Cite this article
Leckenby, E., Dawoud, D., Bouvy, J. et al. The Sandbox Approach and its Potential for Use in Health Technology Assessment: A Literature Review. Appl Health Econ Health Policy 19, 857–869 (2021). https://doi.org/10.1007/s40258-021-00665-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40258-021-00665-1