Enabled by the monitoring framework, as described above, the progress of the BDV PPP is continuously monitored. Below we report the key achievements and impacts in alignment with the development phases described in the SRIA that are backed by the monitoring data.
4.1 Achievement of the Goals of the PPP
According to the Big Data Value PPP SRIA v4,Footnote 14 the programme would develop the European data ecosystem in three distinct phases of development, each with a primary theme:
Phase I: Establish the ecosystem (governance, i-Space, education, enablers) and demonstrate the value of existing technology in high-impact sectors (Lighthouses, technical projects) (Work Programme WP 16–17)
Phase II: Pioneer disruptive new forms of Big Data Value solutions (Lighthouses, technical projects) in high-impact domains of importance for EU industry, addressing emerging challenges of the data economy (WP 18–19)
Phase III: Develop long-term ecosystem enablers to maximise sustainability for economic and societal benefit (WP 19–20)
The PPP goals achieved are analysed based on the defined roadmap. The year 2018 lies between Phase I and Phase II, and thus the progress of the PPP is assessed considering the objectives of both phases.
Phase I: Establish an Innovation Ecosystem (WP 2016–17) focused on laying the foundations needed to establish a sustainable European data innovation ecosystem (Table 1).
Phase II: Pioneer disruptive new forms of Big Data Value solutions (Lighthouses, technical projects) in high-impact domains of importance for EU industry, addressing emerging challenges of the data economy (WP 18–19). According to the SRIA, this second phase is meant to build on the foundations established in Phase I and will have a primary focus on Research and Innovation (R&I) activities to deliver the next generation of Big Data Value solutions. Although the projects implementing Phase II started in 2019 (or 2020), there are some activities in 2018 supporting the implementation of this stage, in particular those listed in Table 2.
Phase IIIFootnote 15: Develop long-term ecosystem enablers to maximise sustainability for economic and societal benefit (WP 19–20). This phase started in late 2019 and will continue until the end of the PPP. As this phase has only just started, the analysis can only be incomplete. Some ideas about possible achievements are provided in Table 3.
4.2 Progress Achieved on KPIs
4.2.1 Private Investments
Through this KPI, we attempt to understand and capture/show the level of industrial engagement within the BDV PPP. This KPI includes both direct and indirect leverage, as described in Fig. 2.
Two hundred and ninety-six companies representing all for-profit organisations participating in Big Data Value PPP projects active during 2018 (including not only project partners but also third parties engaged through cascade funding) and all for-profit organisation members of the BDVA were outreached to provide input to this KPI with an overall response rate of 40.9%.
Table 4 shows the evolution of the reported numbers in private investments from 2015 to 2018, as well as the EU contributions.
Aggregated to the numbers reported in 2015 (€280.9 million), 2016 (€338.5 million) and 2017 (€482.25 million), the amount of mobilised private investments since the launch of the PPP until the end of 2018 was 1569.1M€ (€1.57 billion). Considering the amount of EU funding allocated to the PPP by that time (€201.3 million), the BDV PPP ended 2018 with a leverage factor of 7.8, much higher than the leverage factor of 4 committed contractually.
4.2.2 Job Creation, New Skills and Job Profiles
Seventy-seven per cent of the BDV PPP projects indicated that their project would contribute to job creation by 2023, with an estimation in accumulated numbers of thousands. The estimated numbers surpass 7500 new jobs created by 2023 linked to project activities and many more considering indirect effect.
BDV PPP projects contribute to job creation in Europe by (1) increasing the market share of Big Data Technology providers in Europe; (2) developing new job profiles that generate new jobs… the creation; (3) developing new opportunities for entrepreneurs and start-ups in the new Data Economy; (4) generating job opportunities by increasing data sharing; (5) creating new jobs already during the lifetime of the project; and (6) forecasting jobs created as a follow-up of project results.
On the other hand, 40% of the BDVA members stated that their participation in the BDVA/BDV PPP had already contributed directly or indirectly to job creation, mainly because of the hiring of new experts to develop H2020 projects, start-ups created...), and new profiles hired to develop operations.
Projects reported that 48 job profiles were created or identified in 2018, and 106 new job profiles were reported as expected to be created from 2019 onwards and by the end of the project linked to the project activities.
Sixty-seven per cent of the projects running in 2018 reported contribution to the generation of new skills by the end of the project. In addition to the skills linked to the new job profiles, new skills are expected to be developed in cross-sectorial domains (e.g. in the form of “privacy-aware data processing” and “privacy-aware big data innovation” as reported by the SPECIAL project) and in specific sectors (e.g. analysis techniques using weather data, reported by the EW-SHOPP project). The BDV PPP incubators help start-ups to develop both the technical and non-technical skills needed to develop business in the Data Economy.
Among BDVA members, 51% of organisations reported contribution to the creation of new job profiles, and almost 60% contribute to the creation of new skills linked to the Big Data Value PPP. Finally, 60% of the projects and 51% of BDVA members have reported contributions to the Skills Agenda for Europe.
The BDV PPP organised 181 training activities involving over 18,300 participants during 2018. Projects contributed to this with 85 training activities during 2018 involving over 9700 participants. BDVA members reported 96 training activities involving over 8500 participants. Projects developed 16 interdisciplinary programmes during 2018 outreaching 250 participants.
During 2018, 396 equivalent FTEs masters and PhD students “(260 masters and 136 PhD) were involved in PPP projects, thereby collaborating with industrial players in developing industry-driven solutions and deploying experimentation testing scenarios. Contributing to raising awareness in professionals, users and the general public, the BDV PPP organised 323 events outreaching around 630,000 participants during 2018 contributing to raising awareness in professionals, users and the general public.
4.2.3 Impact of the BDV PPP on SMEs
Results of the Monitoring Report 2018 showed that a wide range of SMEs in Europe benefit from the Big Data Value PPP, considering the size (12% medium-size companies, 41% small companies and 48% micro-companiesFootnote 16), age (20% of the SMEs are 0 to 4 years old, 36% are 5 to 10 years old and 42% are 10 years old or older) and wide geographical distribution. SMEs play a variety of roles in the data value chain. SMEs participating in PPP projects clearly show a trend of an increase in turnover and in the number of employees. It is also important to mention that not all the SMEs involved in BDV PPP projects are technology companies but are also data users or providers, and the overall results and trend indicate an ongoing growth of turnover along the whole value chain.
Total turnover reported for SMEs in 2017 was €260.4 million.Footnote 17 In terms of turnover evolution, there is an increase in turnover in the SME companies that are part of the PPP with reported numbers of 60% increase in turnover with respect to 2014 and 17.7% in the last year. This number is in full alignment with the macro-economic numbers of data companies in Europe, and higher for some specific categories. In particular, young SMEs (5 and 10 years old) show on average the highest growth in turnover in relation to 2014 (up to 284%). The youngest companies (<5 years) show on average the highest growth in the last year (54.8%).
In terms of employment evolution, the trend is also very positive in all companies that are part of the PPP, with an average increase in employment for the SMEs that are part of the PPP of 75% with respect to 2014 and a growth of 11.83% in the last year (2018 compared with 2017).
Special emphasis should be given to PPP instruments focused on supporting SMEs, in particular the Data Incubators and i-Spaces. The average age of the companies receiving cascade funding from the Data Incubators (DataPitch and EDI) is 4.9 years; 41% of those SMEs are younger than 5 years, 50% are between 5 and 10 years, and only 9% are older than 10 years old. Companies reported an increase in turnover of 315% for 2014 and 48.8% for 2017, and an 118.5% increase in employment for 2014 with a 22.4% increase in the last year.
4.2.4 Innovations Emerging from Projects
Innovations arising from the BDV PPP include:
Specific project developments that have a marketable value, including Big Data products, processes, instruments, methods, systems and technologies, offering value to a wide variety of economic and industrial sectors
Services of high societal value developed by projects
Spin-offs arising from projects and start-ups incubated by the programme activities
Patents and solutions enabling advanced privacy- and security-respecting solutions for data access, processing and analysis
Contribution to Standards (individually as projects and coordinated activities at a programme level)
Innovations resulting from cooperation between projects or programme-coordinated activities (e.g. advances in data sharing, innovative skill programmes, reuse of technical solutions across different sectors, etc.)
Transformation of sectors of high economic value (led by the PPP Lighthouse projects, but also triggered by project cooperation): new business models and scaling innovations (advances in TRLs (technology readiness levels), cross-border solutions and bringing technology closer to the market, accelerating adoption)
In its second year of operation, the BDV PPP’s 32 running projects reported 106 innovations of exploitable value as delivered in 2018: 63% have a medium impact and 37% are considered innovations of significant impact. Fifty per cent of the innovations delivered in 2018 are incremental innovations, 6% are architectural, 36% are disruptive and 1% are radical innovations.Footnote 18
Ninety-three per cent of the innovations delivered in 2018 have an economic impact, and 48% have a societal impact.Footnote 19 Forty-one per cent are technologies (including platforms), 32% are services, 7% are products, 8% are methods, 8% are systems, 1% are software, 4% are components and/or modules and 11% are others, including frameworks/architectures, processes, tools and toolkits, spin-offs, datasets, ontologies, patents and knowledge.
Sixteen per cent of the innovations delivered in 2018 are fully cross-sectorial.
Sevety-five per cent provide solutions to the transport, mobility and logistics sector (the one with the best coverage in the PPP by the end of 2018); 20% of the innovations related to public services and smart cities; 19% to industry and manufacturing; 14% to bio-economy; 13% to the Telco sector; 12% marketing activities; 8% relate to health and healthcare; 8% to the ICT market; 7% to geospatial market; 5% to commerce; and 3% to others (including fashion, retail, business services, energy, media, compliance, etc.).
In relation to the maturity levels and TRLs, 7% of the innovations delivered are TRL 3 (experimental proof of concept), 10% are TRL 4 (technology validated in lab), 36% are TRL 5 or TRL 6 (technology validated in relevant environment, industrially relevant environment in the case of key enabling technologies), 32% are TRL 7 (system prototype demonstration in operational environment), 8% are TRL 8 (system complete and qualified) and 1% are TRL 9 (actual system proven in an operational environment—competitive manufacturing in the case of key enabling technologies—or in space).
Figure 3 provides a full overview of the innovations delivered by the BDV PPP during 2018, combining level of significance, type of innovation (incremental, disruptive, architectural or radical) and the TRLs. Although a large number of innovations are classified as incremental innovations of medium impact, it is remarkable to note the high percentage of significant innovations (and expected growth in the upcoming years), the high number of disruptive innovations and the high TRLs in some cases close to deployment. Although at a lower level, the BDV PPP is also delivering some architectural and radical innovations.
Sixty-three new economically viable services of high societal value were developed during 2018 as a result of the projects. Forty-seven per cent (over 30 projects) contributed to this KPI.
Projects reported 204 new systems and technologies developed during 2018. Many of them are already reported as part of the KPI “Significant Innovations to Market”. Systems and technologies developed are not limited to one sector, and, in fact, the majority of the new systems and technologies can be utilised in different sectors/markets, thus stimulating the use of Big Data technologies in many areas.
Finally, many solutions and innovations arising from the Big Data PPP have been promoted in the BDV PPP MarketplaceFootnote 20 developed by the BDVe CSA project to spread knowledge about the outcomes of the PPP.
4.2.5 Supporting Major Sectors and Major Domains with Big Data Technologies and Applications
The BDV PPP Lighthouse projectsFootnote 21 active in 2018 focused on the bio-economy (agriculture, fisheries and forestry) (DataBio project), transport, mobility and logistics (transforming transport project), health and healthcare (BigMedilytics project) and manufacturing (BOOST4.0), with a total of four major sectors supported by Lighthouse projects and therefore widely supported by multiple use cases, scenarios and solutions.
Twenty per cent of the projects are fully cross-sectorial (their outcomes can be used in any sector or application domain) and 80% of the projects are working in more than one sector or application domain (this explains why the total is higher than 100% in Table 5). In particular, the BDV PPP projects address a wide variety of sectorsFootnote 22, as shown in Table 5.
Others (43% of the projects) includes sectors such as insurance, public safety, personal security, public tenders, e-commerce, marketing, fashion industry, citizen engagement, ICT/Cloud services, social networks, procurement and legal domain.
Considering the whole project portfolio, the number of sectors supported is higher than 15, with a solid distribution of use cases, experiments, solutions and outreach activities among different sectors.
Projects reported 224 use cases and/or experiments conducted during 2018 with contributions from 18 different projects. This is an increase of 48.3% with respect to 2017 (151 experiments). The BDVA i-Spaces reported an additional 165 experiments with 6 i-Spaces contributing to this KPI.
Projects reported 82 large-scale experiments developed during 2018, 64 involving closed (private) data (78% of the total). Large-scale experiments either involve a large number of users with high TRLs or are developed in large geographical areas, in many cases involving a large number of users and actors or a combination of data volume, complexity and velocity; a large number of data sources; or integrated complex datasets flowing across borders. The BDVA i-Spaces also contributed to this KPI, reporting in total 38 large-scale experiments performed during 2018, 28 of them involving private data.
In relation to the amount of data made available for experimentation, reported information from projects and i-Spaces (members of BDVA) shows that the amount of data made available by the BDV PPP for experimentation in 2018 is 0.10696 Exabytes (106.96 Petabytes). A total of 0.08625 Exabytes (86.25 Petabytes) was reported by the projects.Footnote 23 It is important to note that some of the projects are not only providing internal access to diverse data sets from different sources but are also improving and creating new valuable datasets (e.g. of DataBio project). BDVA i-Spaces contributed to this KPI, reporting an additional 20.71 Petabytes of data for experimentation.
4.2.7 SRIA Implementation and Update
Concerning SRIA coverage, measured as “% of research priorities covered compared to the overall scope of research priorities defined in SRIA”, projects have delivered contributions during 2018 already covering 100% of all the SRIA technical priorities. The major focus of technical contributions was “Data Analytics”, followed at some distance by “Data Processing Architectures” and “Data Management”. This is a significant change from the 2017 coverage, where “Data Management” was the top priority. A clear trend to focus on technical contributions in the areas of “Data Analytics” and “Data Processing Architectures” was anticipated in the BDV PPP Annual Monitoring Report 2017,Footnote 24 thus supporting our explanation that a solid base of “Data Management” solutions will enable analytics and processing innovations.
In relation to the BDV SRIA update, at the end of 2017 the BDVA released the BDV PPP SRIA v4.0 (detailed process and results reported in the 2017 Monitoring Report). This version was the basis to support the H2020 LEIT ICT WP2018–20. During 2018 a minor update, towards a version 4.1, was released in the community, crystallising in a series of individual deliverables in the format of vision, position or discussion papers that supported the transition towards the next framework programme and the creation of a new strategic agenda and roadmap.
In total, there were at least 12 events organised during 2018 that contributed to input in the BDVA strategic papers – multiple online meetings with a total of 2085 participants/contributions.
In total, since the launch of the BDV PPP, we can count 6422 potential contributions to the strategic roadmapping activities.
4.2.8 Technical Projects
The BDV PPP contributes to enabling advanced privacy- and security-respecting solutions for data access, processing and analysis. For 2018, 97 contributions were reported (2 patents,Footnote 25 61 publications and 24 OSS/SW/Products).
Fifty per cent of the projects confirmed that they are assessing quality, diversity and value of data assets. These results show the intense usage of metrics to measure quality, diversity and value of data assets in projects, and some projects have developed specific metrics and methods to ensure quality, diversity and value in the data (e.g. I-BiDaaS has developed a Data Quality Assurance Process (DQAP) aiming at ensuring the high quality of the data generated/collected during the lifetime of the project). However, we cannot talk yet (2018) about the “PPP”-developed metric expected for 2019+.
Concerning the speed of data throughput, 40% of the projects reported that they expect the project to improve data throughput. Some projects, such as BigDataOcean and FashionBrain, measured improvements over 1000%. Others such as I-BiDaaS have specific objectives to develop data processing tools and techniques applicable in real-world settings and to demonstrate a significant increase in speed of data throughput and access.
4.2.9 Macro-economic KPIs
The monitoring of macro-economic KPIs is based on input from the European Data Market Monitoring ToolFootnote 26 as they are presented in the most recent report by IDC (https://www.idc.com/).Footnote 27
Development of the market share of the European Union in the global Big Data Market. As an indicator, we compare the total revenues of EU Data Companies with other economies, i.e. the US, Japan and Brazil, as they are used as a benchmark in the IDC report.Footnote 28 The EU share of the total revenues in these economies the 2013 baseline was 27.7%. This share increased slightly to 27.9% in 2018, which is remarkable because the international indicators grew very fast in this period, but the EU kept pace with them. In absolute terms, the total revenue of US data companies in 2018 was approximately twice that of EU28 data companies in the same year (€162 billion vs. €77 billion). Seventy per cent of PPP projects active in 2018Footnote 29 reported contribution to increasing the revenue share of EU companies. Projects contributed by:
Accelerating adoption of new technologies
Supporting EU data-driven companies to build innovative solutions that can be scaled internationally
Developing innovative technologies to make European companies more competitive (e.g. news data protection approaches)
Enabling industries to exploit their big data efficiently and therefore increase their market share and services provided to their customers
According to the most recent report,Footnote 30 the number of data companies increased to 283,100 by 2018, compared to 271,700 in 2017, with a growth rate of 4.2%. It should be noted that almost half of them are based in the UK, due to the high concentration of the ICT industry there. BDVA i-Spaces and Data Incubators (ICT 14-b projects, i.e. DataPitchand EDI projects) are in particular designed to contribute to this KPI as they support start-ups and entrepreneurs from early ideas to technical and business development until the go-to-market stage.Footnote 31 Seventy-seven per cent of the BDV PPP projects active in 2018Footnote 32 reported contribution towards increasing the number of European companies offering data technology and applications. The projects contributed in different ways, such as:
Creating tools that will stimulate the creation of new companies
Creating new companies as a result of a project (e.g. BigDataOcean)
Supporting EU data-driven companies
Building innovative solutions to solve data-related challenges
Supporting companies in complying with the GDPR
Lowering the threshold to create new business in a particular sector
In addition, 25% of BDVA members reported that their organisation ran or supported a programme that is specifically targeted at supporting start-ups or entrepreneurs in the field of Big Data.
The revenue of data companies in the European Union, according to the IDC report,Footnote 33 reached €77 billion in 2018 compared to €69 billion the previous year, with a growth rate of 12%. The revenue share of SMEs in 2018 amounts to €55.5 billion (72% of the total revenue), an absolute growth of €5.7 billion on the year before. The growth rate of revenue increases in proportion to company size, with the revenue of large companies with over 500 employees growing at 16% in 2018 over 2017. Seventy-seven per cent of the PPP projects active in 2018Footnote 34 reported contribution (or plan to contribute) to the revenue generated by European data companies. Project contribution to this KPI is mainly by:
Opening up sectors to data-intensive companies
Offering direct support and getting funding for data start-ups
Making data processing easier and cheaper for companies
Creating new opportunities through privacy-preserving analytics solutions
Commercialising new services with a marketable value
Creating opportunities for common exploitation based on joint Big Data technology pipelines
Developing simplicity in some business ecosystems
The baseline for data professionals in the European Union in 2013 amounts to 5.77 million. The number of data professionals increased to a total of 7.2 million by 2018, resulting in an absolute growth rate of 1453 million professionals since 2013. The rate of growth of data professionals is increasing, with approximately 559,000 positions added in 2018 and an increase of 8.4% on the year before.Footnote 35 Eighty-seven per cent of the PPP projects active in 2018Footnote 36 reported contribution from their project to increase the number of data workers in Europe. Projects contribute to this KPI in different ways:
New organisations created as a result of the projects hiring new data professionals
Supporting growth of emerging start-ups
Developing more data-driven services that will require new data workers
Unlocking the value of data services by introducing privacy-preserving technologies
Creating new job profiles
Supporting the adoption of data solutions in different sectors
Supporting education and training
4.2.10 Contributions to Environmental Challenges
Over 20% of the projects running in 2018 reported that they contribute to the reduction of energy, and 30% contribute to reduction in CO2 emission. Quantitative results are provided by some projects, such as the Transforming Transport (TT) project that shows that in some specific monitored items improvements in efficiency range between 25% and 51% in energy reduction, and improvements concerning CO2 emissions reach up to 29% and emission reductions in general (including PM and NOx) up to 23%.
The three Lighthouse projects running in 2018 (DataBio, Transforming Transport and Boost4.0) have reported contribution to reduction in waste. For example, in DataBio and in particular in forestry, although still with early data and experiments, the experience from customer cases shows a reduction in waste of up to 10%. Some pilot TT projects show approximately 25% improvement in the management of assets, which can adequately demonstrate a relative high-level achievement in waste reduction at this final stage of the project.
Seventeen per cent of the projects running in 2018 have reported contribution to reduction in the use of material resources; e.g. BigMedilytics provides quantitative data in a particular scenario, reporting that the Asset Management pilot aims to reduce the number of unused mobile assets in hospitals by up to 20%.
Finally, in relation to energy reduction in big data analytics, there is no quantitative input in results provided by any project but, e.g., the E2Data project develops a framework that optimises calculations, leading to decreased use of energy.
4.2.11 Standardisation Activities with European Standardisation Bodies
During 2017, the BDVA and the BDV PPP set up some foundations defining priorities for the PPP in Big Data standardisation implemented during 2018 as follows:
Establish an official liaison between the BDVA Standards Group and the AIOTI WG3; this activity was developed through different workshops during 2017 and implemented in 2018 with the signing of an MoU with AIOTI and common activities organised during the year.
Further develop the BDVA Reference Model pursuing alignment with others, such as oneM2M, BDE Platform, AIOTI and RAMI 4.0, implemented through different workshops organised during 2017 and 2018.
Open an official dialogue with CEN, CENELEC and ETSI on standards harmonisation, implemented through different workshops during 2017 and 2018. The BDVA intends to sign an MoU with CEN/CENELEC in 2019, and it is under discussion with ETSI.
Create the BDVA Roadmap for Big Data standards harmonisation and industry engagement in Global Big Data standards development.
Thirty per cent of the projects running in 2018 reported that they perform activities leading to data/Big data standardisation. Three projects reported contribution to European standardisation bodies (ESBs) activities and reported 11 working items in ESBs. Twenty per cent of BDVA members reported that their organisations perform activities leading to data/Big data standardisation. In particular, BDVA members have reported contributions to IEC, DIN DKE and other consensus-based standardisation bodies; OPC foundation and other consortia-based standardisation bodies; OASIS; W3C committees and community group discussions; open data harmonisation national activities; ISO/IEC JTC1; and defining standards in georeferenced data for geoscience (Open Geospatial Consortium (OGC) and Commission for the Management and Application of Geoscience Information (IUGS/CGI)) and ETSI.