1 Introduction

One of the key aspects of service-oriented computing is the capability to compose various software components to offer on-demand services and applications. This key aspect is even more important in the current computing landscape where new IoT, network functions [34] and cloud resources are emerging on daily basis, enabling us to build new services and applications for solving various problems. With today’s tools and frameworks, it is quite easy to expose capabilities of IoT, networks and cloud resources through well-defined service interfaces for on-demand needs with pay-per-use models, and the concept of resources is no longer limited to computing power with CPU and memory. We have seen different types of resources, such as data, analytics, firewall, intrusion detection, messaging, gateways, virtual machines, to name just a few, to be explicitly specified and composed into services and applications and to be controlled on-demand [28, 34, 67].

We have seen many advances in composition and provisioning services and IoT, network function virtualization (NFV) and cloud computing [21, 30, 38, 47, 50, 62]. However, our interest in this paper is not about the discussion of silo works in individual IoT, network or cloud systems. We want to examine IoT, network functions and clouds as a whole, with specific application contexts. This leads to the key question of our work: how can we blend resources from IoT, cloud (micro) data centers and edge/core networks to build virtual continuum-based resource slices across various layers and systems? It is an important academic question [71], as well as crucial in industryFootnote 1, but we lack theoretical foundations. In our view, this needs strong support from capabilities of tools and frameworks for services engineering, provisioning, operation and analytics. This is, however, not easy to achieve due to the complexity and diversity of IoT [24], distributed clouds [16, 54], and Fog/Edge computing [9, 14]. While in these fields the use of service computing is extensive—almost all resources are provided by web services in IoT, network functions, and clouds—our software components need to deal with a lot of data generated by Things [64]. We know that distributed analytics are carried out in different places [25, 58], software interactions are not just based on REST [59], and the service response is almost near real time.

Realizing the impact of IoT and Fog/Edge computing, several calls have been made for services models and corresponding techniques for them [11, 24, 75]. While many works have started to produce and deploy services in the edge, there exist also many open research and challenging issues [18, 29]. Conceptual, generic integration models between IoT and Cloud have been discussed and surveyed intensively [10, 20]. Our goal in this editorial paper is not to discuss many aspects of the integration between IoT, network functions and clouds. Instead, we focus on building ensembles of IoT, network functions and clouds for SOCA. We consider this research direction important because—with the availability of various services for IoT, network functions, and clouds—we can compose, provision and operate such ensembles of resources across the edge and data centers for various application domains. However, not only we need to understand the potentials, we also have to analyze the challenges that one will have to face and what we need to do.

In this paper, we will focus on the key question of ensemble models, and the state-of-the art in terms of service composition, runtime aspects and analytics. We provide motivations for the concept of ensembles before focusing on state-of-the art. Since the work on ensembles will involve complex, cross-layer and cross-system activities, we will concentrate on certain important aspects related to SOCA, such as service models, composition, testing, and interoperability. We believe that, in this way, our editorial will provide a broader view onto existing problems and present guidelines for future research.

The rest of this paper is organized as follows: Sect. 2 discusses emerging ensembles for IoT, network functions and cloud services, and key issues in composition and management of ensembles. We analyze state-of-the-art in Sect. 3 before presenting key suggestions in Sect. 4. We conclude the paper in Sect. 5.

Fig. 1
figure 1

High-level abstract view of ensembles

Full size image

2 Emerging ensembles of IoT, network functions and clouds

2.1 Ensemble model

The first question we need to clarify is what does it mean by “ensembles” of IoT, network function and cloud resources? We (re)emphasize that our notion of resources can be used to indicate, e.g., data, network, analytics, and messaging. Thus resources can be mapped to virtual machines, containers, network firewall services, complex processing services, data streams of Things, to name just a few. Using this notion, in our work, ensembles consist of resources from IoT, network functions, and cloud services that establish a set of resources utilized for the application in a specific context (with clearly time, performance, cost, etc.). Resources in an embemble work together, clearly establishing the so-called resource slice [71]. At runtime, ensembles might be “intelligent” (or not) by being able to reconfigure and adapt themselves. In our work, we do not really emphasize the intelligence aspect of ensembles like in work in the artificial intelligence domain, but are concerned more on the question of resource management and analytics of ensembles.

Figure 1 presents the very high-level view of ensembles of IoT, network function and cloud resources. Such ensembles can be built by using service techniques because we have observed that not only current cloud resources provide excellent service models but also:

  • NFV/5G is a key emerging area and more providers deploy services at the edge. In fact, many features of NFV are built based on elastic cloud models and cloud technologies

  • IoT infrastructure-as-a-service and pay-per-use IoT communication are on the rise.

  • IoT data-as-a-service and edge analytics as a service follow the existing cloud service models, provided by public and private IoT providers

Given these trends, eventually providers of IoT and network functions will present programmable APIs for us to utilize their resources, enabling the combination of such resources with well-utilized cloud resources for application-specific contexts.

The second question is whether we really need such ensembles, as the integration of IoT, network functions and clouds can be continuously done with other means. To answer this we need to distinguish between generic integration models, such as discussed in [10, 20], with specific, concrete models for integration in our vision. Let us consider various application scenarios that demonstrate the need to combine various resources (data, network functions, cloud data services, etc.) for the application explicitly:

  • Geosport/sport analytics: consider a sport analytics application we might analyze data of athletes in a short time, e.g., 2 hours of games. In such analytics, we need IoT systems to bring data to the cloud services but we also need network functions to guarantee the network traffics and to prevent data from being sent to wrong destinations. Concrete scenarios can be found in the H2020 U-Test projectFootnote 2 but as well as in various real use cases which could be extracted from those described in [12].

  • Video analytics in smart cities: consider the case that we have many public and private cameras in the city that are exposed through Web services. For a particular event, e.g., football events, various clients want to obtain camera video as well as cloud and network, such as police and hospital for a few hours of data analytics. Huge amount of resources from IoT, network functions and clouds are needed in different contexts.

  • Emergency responses in seaports: consider the case of an accident in a seaport, e.g., in the Inter-IoT projectFootnote 3. We need to gather IoT data from terminals, vessels, cranes, containers, etc., in order to make the right responses. Such IoT data will need to be analyzed by services, which might be in the cloud and in the edge system within the seaport. The resources and their configurations are very specific for this response which might last a few hours only.

Here, it is important to see that typically many applications have utilized resources from IoT, network functions, and clouds but not all of them need to explicitly acquire and manage all resources. However, in the aforementioned scenarios, applications need ensembles with specific constraints. All of these applications lead to the question of how to establish ensembles of resources that we need to manage, control and optimize at IoT, edge and cloud sides, not just data flows from IoT to clouds [70].

2.2 Key research problems

It has been argued that the service technique is the right approach for modeling and implementation of IoT-based systems [42]: for example, the sensing and actuation capabilities of devices are exposed through microservices [51]. Then, higher level services are composed from the basic microservices for various data processing and decision-making functions that are closer to the applications. This naturally fits well to the extensive use of cloud resources as services, leading to a natural way of integrating IoT and cloud for SOCA.

However, the network between the cloud and IoT has also been provided with service capabilities. This leads to the question is how to build services through the composition of IoT, network functions and cloud services. What would be the direction that we should follow? In our view, we should take a fresh look at how IoT, network functions and clouds can be exposed as services that eventually bring on-demand resources from different layers from different systems to be available as programmable objects for application-specific needs. This means that we need to develop techniques for composing such resources and provisioning them for applications, without worrying about the low-level particular details of specific layers or systems.

3 State-of-the-art

3.1 IoT, network functions and clouds as a service

It is needless to discuss about cloud as a service as it is very matured in today’s computing landscape. In cloud computing, the developer and end-user can easily access cloud services and utilize them. However, in the core network and/or the network between the edge and the cloud, network functionsFootnote 4 are not really exposed to the developer for composition and/or customization for applications. Typically, network functions are controlled and managed by providers as underlying infrastructures to enable more dynamic service provisioning for rapidly meeting service-level agreements. Certain network functions associated with cloud infrastructures can be programmed, such as virtual network connections and firewalls. However, they are not the network functions in the edge or between the edge and the cloud; they are in cloud data centers.

For ensembles, we do not expect that all Fog/Edge network systems will be opened for application developers. However, we expect that the Fog/Edge system providers will expose certain network and computing capabilities under the services when the providers claim their infrastructure as-a-service. In this view, similar to the current cloud computing offerings where the developer can easily program private network connection or firewall settings, network functions can be requested, provisioned and controlled by the applications. For instance, one can deploy a firewall (a completely software service) to the edge to prevent sending of sensor data to cloud instances outside Europe.

To this end, network functions as a service [57] technologies could enable us to compose and orchestrate network services as part of ensembles. In many situations, the use of containers to establish overlay networks, such as with WeaveFootnote 5, could also be a solution for deploying network functions for ensembles. Other approaches on utilizing microservices and service function chaining to establish the link from IoT to the cloud like [45] are interesting. However, they are still quite low-level technologies that need to be exposed to high-level services for composing ensembles.

On the IoT side, IoT is also provided as a service [55]. Mostly, this means that the IoT network can be controlled and managed through the service model in the sense of dedicated IoT services. However, advanced providers can also offer IoT features similar to existing public clouds: IoT services can be rented and paid per use for different customersFootnote 6. Another aspect to leverage IoT as a service is to offer APIs for exploiting IoT. The issue of API management for IoT has been raised [74], but from the perspective of IoT providers to make sure that they can hide the complexity of the access to their resources.

All of the above-mentioned models enable us to compose, acquire and operate IoT, network functions and cloud resources based on our special needs. Thus they enable the concept of ensembles of IoT, network functions and cloud that we advocate in this editorial note.

3.2 Modeling IoT, network functions and cloud elements

To be able to provision IoT and network functions together with cloud services, one needs to be able to capture their information and capabilities. We have seen several information models for network functions, such as [73], NetJSON (http://netjson.org/rfc.html) and YANG (http://www.netconfcentral.org/yang_docs). However, these information models have not been linked to IoT resources.

For modeling the relationship between things, sensors, and cloud services, many works have been introduced [7, 27, 37, 53, 76]. Such efforts produce information models that can be part of ensembles and thus can be leveraged for abstracting capabilities of IoT services and network services. However, they do not address the modeling of ensembles in our view.

One approach to modeling IoT, network functions and clouds is to use Model-driven Engineering (MDE). Although IoT is an emerging topic for MDE [46], we have seen several papers introducing different techniques for modeling IoT and Cyber-physical Systems (CPS) [15, 66]. In cloud engineering, there already exist many MDE tools for cloud services [13, 22]. However, these MDE tools are focused on silo systems (either IoT or Cloud). They are not developed for ensembles and they do not include network functions as modeling resources.

3.3 Programming and composition models for ensembles

In terms of services selection and composition, various works about service selections can be used but their focus is not on IoT services, especially for data-aware service selections and mashups [31]. Most of the work have no connection to the network services, reflecting network functions and back-end clouds. Searching and finding IoT services is an active research area but it focuses mainly on finding devices and services (semantic service selection) [33]. In [40] a declarative language is presented for creating ensembles of autonomic components. This work too does not focus on IoT and networks. In [75] orchestration challenges for IoT have been discussed. Key issues of optimization have not been outlined. Recently, there has been work discussing about the intelligent distribution between the edge and the cloud [32]. However, they do not introduce concepts to manage and programming resources for specific IoT services across the edge and the cloud. In terms of composition algorithms, network-aware service composition [30] presents an integrated QoS-aware composition method that integrates application services and network services together. The work in [62] presents algorithms for IoT service composition that consider quality of service (QoS) and network latency. Such works however have never been considered in the context of IoT services, network services and clouds together. In [39] user-centric IoT application composition is shown. This, however, does not include networks.

3.4 Resource management and adaptation

In cloud computing, sophisticated features to make cloud services autonomous (or, self-*) are based on MAPE-K [36]. Examples of cloud service adaptation are described in [48, 80]. As surveyed in [52] there are a huge number of IoT middleware. But most of them do not manage network services between IoT and clouds, thus they do not adapt network functions at runtime. There are a few works to coordinate the resources between IoT and Cloud [17] but not the networks in between. Works for virtualizing the network functions, such as OPNFV ArnoFootnote 7, allow us to operate and maintain complex network systems. Although arising for different purposes, the combination of NFV and SDN presents powerful solutions for network control and reconfiguration of complex distributed systems. However, for ensembles, the adaptation of such complex networks must be carried out in sync with the IoT and cloud counterparts. To date, we do not have frameworks to do the adaptation across IoT, network functions and clouds.

In earlier work [6, 43, 44] we proposed the notion of an autonomic middleware for IoT-based systems. The key aspect of this middleware is the use of MAPE-K concepts to facilitate context-aware adaptation of IoT service compositions. This also necessitates the automatic generation of these compositions, especially in two cases: (1) possible large latency of computation required to generate the compositions, and (2) the composition generation depends upon specific formalisms to specify precondition/effects and the planning algorithms. We also present the concept of a multi-layered context model—application layer, environment layer, device layer—so as to facilitate contextual adaptation. We envision the application of this idea for automated generation and adaptation of ensembles.

3.5 Monitoring and policy management

Given the resources across IoT, network functions and clouds, a key question is how do we monitor and control the execution policies across subsystems. In terms of monitoring, there is no lack of monitoring tools to capture various information from different layers, e.g., industrial tools such as PrometheusFootnote 8, TelegrafFootnote 9, LogstashFootnote 10, FluentdFootnote 11 and academic works [1, 3]. Tools such as [63] would provide valuable information. However, the main problem is that these tools do not give an end-to-end view on the ensembles. They provide various metrics but leave the metrics correlation and cross system analysis to the developer.

Another aspect is from the provider viewpoint: how would the provider enforce resources provisioned for application-specific ensembles? Generally, execution policy enforcement should be carried out across IoT, network functions and clouds. The policy execution at the IoT side is different from that for clouds and networks, such as, based on software-defined machines profiles and policies [56] and blockchain-based permission control [61]. Another tricky problem is that if a solution provider develops an ensemble and deploys the ensemble as a service for his/her customers, how does this solution provider work with underlying IoT, network functions and cloud providers w.r.t. execution policy? This problem still remains unclear.

3.6 Testing

There are many testing techniques for cloud services. DevOps and testing tools, such as automatic testing with JenkinsFootnote 12, GatingFootnote 13, and TsungFootnote 14, and academic works have demonstrated powerful applications to test cloud services. Essentially, such testing techniques can also be extended to network services and IoT, as long as they offer well-defined interfaces. However, ensembles are established at runtime, hence it is not sure how such tools can be used.

Testing IoT is not a very well-researched topic. The work in [60] analyzes various challenges in testing IoT. Some authors also try to apply traditional testing methods for IoT, like fault injection [5]. All of these works are valuable but they are far from the requirement for testing ensembles. The key point is that, from the ensembles perspective, we see IoT and network functions as services. Thus, in order to test them we must have a mechanism, similar to cloud testing, but for IoT and network services, to dynamically test performance and failure. Furthermore, the test has to be within the context of ensembles, imposed by specific constraints with respect to performance and costs, for example.

Another important aspect of testing for ensembles is to test their uncertainty. For complex IoT, network functions and clouds, it is easy to see the lack of knowledge about resources, as well as the lack of knowledge about the change of underlying systems. This lack of knowledge brings uncertainty that requires novel techniques to monitor and test [2, 4, 49, 77, 78]. While work like in H2020 U-test have presented taxonomy and uncertainty profiles and tools [72], the impact of uncertainty is actually not well studied for IoT, network functions and clouds.

Overall, the work of testing in cloud is quite well developed. Similarly, testing performance of network layers has been done intensively. The current work on testing IoT is very primitive and we do not see any testing tools for ensembles. One possibility is to extend certain types of IoT Cloud testing, for example, to run a lot of IoT data to test its impact on cloud services. But this testing is only for the two ends and it is not clear how we could do it for the network functions.

3.7 Interoperability

In the cloud, interoperability has been addressed extensively [79]. Interoperability ranges from standardization [65] to tools for specific solutions. In IoT, a published special issue [23] has introduced various interoperability problems and tools. Furthermore, there are many other solutions for interoperability, such as middleware-based [8], MDE-based [26], and protocol translation-based [19].

If we consider IoT, cloud or network functions individually, we see that each of them has to deal with various interoperability problems, such as mismatched interfaces, incompatible protocols, semantic and syntax data problems. The key interoperability problem for ensembles is that the ensembles are established across various layers and systems. Therefore, we need to deal with a lot of individual interoperability solutions and combine them together for ensembles. Here, the composition and provisioning techniques of SOCA could be important for selecting the right interoperable solutions based on meta-data and runtime data.

4 Recommendations

From our high-level analysis of the state-of-the art, we make the following five recommendations:

Rec #1—resource modeling and interoperability For us it is quite clear about the service models for IoT, network functions and clouds, especially microservices and REST-based implementations. We have seen that mostly service interfaces are provided through REST. However, especially at the IoT side, various IoT services offer different ways to access data and controls, e.g., using messaging protocols like MQTT and AMQP. The nature of interaction models for data acquisition and control is not in one-to-one interaction like typical enterprise services but is in one-to-many or many-to-one. Therefore, we recommend the service interaction models to be developed with one-to-many or many-to-one protocols to suit IoT, while still maintaining the key principle of services, such as composability and discovery.

We recommend IoT resources be modeled with many new types of meta-data about data quality, execution policies, etc., to enable the composition of interoperable services. On the one hand, we need to continue the current approach on implementing interoperability solutions by introducing middleware, protocols translation, etc. On the other hand, the dynamic approach for software composition and provisioning would also be another way.

Rec #2—programming We recommend to follow the current two approaches: the top-down approach with model-driven engineering (MDE) and the bottom-up approach with programming frameworks and languages. In MDE, we have observed various tools and MDE profiles for IoT and Cloud. However, frameworks for supporting ensembles of IoT, network functions and clouds are missing. There are also work in business processes [41]. One of our previous works [69] for example allows us to model IoT Cloud systems. However, it has not included network functions yet and it is still focused on modeling without code generation. Here our suggestion is to combine various MDE approaches to develop a unified framework for ensembles. The approach from programming frameworks and languages introduce various techniques. In particular, the complex service compositions mainly are described by workflows or data processing pipelines in which clouds are used for complex processing and IoT are reflected through data inputs. However, we have not seen the incorporation of network functions into such languages. Thus we suggest to incorporate network function APIs into existing languages to allow us to build ensembles.

Rec #3—service management and adaptation Adaptation needs to be context-aware [6] and driven by stored knowledge about the system and earlier adaptation implementations. However, current adaptation techniques and implementations are only for specific systems. We recommend to exploit end-to-end context-aware adaptation [43, 44] based on issues triggered from the IoT side as the ensembles are mainly for addressing problems from the IoT side with also resources in the edge and cloud. Considering a huge number of uncertainty across layers and systems as well as the vast knowledge that one needs to master to work with ensembles, we recommend to develop uncertainty measure for ensembles. This can be started from existing work on uncertainty for IoT/CPS [72]. Furthermore, uncertainty must be included into ensemble composition and adaptation. Works like uncertainty adaptation in IoT Cloud [49] could be a starting point.

Rec #4—end-to-end monitoring Given a large number of tools for infrastructure monitoring and application monitoring as well as their related services for managing monitoring data, we recommend a focus on correlation metrics and end-to-end monitoring, as they are missing at the moment. One approach is to establish end-to-end monitoring by leveraging existing tools, such Prometheus and Fluentd, by combining with end-to-end techniques, such as from [35]. However, new metrics characterizing ensembles must be defined, for example, how to calculate the availability of an ensemble based on the availability of IoT, network functions and cloud.

Rec #5—testing We suggest to focus on end-to-end testing through integration and system testing. We should see ensembles as a uniform unit to test flows from IoT to clouds via network functions. In this way, we can combine different testing techniques for IoT, network functions and cloud but we need to (i) define combination/composition of tests, (ii) testing utilities for different systems, (iii) test emerging properties like uncertainty, elasticity, actuation. One approach we suggest is to combine IoT testing with elastic testing of cloud services [68].

5 Conclusions

In this paper, we motivated the building of application-specific ensembles of IoT, network functions and clouds, given the current offering and technology trends in IoT and Fog/Edge computing. We presented the concept of ensembles of IoT, network functions and cloud resources. Based on our preliminary works and discussions, we believe that such ensembles are important because many applications need such ensembles for their specific context with respect to performance, time and costs that they can manage and control. We have discussed the current state-of-the art, various important aspects as well as presented recommendations for the composition and analytics of ensembles.

Our current work is to deal with theoretical concepts of ensembles, such as the formal way to describe and represent ensembles. We are working intensively into the tools to support this concept, especially, monitoring, uncertainty analytics and programming and execution management for ensembles. Certain samples of services and ensemble structures are being updated at https://github.com/rdsea/IoTCloudSamples.