The SODALITE Model-Driven Approach

Abstract

The specification of deployment topologies for complex applications distributed across multiple heterogeneous infrastructures is a difficult process that encompasses multiple modeling tasks, engaging several actors, including application ops experts, resource experts on the specification of the target infrastructure resources, quality experts on the application optimization, and application administrators on the deployment governance. SODALITE proposes a novel infrastructure as a code (IaC) modeling framework that provides a model driven engineering approach for the authoring of application- and infrastructure-level specifications, realizing an instantiation of an infrastructure as a code (IaC) modeling framework. This chapter introduces the SODALITE IDE and the IaC services. The IDE enables SODALITE expert roles to model (conforming to the SODALITE DSMLs) and generate IaC artefacts facilitating the app deployment. Experts are assisted in the modeling phase by the semantic knowledge inference and validation capabilities of a Knowledge Base (KB), which is populated with IaC descriptions for resources semi-automatically discovered from target heterogeneous infrastructures. The IDE leverages the SODALITE IaC services for automatic target image preparation and IaC artifacts generation upon deployment.

3.1 Introduction

Cloud computing and infrastructure virtualization are one of the well-known leading technologies that companies are increasingly embracing.

The issues that virtualisation and expansion of infrastructure as a service based approach introduced are the scalability, inventory management, complex networking management, security policies, etc. Large scale virtual infrastructure systems are difficult to control and manage and therefore usually need high level code snippets, scripts or other advanced software artifacts to manage resources, services, deployments, upgrades, etc. The concepts of handling the code produced to manage and automate infrastructure provisioning and manage systems can be defined as Infrastructure as Code (IaC) management. The usage of IaC in the DevOps toolchain enables:

• Cost effectiveness as automation efforts reduce simple and repetitive tasks;

• Speed and efficiency as DevOps teams have tools for releasing infrastructure updates and services much faster than in manual configuration scenario;

• Immutable infrastructure as changes are applied by rebuilding resources instead of modifying the existing resources;

• Possibility of applying traceability, validation and testing to help reducing the number of errors, mitigating risks, and leading to robust setup for built in security.

However, the adoption of IaC implies also an increased complexity at it requires a deep knowledge of the target infrastructures and of multiple IaC scripting languages to support the initial application deployment and application lifeycle management, including upgrades, re-installations, reconfiguration, re-adaptation etc.

Many cloud resource management standards have been proposed for addressing those interoperability and portability issues, such as TOSCA,¹ CIMI,² and OCCI.³ However, those standards still cause interoperability problems, since the modeling languages and the semantics differ among standards, resulting in non-reusable resources. Thus, there is a high need for modeling the application components and the resources in a standard machine-readable format.

To this end, Semantic Web technologies can promote interoperability, re-usability, and intelligent decision support to various cloud-based systems. In SODALITE, we have developed a rich conceptual meta-model that is based on the best practices on ontology engineering in order to create a formal abstraction of the application and the resources by representing the functional and non-functional requirements and interlink them with other dependencies, and relationships. This meta-model aims to create a formal abstraction of the application and of its resources by representing the functional and non-functional requirements and the dependencies among components.

However, the ontological model is not preferred as a front-end language, as it is not user-friendly. Thus, a Domain Specific Language (DSL) has been designed to provide a programming-oriented way to describe these models by offering a lightweight language abstraction that hides the complexity of the ontologies. Accordingly, our SODALITE Integrated Development Environment (IDE) enables the application developer to create DSL-compliant models for her application and infrastructure resources. Then, the following processes for container-image preparation and IaC scripts generation result in the creation of a valid TOSCA CSAR blueprint.

The rest of this chapter is organized as follows. Section 3.2 presents the SODALITE Domain Specific Languages and the corresponding editors that are part of the IDE. Section 3.3 presents how the knowledge is represented in terms of ontologies. Section 3.4 shows the smart features offered by the IDE thanks to the ontological inference. Section 3.5 describes the procedure how a user can prepare container images of the applications. Section 3.6 presents the final steps toward the preparation of proper and ready to be executed Infrastructure as Code (IaC). Finally, Sect. 3.7 concludes the chapter.

3.2 The Modeling Approach and the IDE

The modeling of the deployment topology of complex applications across heterogeneous infrastructures engages a multidisciplinary team, consisting of several roles, namely Application Ops Experts (AOEs), Resource Experts (REs) and Quality Experts (QEs), introduced in Sect. 2.2.1, who are involved in the specification of different deployment concerns. The SODALITE IDE supports these roles, through different editors, to address their modeling needs. Each editor is specialized in the creation of a kind of models, which are compliant to one of the SODALITE DSLs, that describe a concrete aspect relevant for specification of the application deployment topology. The SODALITE DSLs are described in the following paragraphs.

3.2.1 Abstract Application Deployment Model (AADM)

AOEs tackle the modeling of an application topology as a model instance of the AADM DSL. An AADM is a topology, that is, a connected graph of application components that declares the relationships among them and the requirements to the infrastructure resources they need. They can be resolved, either at design or deployment time, through the concrete selection of the suitable resources from a target infrastructure. As the ultimate purpose of this AADM DSL is to simplify the modeling of the application deployment topology, and its conversion into a TOSCA [12] blueprint by the IaC layer (see Sect. 3.6), this DSL borrows modeling concepts for TOSCA topologies, such as node templates and policy definitions. Moreover, leveraging TOSCA in SODALITE DSLs largely simplifies their adoption among end-users already familiarized with this de-facto Cloud deployment language. Listing 3.1 shows an excerpt of an AADM for the Snow UC application. In there a snow-weather-condition-filter component is declared as an instantiation of the abstract type DockerizedComponent. It overrides some of the properties inherited from that type, in order to further specialize the component. Moreover, it declares an explicit dependency on another component, snow-mysql, and expresses some requirements such us the registry and network to be used, as well as the infrastructure resource where to host the component. Finally, it makes use of a specific optimization that is described in the next paragraph. Despite these requirements have been specified at design time in this example, the abstract nature of the AADM permits the resolution of the mandatory requirements in the topology at deployment time, assisted by the inference and reasoning capabilities of the SODALITE ontology (see Sect. 3.4).

3.2.2 Optimisation Model (OM)

QEs tackle the design of deployment optimizations as instances of the Optimization Model (OM) DSL (see Sect. 4.4.2). Such instances can be bound to components declared in AADMs as in Listing 3.1. Listing 3.2 shows an example of optimization model for components that use the TensorFlow AI training library. At a top level, QEs build the optimisation model by specifying whether to enable optimal build for the target (enable_opt_build) and auto-tuning (enable_autotuning) and the application type (app_type). Then, QEs add section-specific options that the optimiser can use to select an optimised container and provide target-specific runtime options.

In the example, opt_build node specifies the node architecture of the target and autotuning specifies the tuner tool and its input. The application-specific options are listed in the app_type-ai_training node, with sub-node config listing the application configuration and data listing the data settings. The ai_framework-tensorflow shows the AI framework-specific options for optimisation. Based on the optimisation model, the application optimiser maps the optimal application parameters and deployment settings to that of the target hardware and selects an optimised container.

Optimization models are shareable and can be linked to any application component node declared in an AADM topology by setting the optimization parameter. As an example, the snow-weather-condition-filter component node, described in Listing 3.1, can be associated with the optimisation model (for a TensorFlow AI trainng application) in Listing 3.2. When a node in the AADM is associated to an optimization model, the semantic reasoner validates the association by cross checking entries like the application type and target architecture. At deployment time, the AADM and the associated optimisation model are passed on to the application optimiser that actuates the definition.

3.2.3 Resource Model (RM)

REs tackle the modeling of infrastructure resources as model instances of the Resource Model (RM) DSL. This DSL defines classes for describing new resource types, their capabilities, requirements, relationships and interfaces. Alike AADM DSL, RM DSL borrows concepts from TOSCA for the specification of resource types, their relationships, associated policies and other aspects. Listing 3.3 shows an example of new resource type, the ConfigurationDemo, a specialization of a SoftwareComponent. The RE declares a new property, mysql-db-pass and expresses the requirement remote-server restricted to VM resources that offer a Compute capability, being this requirement bound through the DependsOn relationship. Moreover, the RE specialises the create operation of the Standard interface, by declaring new inputs, namely the remote-server and the mysql-db-pass, and by providing an Ansible implementation for this operation. This can be done either directly writing or reusing an Ansible playbook or by exploiting the Ansible DSL and associated editor as described in Sect. 3.2.4.

AADM, OM and RM are interlinked. The AADM imports both the OM and the RM so that application components in an AADM model can refer to optimization models and can be instantiated as component instances of the types defined within RM models. This split of metamodels for the different modeling concerns permits these three roles to focus only on one of those concerns by using a single metamodel.

3.2.4 Ansible Model (AM)

As mentioned in the previous section, the Resource Expert may decide to implement operations defined in a Resource Model as Ansible Models that are then translated into executable Ansible playbooks. The IDE, in fact, implements an Ansible DSL and offers to the users suggestion and verification facilities that allow them to create Ansible Models in an easy way.

For each specified operation within a Resource model, the user can select the option to generate the abstract Ansible model and, then, the corresponding Ansible script for further development. As a result, as shown in Fig. 3.1, a folder is created for each defined node type, together with a subfolder with the name of each of its TOSCA interfaces. Finally, for each TOSCA operation, a .ans file, containing the abstract Ansible model is created. After this model is completed, the user can then generate the corresponding .yaml file, containing the Ansible concrete code.

Fig. 3.1

Ansible Model folder structure

During the Ansible Model creation task, the user is supported in all steps. For instance, the IDE suggests the names of the operations to be defined, based on the RM definition and highlights syntax errors. Moreover, it gives to the user the possibility to use as variables in the Ansible Model the inputs that have been defined in the Resource Model (see Fig. 3.2).

Fig. 3.2

Ansible Model editing support

An interesting feature that enables reuse of preexisting code, is the possibility for the user to import and use in an Ansible Model the modules available within the Ansible Galaxy repository.⁴ The IDE, in fact, first, enables the user in browsing through the Galaxy modules. Then, it provides content assistance for each Ansible module parameter, emphasizing the required ones. It also informs the user about inserting values for each parameter by displaying the value type that each parameter expects and presenting the acceptable values and the official description that helps the user understand its purpose. Moreover, the Ansible editor’s content assistance offers suggestions for standalone roles available on Ansible Galaxy and Ansible roles included in Ansible collections.

The validation mechanisms are a significant addition to the Ansible editor because they allow the end-user to identify mistakes in the Ansible model directly and avoid the repetitive execution of the Ansible script before fixing all the defined errors.

3.2.5 Monitoring Alerting Rule Model

AOEs can also describe the rules that determine the triggering of alerts when the monitoring of their deployed applications reveal some anomalous situations. When the rule condition applies, the associated alert is dispatched by the SODALITE Monitoring (see Sect. 5.4) and captured by Refactoring (see Sect. 5.5), which eventually computes corrective actions on the affected deployed application. Listing 3.4 shows examples of alerting rules for Snow UC. These alerting rules are based on Prometheus PromQL⁵ language. Each alerting rule model consists of one or more rules, organised in groups. Each rule consists of an expression, formalized in PromQL, which describes the condition, expressed as a boolean-evaluated expression, that has to be held during a given time duration (expressed in the for attribute) to trigger the alert. The expression consists of a combination of monitoring metrics, processed by functions, aggregation functions and filters. The severity label attribute specifies the severity associated with the triggered alert, and it is interpreted by the endpoint that captures it, that is, the Refactoring.

3.2.6 The IDE

SODALITE DSLs modeling is supported by the SODALITE IDE,⁶ which is based on Eclipse.⁷ The IDE offers textual and graphical editors for creating AADM models, and textual editors for OM and RM and Ansible models. The DSL grammars, parsers, serializers and textual editors have been developed by using XText.⁸ The graphical viewers and editors, and the form-based property views have been implemented by using Sirius.⁹ The synchronicity between textual and graphical editors has been set up by using the XText-Sirius Integration.¹⁰ Textual editors for AADM, OM, RMs and Ansible models are intended for skilled modelers who require fast and high modeling productivity. These editors provide context-aware intelligent content-assistance to guide modelers through the syntax and content of the SODALITE DSLs, suggesting possible elements to be inserted into the model at the point of typing (see Sect. 3.4). A graphical editor (see Fig. 3.3) and its accompanying form-based editors for AADM are also available, and support similar intelligent content-assistance. They are intended for modelers who prefer the visual modeling based on a canvas where to drag and drop entities selected from a palette, but also for textual modelers that appreciate a visual representation of the AADM model. They are synchronized with the textual editor so that changes in the textual model are immediately reflected (upon saving) in the graphical and the form-based property views, and vice versa. The AADM visual representation resembles the one adopted in TOSCA documentation (which is not a standardized one).

Fig. 3.3

Excerpt of Snow AADM graphical view

The IDE only requires from AOEs and RMs the provision of mandatory information they own about their application topology and/or required infrastructure resources, and relies on the Knowledge Base (KB) to complete the model either at the design or deployment time. The IDE also exploits the reasoning engine to assist modelers during the authoring of AADM or RM models, by suggesting them suitable choices to fulfil certain model entities, including the overriding of properties inherited from super types, the resolution of mandatory requirements, the proper selection of component types, etc. (see Sect. 3.4). AADM and RM models are stored into the SODALITE KB to be reused and shared with other modelers.

The IDE textual editors conduct syntactic validation (i.e. to ensure DSL compliance) during the modeling phase. Semantic validation is conducted by the reasoning engine and by the optimization sub-system. All detected errors and warnings, as well as possible fixes, are presented to the user in the IDE and associated to the affected component.

Fig. 3.4

IDE KB View

Fig. 3.5

Deployment Wizard

Fig. 3.6

Deployment Governance View

Besides supporting DSL modeling, the IDE offers an user interface for most of the SODALITE user-driven processes, namely:

• Model Management in KB. AADMs and RMs can be saved (wizard assisted) into a remote shareable Knowledge Base for further reuse by other users. In case of AADMS, versioning is available to distinguish between different versions of the same application deployment. Moreover, the IDE offers a KB Browser View to browse all those available models in the Knowledge Base for which the user has read permissions. Selected models can be retrieved into the user’s local Eclipse workspace for further edition, or deleted (if user’s has write permissions) when not needed (see Fig. 3.4).

• AADM Deployment. AoEs can select AADMs in the IDE project explorer view for deployment. A popup wizard guides the user through the deployment process, requesting the AADM inputs, as well as the orchestrator configuration (see Sect. 5.2), such as the number of workers for parallel deployment (see Fig. 3.5). Optionally, AoEs can request the wizard to complete the AADM by resolving unfulfilled component requirements, leveraging the Knowledge Base inference capabilities (Complete AADM check box in wizard). The deployment process continues in the IDE background notifying the user about its status.

• Deployment Governance. AoEs can browse all their owned deployments in the IDE deployment governance view (see Fig. 3.6), which shows them grouped by application (i.e. blueprint in the orchestrator terminology). Details of each deployment are shown, including external links to associated monitoring dashboards, which are accessible in Web browsers. Failed deployments can be resumed for initial state or from the first failing node. Selected deployment and empty blueprints can be removed when unneeded.

• Image building. Images for VM creation, required by applications to be deployed, can be created by the image builder and registered into the image registry from the IDE by providing a image file descriptor (see Sect. 3.5).

• Resource discovery. Target infrastructure resources can be automatically discovered and their associated RMs created by the Platform Discovery Service (see Sect. 2.2.1) and stored into the Knowledge Base. This process can be triggered from the IDE for selected infrastructure types.

3.3 Knowledge Representation and Ontologies

Semantic Web technologies, and particularly, ontologies represent the domain knowledge in a formal and abstracted manner that fosters advanced reasoning. Therefore, in SODALITE, ontologies were adopted for capturing the TOSCA meta-model that contain complex semantics such as subsumption hierarchies and multi-role concepts. Through the use of ontologies, SODALITE mainly aims to (i) follow a common, extensible, and formal standardised model in order to describe cloud-related concepts (ii) capture both the structural and semantic relationships in an unambiguous manner by managing the information in the form of Knowledge Graphs (iii) create an interpretation and validation layer, for example, for inferring validation errors and smells [2] by following the semantics of TOSCA [3], and by reusing existing rule languages and logic-based frameworks.

3.3.1 Background in Ontologies

An ontology is a formal, explicit specification of a shared conceptualisation [10]. The world “ontology” is used with different meanings in different communities [14]. Namely, both the philosophy and computational field share in common the attempt to represent the knowledge formally as a set of concepts along with the relations among them. They have become such popular mainly due to the promise of the semantic interoperability and common understanding among different parties [5]. Their expressivity and level of formalisation depend on the knowledge representation language used. The Semantic Web is an extension of the current Web by annotating the resources with meta-information for establishing a common representation across heterogeneous sources. Ontologies play a key role within the Semantic Web by enabling the weaved knowledge to be interpreted into a machine-understandable format. In pursuit of that objective, the Web Ontology Language (OWL) is a key Semantic Web Ontology Language that is part of the W3C standard and was designed to represent complex knowledge.

The OWL language has been heavily influenced by the Description Logics. Description Logics (DLs) is a popular knowledge representation formalism. A DL knowledge base consists of two different types of statements, an ABox (assertional knowledge) and a TBox (terminological knowledge) [4]. The TBox describes the structure of the data stating general properties of concepts and roles, in other words the ground truth, while the ABox contains the instances of the concepts defined in the TBox. For instance, the TBox axiom Compute $\sqsubseteq$ Root asserts that all objects that belong to the concept Compute, are members of the concept Root too. The Abox are the real world entities, the instances of the TBox classes. For example, Compute(vm) and hasProperty(vm,name) express that vm has a property, which is described by the name instance. For representing the data of the Semantic Web, a set of web-based knowledge representation languages has been developed. Some of those languages are: RDF (Resource Description Format), RDF(S) (RDF Schema), and OWL [1].

• RDF: The RDF is a language for describing resources on the Web, was originally released as a W3C recommendation in 1999, and updated in 2004 and in 2014. The RDF data model is based on graphs, as opposed to tuples that underlie relational data models. In RDF, a data graph is constructed by the union of a number of three part assertions, called triples. A triple is composed of three parts: a subject, a predicate, and an object.

• RDF(S): RDF provides the basis for the Semantic Web, but it is limited in expressing the definition of the resources and their relations. RDF(S) is a standard that released along with the second generation of RDF in 2004 (and updated in 2014) and based on RDF. It defines classes and properties extending the base RDF vocabulary and provides support for more expressive knowledge modelling semantics. Using the RDF(S) vocabulary, it is possible to model lightweight ontologies, consisting of concepts, relations and their hierarchies.

• OWL: The OWL was developed simultaneously with RDFS to provide more high-level expressiveness. It is a knowledge representation language widely used within the Semantic Web community for creating ontologies. OWL has three expressive sub-languages: OWL Lite, OWL DL, and OWL Full. OWL Full is the most expressive, but due to the high degree of expressiveness, it is undecidable. As a result, OWL DL is primarily used as a more expressive dialect. However, due to the tree model property [16], OWL can model only domains where objects are connected in a tree-like manner. This constraint is quite restrictive for many real-world application, therefore, the W3C working group produced the OWL2 [9]. OWL2 is a revised extension of OWL, commonly referred as OWL1. Some of the most prominent extensions are constructs for specifying cardinality, and value restrictions, and complex property inclusion axioms (property chains). OWL2 is divided into three different profiles, OWL2 EL, OWL2 QL, and OWL2 RL.

The SODALITE capitalizes on the most expressive features of OWL2:

• Meta-modeling: It is the practice of treating individuals as classes. One feature of meta-modeling is that it must be possible to assign properties to classes in the model. This way it is possible to assert the membership of classes in meta-classes and interconnect them via meta-roles [8]. The direct semantics of OWL2 do not allow for meta-modeling, but OWL2-DL supports it by the use of punning [11]. Punning allows for using the same identifier e.g. for an individual and a class. The main motivations for meta-modelling are that a model often needs to play more than one role in an application, reusability is promoted, and complex situations can be modeled more effortlessly.

• Ontology Design Patterns: An Ontology Design Pattern (ODP) models a recurrent ontology design problem. ODPs can be viewed as small, modular and reusable, and templates based on these patterns or other regularities in the ontology [15]. They can be also viewed as as a way of bottom-up pattern finding that is then reused across the ontology and offerred a a ’best practices’ design solution. For achieving better degree of knowledge sharing, reuse and interoperability, SODALITE’s conceptual model reuses the DnS ontology [6] pattern of DOLCE+DnS Ultralite (DUL) ontology [7].

3.3.2 SODALITE Conceptual Model

In SODALITE we propose an ontology-based framework [13] for capturing and interlinking TOSCA-based descriptions of cloud applications and resources. In this section, it will be described the conceptual model and the modeling decisions for implementing the ontology-based semantic abstraction layer of SODALITE.

The SODALITE semantic models include the SODALITE meta-model and the domain ontology: (i) The SODALITE meta-model is the formal ontology pattern that is used in all the different levels of abstraction (ii) The domain ontology, which is Tier 0, contains the TOSCA normative types and provides the vocabulary that will be used in the other two modelling layers (tiers), namely Tier 1 (resources) and Tier 2 (pattern instantiations).

3.3.2.1 Tiers

SODALITE follows a modular, 3-tier approach to capture the knowledge. In all the tiers, the SODALITE ODP is used in order to foster a unified representation paradigm for enabling a unified and complete model that promotes interoperability, extensibility and smooth knowledge management. The tiers are described in detail as follows (Fig. 3.7):

• Tier 0: This is the static tier of the model. It contains the TOSCA vocabulary, namely the TOSCA meta-model in the conceptual SODALITE ODP. For example, all the TOSCA built-in types are modelled in this tier with all their relationships and associations.

• Tier 1: This tier involves all the custom resource types created by Resource Experts. Those custom types extend the TOSCA built-in types.

• Tier 2: These are the instances composing the Abstract Application Deployment Model (AADM), named also as “templates”, and are reusable combinations of Tier 0 and Tier 1 types. Application Ops Experts create the AADM using the DSL in the SODALITE IDE (see Sect. 3.2).

Fig. 3.7

Overview of the SODALITE modelling layer

3.3.3 Descriptions and Situations Pattern (DnS)

For better degree of knowledge sharing and reuse, the SODALITE ODP is a specialised instantiation of the Descriptions and Situations (DnS) ontology pattern that is part of DOLCE+DnS Ultralite (DUL). The Descriptive Ontology for Linguistic and Cognitive Engineering (DOLCE) focuses on cognitive issues and aims at capturing ontological categories underlying natural language and human common sense. The DOLCE + DnS Ultralite is a light version which has been simplified and improved. In SODALITE, we adopted the DnS design pattern which is part of the aforementioned light version of DOLCE ontology. The purpose of DOLCE + DnS Ultralite ontology is to provide the basis for easier interoperability among middle and lower level ontologies.

Fig. 3.8

Core DnS pattern in DuL

The DnS pattern is presented in Fig. 3.8. This pattern captures the notion of “Situation” out of the state of affairs, with their interpretation being provided by a “Description”.

• Situation: a set of domain entities that are involved in a specific pattern instantiation

• Description: represents the descriptive context of a situation that defines the concepts that classify the domain entities of a specific pattern instantiation. In such a way, views are created on the situtations.

• Concepts and parameters: Classify domain entities describing how they are interpreted in a particular situation. Each concept might have one or more parameters for describing additional descriptive context.

3.3.4 SODALITE Meta-Model

The SODALITE meta-model, which is depicted in Fig. 3.9, extends the core DnS pattern by specializing the core DUL concepts in order to model TOSCA definitions.

• SodaliteSituation: It represents a situation such as a node template or type (SodaliteSituation $\sqsubseteq$ dul:Situation).

• SodaliteDescription: Each situation is related with (soda:hasContext) a description (SodaliteDescription $\sqsubseteq$ dul:Description) that contains the attributes properties, interfaces, requirements etc.

• SodaliteConcept and SodaliteParameters: Each description has a specification (soda:specification $\sqsubseteq$ dul:defines) that involves one or more zero concepts such as attributes, properties, requirements e.t.c) and each Concept has one or more parameters (soda:hasParameter $\sqsubseteq$ dul:hasParameter) for describing more nested content. Each SodaliteConcept classifies one SodaliteEntity (soda:SodaliteEntity $\sqsubseteq$ the dul:Entity).

Fig. 3.9

SODALITE meta-model (extension of DUL)

3.3.5 Example: Tier 1, Tier 2

In Tier 1, it is noteworthy the OWL2 meta-modelling capabilities of the resources, as they are both classes and instances. For example, in Fig. 3.11, it is presented that the sodalite.nodes.DockerizedComponent is a class (rdfs:subClassOf tosca.nodes.SoftwareComponent), and also an instance, as it participates in a property assertion soda:hasContext. By using the meta-modelling (punning), we promote reuse as we can have subsumption hierarchies since the node types can be classes, and also have descriptive context (e.g. properties, capabilities) by the role of instance. Henceforth, the templates, which are instances of the types, inherit this descriptive knowledge of the types leading to more comprehensive and reusable knowledge components. Especially, in this domain where the knowledge is captured in different levels, namely normative types, resources and applications, the adoption of the meta-modelling enables this knowledge representation. Some examples of the Tier 1 and Tier 2 follow:

• Tier 1: In Fig. 3.10, the SODALITE ODP capabilities are demonstrated through an instantiation of a pattern that captures the definition of a node type. The depicted node type is custom and inherits a normative TOSCA node type, the tosca.nodes.SoftwareComponent.

  Since the purpose is to show the SODALITE ODP, only an excerpt of the node type is depicted. It is presented the correspondence between the TOSCA elements and the ODP concepts. Namely, each node type is a Situation, that contains a Description containing various concepts. The same rationale is followed in the static layer, Tier 0.

  Figure 3.11 depicts an excerpt from the Knowledge Graph of the aforementioned node type. More specifically, the requirement and the attribute are captured as instances of the tosca:Requirement and tosca:Attribute correspondingly. Each concept classifies the property that is modelled, for example tosca:host. Any other nested information is captured through the definitions of the SodaliteParameters. Each concept, such as ex:SodaliteParameter_2 in our example, can have a description by using the dcterms:description property of Dublin Core.¹¹

• Tier 2: In Fig. 3.12, an excerpt of the snow-weather-condition-filter node template’s knowledge graph is depicted, and its full version is shown in Listing 3.1.

  

  The ODP is used similarly with the Tier 1. Precisely, each template is captured as a situation that has a description. Each description contains concepts such as properties, attributes, requirements etc. Each concept classifies a property and represents the nested knowledge as parameters. Listing 3.5 depicts the definition of the snow-weather-condition-filter as an instance of the sodalite.nodes.DockerizedComponent resource.

Fig. 3.10

Example TOSCA custom node type and high-level assignment of SODALITE ODP concepts

Fig. 3.11

Example node type

Fig. 3.12

Example node template ontology

3.4 A Smart Environment for Developing IaC

The smartness in the SODALITE Modelling layer primarily arises from the semantic inference capabilities of the KB. This task is performed by the Semantic Reasoner. In this section we focus on the main features it offers.

3.4.1 Context-Aware Content Assistance

Smart context-aware content assistance is offered to the user by means of suggestions during the authoring of the models. This modelling assistance is mostly based on the inference and reasoning capabilities of the KB. Since the models are saved as interconnected RDF knowledge graphs, the information is represented as a network of relationships that capture both the structural and semantic relationships in an definite manner. Thereby, the KB can be easily reasoned for supporting discovery, reuse, and validation. In respect of the validation of the models, in the Sect. 4.2, we describe the intelligent services checking the validity of the AADM during the design phase. Regarding the discovery and reuse, some of the cases that the KB suggests to the modelers are:

• Available infrastructure resources that satisfy the requirements of the application components. Thus, the requirements of an application component can be completed more easily, as the KB can suggest suitable resources that can serve as a host, dependency, network and other requirements. An example SPARQL query, that is used for discovering the aforementioned resources, is depicted in Listing 3.6, which retrieves the type (value) that is defined in a specific requirement (requirementName) of a resource definition. Then templates of this specific type are discovered in order to satisfy requirements of an application component.

  

• Concepts. Concepts such as the interfaces, properties, and capabilities. They can be assigned to the infrastructure and application components depending from which type they are derived or instantiated. In Listing 3.7, a SPARQL query that discovers properties for an application component is depicted.

  

• Infrastructure and application components that belong to a specific group. For example dockerized application components, or components of a specific use case e.g. snow. In Listing 3.8, a SPARQL query returning all the templates that are saved in the workspace of the snow use case. By leveraging how the data set is split in a Knowledge Base, the models can be grouped in graphs enabling the reusability as the users can discover models and resources among them.

  

Fig. 3.13

Content assistance for properties

Fig. 3.14

Content assistance for requirements

The IDE relies on the KB Reasoner to assist DSL modelers with context specific suggestions at the point of edition. Upon request, the IDE displays a drop-down list of suggestions obtained from the KB Reasoner after sending a query that include the edition context. The user can browse the list, select one option, which will be inserted at the cursor point. In Fig. 3.13 IDE offers the properties defined in the hierarchy of type sodalite.nodes.DockerizedComponent, whose defaults can be overridden for the instance snow-weather-condition-filter. In Fig. 3.14 IDE offers available infrastructure resource instances (defined locally within the AADM or available in the KB) that can fulfill the host requirement of the snow-weather-condition-filter component.

3.4.2 Abstraction

Since the creation of the deployment model requires the acquaintance with various IaC languages, their execution mechanisms, and with different heterogeneous infrastructures, the abstraction of the SODALITE DSL is uppermost for lessening the modeller’s effort. In pursuit of making the DSL as abstract as possible, we apply deep inference reasoning on the interlinked knowledge that is saved in KB so as to concretize the AADM. Precisely, with the abstracted DSL, some information in the deployment model can be omitted by the modeller.

Fig. 3.15

Abstraction DSL example

The abstraction primarily focuses on the requirements of an application component. The user can totally omit the requirements where, for instance, it is defined where an application can be hosted, which network is needed, which resources are dependencies such as database.In Fig. 3.15, an excerpt of the snow-weather-condition-filter is depicted, where it is shown that all the requirements can be totally omitted by the modeller, and the KB can autofill the model.

3.5 Preparation of Container Images

Unlike the usage of Hypervisors for provisioning and virtualizing underlying hardware through virtual machines, container engines virtualize the operating system (such as Linux or Windows) having each container holding only the application and its libraries and the needed dependencies. Containers can leverage the features and resources of the host OS making them small, fast, and portable. Being lightweight containers improve CPU and memory utilization of physical machines. Containers also facilitate building of granular applications making them an ideal choice for service oriented architecture approach such as microservices architectures, making components independently deployable. At the same time containers provide the DevOps teams the level of flexibility and portability making them a great choice for running applications on heterogeneous environments such as multi-cloud, HPC or Edge.

Having the possibility to create deployable images and the tools for building these application images within SODALITE, a user can leverage transportability of application deployments between different systems and architectures. SODALITE uses the Image Builder component to pre-build application images for targeting an OS virtualizer such as Docker.¹² The Image Builder component itself is a dockerized REST API encapsulation of the xOpera lightweight orchestrator and a TOSCA/Ansible blueprint that is executed by the orchestrator. It can be configured to run different image building workflows enabling the user to build the application from source or tar images and push the created image to a Docker registry. Figure 3.16 shows the architecture of the Image Builder component. The image building workflows for building runtime images are running prior to deployment of the TOSCA blueprint, before the orchestrator starts with the execution of the blueprint deployment e.g., provisioning the infrastructure and deployment of the application. The building process can be automated through REST API calls or run manually from SODALITE Smart IDE (Sect. 3.2). In this case the user is able to check periodically for the status of the image building process. Since the image building process can take some time the REST API provides an asynchronous implementation of the image building. Listing 3.9 shows an example of a JSON build parameters for the Snow WebCamCrawler component.

The encapsulation of the xOpera lightweight orchestrator and TOSCA blueprints into the REST API enables the image building functionality to be accessible from any component in SODALITE framework or be just reused as a separate blueprint if needed. The extendable nature of these TOSCA blueprints provides a high level of reusability of the code for supporting the image building process. Image Builder implements the process of building the images through TOSCA v1.3 blueprint. Image builder also supports session handling and authentication/authorization by JWT tokens making it easy to integrate with Identity and Access Management providers.

Fig. 3.16

Image Builder Architecture

An important innovative feature has been implemented giving the user the possibility to create multiple image variants in a single image building workflow run. Image builder exposes its functionalities through both multifunctional CLI and REST API, and can be also used in a CI/CD scenario. The image is regularly pushed to the public DockerHub registry under sodaliteh2020/image-builder-api.¹³ The source code and extensive information on how to build, use and deploy the Image builder is provided in the Image Builder GitHub repository.¹⁴

3.6 Infrastructure as Code Preparation

The SODALITE runtime layer relies on the declarative OASIS TOSCA standard v1.3¹⁵ to receive the definition of infrastructural resources needed for a certain application, their relationships, and the mapping of application components onto these infrastructural elements. TOSCA is implementation agnostic, meaning that it does not suggest a recipe for the implementation of the node operation lifecycle, but, instead, specifies the interfaces associated to these lifecycle operations and can rely for their implementation on practically any high or low level language. SODALITE has chosen Ansible as a high level configuration management declarative language for the implementation of lifecycle operations for having maximum impact on DevOps teams that already use configuration management tools.

The goal of IaC preparation is the one of generating correct TOSCA code from AADMs written in the SODALITE DSL.

All SODALITE models, including abstract application deployment models (AADMs) and resource models (RMs), are stored in the semantic Knowledge Base (KB), as explained in Sect. 3.3. All the interactions to the KB are implemented through the Semantic Reasoner API, which handles the export of the models into a JSON model definition (AADM JSON). The AADM JSON model definition is self-sufficient and provides all the node definitions, optimization models (Sect. 3.2.2), application deployment topology definitions, as well as references to Ansible playbooks needed for the configuration of the application deployment and the creation of a valid self-contained TOSCA v1.3 blueprint in the Cloud Service Archive (CSAR) format, defined in the OASIS TOSCA standard. A CSAR contains definitions of all TOSCA node, relationship, artifacts types and templates, needed to deploy an application.

IaC Blueprint Builder is the component that takes the AADM JSON model definition and creates a valid CSAR, deployable through the SODALITE Orchestrator (xOpera) via its REST API endpoint. The IaC Blueprint Builder is implemented in Python and encapsulates three distinct subcomponents:

• Swagger REST API exposes and implements a REST API that forwards the calls for parsing of AADM JSON and creation of the TOSCA CSAR;

• Abstract Model Parser internally parses an AADM JSON and builds a TOSCA blueprint representation as a tree data structure,

• IaC Blueprint Builder creates a TOSCA blueprint based on the internal tree representation, packs the blueprint in a CSAR and registers the CSAR with the Orchestrator via xOpera REST API endpoint.

Additional actions are performed by the IaC Blueprint Builder, in cases an AADM JSON contains a reference to an Optimization Model (OM). The Abstract Model Parser extracts the node template, which has an optimization model associated to it, and creates an request to MODAK (Sect. 4.4) to return the reference to the optimized container image and content of the optimized job script. The Parser then substitutes the returned values with respective properties (container runtime and job script content) of the node template, allowing the Orchestrator to retrieve and use the optimized container images and job scripts to deploy an application. Figure 3.17 presents a complete sequence diagram, where IaC generation from an AADM JSON and internal and external components interactions are outlined.

Fig. 3.17

UML sequence diagram showing the IaC generation

The IaC Blueprint Builder has been integrated into the SODALITE security pillar to check authentication and authorization of the calling client with the IAM Service API (Keycloak). The IaC Blueprint Builder is dockerized and its container image can be found in the public DockerHub under sodaliteh2020/iac-blueprint-builder.¹⁶ Additionally, the source code is available in IaC Blueprint Builder GitHub repository.¹⁷

3.7 Conclusion and Future Work

In this chapter, we have presented the SODALITE design-time IaC modeling framework. It permits SODALITE roles to model different facets of the deployment topology of complex applications across heterogeneous infrastructures. It adopts a combined meta-modeling (based on DSLs) and semantics (based on ontology graphs) approach, which leverages the high expressiveness of DSL human-centric languages and the deep knowledge inference and reasoning capabilities of ontology graphs. We have introduced the SODALITE DSLs, designed for modeling different aspects of optimized applications and the required infrastructure resources. We have also introduced the IDE that offers textual and graphical editors for creating DSL model instances and for accessing the main SODALITE processes. The joint collaboration of the IDE and the KB helps the modeler with the intricacies of the models, helped by the context-aware content assist, the semantic validation and the smell detection. We have also introduced the ontology schema, created for assisting on the modeling of DSL model instances, split into tiers that accounts for the different facets of the deployment topology, leveraging the Descriptions and Situation Pattern. And finally, we have introduced the IaC framework, which consumes DSL models to generate the artifacts required for the application deployment by the Orchestrator of the SODALITE Runtime Layer. As future work, we plan to extend SODALITE DSLs and the semantic inference to cover the unsupported modeling capabilities of the TOSCA standard, to extend the content assistance to offer smarter recommendations and wider semantic validation, as well as the graphical modeling support for RMs.