Now that we introduced the concepts behind MBT and cloud computing work, we come back to our initial vision from the introduction: massively scaling up test campaigns using MBT and cloud computing. The general concept behind this idea is quite simple: generate a huge amount of tests using MBT and use the elasticity of the cloud to scale the test execution. However, there are several conceptual and technical problems that must be resolved to achieve this. Within this section, we will outline the problems and how to overcome them. We will discuss how an IaaS cloud can be used to define the infrastructure for a PaaS solution for testing. By implementing applications on this platform, we show how a MBT test application can be deployed on a cloud to create a Testing as a Service (TaaS) solution.
This discussion will mainly be based on the results of the MIDAS European project [18]. The aim of MIDAS was the development of such a platform for the testing of Web services based on the SOA, specifically the orchestration of multiple services. The problem that exists here is that services by themselves can already be quite complex and offer lots of operations. If multiple services are used together in a so-called service orchestration, this complexity grows exponentially. Hence, the testing of such orchestrations is a hard problem. The approach of MIDAS was to use the advantages of MBT and cloud to solve this problem by allowing large-scale test campaigns.
Setting up the infrastructure
The first step to achieve TaaS is to create an appropriate cloud computing infrastructure. The approach chosen within the MIDAS project was to create a PaaS solution based on an existing IaaS infrastructure. The rationale behind this was that elasticity and pay per use can be achieved with IaaS. Moreover, IaaS allows for a nearly arbitrary definition of the software stack that is installed. While existing PaaS solution also supports elasticity and pay per use, they lack the required flexibility to allow for testing Web services, e.g., they do not provide a test execution engine and do not allow for potentially required firewall configurations in order to communicate with the Web services that shall be tested.
The PaaS of “Model and Inference Driven - Automated testing of Services architectures” (MIDAS) is based on the SOA paradigm, a quite common solution for PaaS services, e.g., supported by Amazon Web Services (AWS) [1] and by Google’s AppEngine [11]. MIDAS provides SOAP interfaces based on Web Service Description Language (WSDL) that can be used for the definition of services for the MIDAS PaaS. The platform itself already provides a set of services which are required independent of testing:
Additionally, some services to support the test activities are provided:
-
definition of test campaigns,
-
test script compilation, and
-
test execution.
The test scripts and execution are based on Testing and Test Control Notation version 3 (TTCN-3). All additional test services (i.e., the services that generate the TTCN-3 test cases from models) that are developed for MIDAS must implement the SOAP interfaces prescribed by the MIDAS PaaS (Sect. 4.2).
While the above sounds straight forward to implement, there is a major problem for many applications: many existing model-based tools as well as high-quality test execution engines are proprietary and require licensing. These licenses are usually bound to a single machine. However, for scaling an IaaS multiple VMs are created, thus requiring multiple licenses. Hence, while the provisioning of VMs with the required software installed may be unproblematic, the actual execution of the installed software may not be possible due to missing licenses. While this is a common problem for porting applications to the cloud, no general solution is yet available, because the underlying problem is not of technical nature, but rather due to the licensing models.
Within the MIDAS project, we faced these problems with our TTCN-3 compiler and execution engine. Both were based on TTworkbench [23]. Fortunately, Testing Technologies [22], the provider of TTworkbench, was so kind as to allow us to use a licensing server that could distribute licenses to VMs dynamically. Thus, we could scale up and down the usage of licenses with the currently required VMs for executing tests. We think that this approach provides the blueprint for a feasible solution to structure flexible licensing that allows the support of a pay-per-use model as is common for cloud applications. By monitoring which licenses are used for which amount of time, a payment structure between the tool provider and the cloud platform provider can be defined. More details on this problem, as well as other technical problems which were solved within the MIDAS project, can be found in the literature [7, 8].
Model-based testing on the cloud
Using the MIDAS platform, it is possible to plug in Web services that provide the capabilities required for MBT to the platform. In order to maximize the flexibility and scalability of the platform, each service should be defined as a closed unit, i.e., without directly calling any other service of the MIDAS platform, except for the services offered by the MIDAS platform for file management and user authentication.
The test services communicate via files. The idea behind this approach is to allow test services to exchange information as serialized representations of the artifact that was generated. Thus, if we have a test case generation service that uses the serialization format X, all other test services provided within MIDAS that can read X as input may use the result, e.g., test prioritization services, or TTCN-3 generation services. To demonstrate the power of this concept, MIDAS itself uses a modeling approach based on UML [20] augmented with concepts from the UML Testing Profile (UTP) [2] and some additional restrictions and stereotypes required for testing of SOA applications. The serializations of these models are the files that are exchanged between services and allow a seamless interaction between different services.
Figure 7 shows an example for services deployed on the platform and their interactions. Test case generation services may use a MIDAS DSL model as input to generate tests using different test strategies. The results from the test case generation can then be either directly used by the TTCN-3 generation service or first used by test prioritization services that define an order for the tests for their execution. Finally, the generated TTCN-3 is used by the platform components for compilation and execution against the SUT.
Since each of the services is defined as a single and isolated unit, new services can be added without much effort, e.g., a new test case generation service can be plugged in that follows a different test strategy. This makes the parts interchangeable and allows for the definition of a variety of test approaches. The only limitation, from a developers point of view, is the need to adhere to a commonly shared input and output format of the models that are exchanged between the files, i.e., in case of MIDAS the MIDAS DSL. As long as this format is used, one can reuse all existing services that also utilize the same approach.
However, even this limitation can be circumvented. Any exchange format is possible, if all services are replaced, i.e., also a TTCN-3 generation service is provided. This allows the usage of other MBT approaches not based on the MIDAS DSL. This is exemplified by the MBT approach based on Service Component Architecture (SCA) XML [19] and State Chart XML (SCXML) [25] discussed by Hillah et al. [14]. Thus, the architecture of the PaaS serves as a versatile way to define MBT approaches on a cloud infrastructure. However, the effort for developers is higher as a new implementation of all services is required, including test case generation and TTCN-3 generation.
Definition of test campaigns
The previous sections explained how a platform for testing can be built on the cloud and how MBT solutions can be implemented from a developers point of view. Once these services are available, the platform looks from a testers point of view like a SaaS application, to which we refer to as TaaS: all software is available and running, and the testers only need to provide the required inputs for running test campaigns.
These inputs are twofold: (1) a MIDAS DSL model and (2) an orchestration for the services provided by MIDAS. The DSL model provides the structural and deployment information about the SUT. Additionally, the DSL model must provide a behavioral model that can be used by the test strategies that shall be used by the test case generation services. The orchestration defines which of the services provided by MIDAS are called with which files as input and in which order. Basically, this can be thought of as one path through the data flow diagram of the service deployment we described above (see Fig. 7). Figure 8 shows an example of such a path. With this orchestration, the test generation service 1 is called; then, the TTCN-3 code is generated for the generated tests and afterward compiled and executed. The orchestrations are soft-coded. Thus, theoretically any order of services installed on the TaaS can be called. This is only restricted by the required inputs and outputs of the services which must be compatible.