Abstract
Usage-based testing focuses quality assurance on highly used parts of the software. The basis for this are usage profiles based on which test cases are generated. There are two fundamental approaches in usage-based testing for deriving usage profiles: either the system under test (SUT) is observed during its operation and from the obtained usage data a usage profile is automatically inferred, or a usage profile is modeled by hand within a model-based testing (MBT) approach. In this article, we propose a third and combined approach, where we automatically infer a usage profile and create a test data repository from usage data. Then, we create representations of the generated tests and test data in the test model from an MBT approach. The test model enables us to generate executable Testing and Test Control Notation version 3 (TTCN-3) and thereby allows us to automate the test execution. Together with industrial partners, we adopted this approach in two pilot studies. Our findings show that usage-based testing can be applied in practice and greatly helps with the automation of tests. Moreover, we found that even if usage-based testing is not of interest, the incorporation of usage data can ease the application of MBT.
Similar content being viewed by others
Notes
E.g., JAX-WS (https://jax-ws.java.net/) creates for each operation an input action and an output action. The input action gets the suffix Request and the output action gets the suffix Response.
Please note that this does not have to be the case in synchronous communication if a service internally sends a request to another service before responding to the client.
References
Dedalus. http://www.dedalus.eu/
Health Level Seven International. http://www.hl7.org.uk/
Itainnova-instituto tecnológico de aragón. http://www.itainnova.es/
Object management group (omg). http://www.omg.org
Selenium webdriver. http://www.seleniumhq.org/
ALL4TEC: MaTeLo. http://www.all4tec.net/index.php/en/model-based-testing/20-markov-test-logic-matelo(linkcheckedJune2nd,2014)
Baker, P., Dai, Z.R., Grabowski, J., Haugen, O., Schieferdecker, I., Williams, C.: Model-driven testing: using the uml testing profile. Springer-Verlag New York Inc, Secaucus (2007)
Barcelona Liédana, M.A., López-Nicolás, G., García-Borgon̋ón, L.: Practical experiences in the usage of midas in the logistics domain. Software Tools for Technology Transfer (STTT), accepted (2016)
Chen, C., Zaidman, A., Gross, H.G.: A framework-based runtime monitoring approach for service-oriented software systems. In: Proceedings of the International Workshop on Quality Assurance for Service-Based Applications, QASBA ’11, pp. 17–20. ACM, New York, NY, USA (2011). doi:10.1145/2031746.2031752
Cheung, R.C.: A user-oriented software reliability model. IEEE Trans. Softw. Eng. 6(2), 118–125 (1980). doi:10.1109/TSE.1980.234477
Cover, T.M., Thomas, J.A.: Elements of information theory, 2nd edn. Wiley, Hoboken (2006)
De Francesco, A., Di Napoli, C., Giordano, M., Ottaviano, G., Perego, R., Tonellotto, N.: A soa testing platform on the cloud: The midas experience. In: Intelligent Networking and Collaborative Systems (INCoS), 2014 International Conference on, pp. 659–664 (2014). doi:10.1109/INCoS.2014.62
Di Napoli, C., De Francesco, A., Giordano, M., Ottaviano, G., Tonellotto, N., Perego, R.: Midas: a cloud platform for soa testing as a service. International Journal of High Performance Computing and Networking (2015) (in press)
Dulz, W., Zhen, F.: MaTeLo—statistical usage testing by annotated sequence diagrams, Markov Chains and TTCN-3. In: Proceedings of the 3rd International Conference on Quality Software (QSIC) (2003)
Eclipse Foundation: Papyrus. https://eclipse.org/papyrus/
Feliachi, A., Le Guen, H.: Generating transition probabilities for automatic model-based test generation. In: Proceedings of the 3rd International Conference on Software Testing, Verification and Validation (ICST) (2010). doi:10.1109/ICST.2010.26
Feller, W.: An introduction to probability theory and its applications. Wiley, Hoboken (1971)
Grabowski, J., Hogrefe, D., Réthy, G., Schieferdecker, I., Wiles, A., Willcock, C.: An introduction to the testing and test control notation (TTCN-3). Comput. Netw. 42(3), 375–403 (2003). doi:10.1016/S1389-1286(03)00249-4
GS1: Logistics interoperability model version 1. http://www.gs1.org/lim (2007)
Health Level Seven International: Hl7 version 3 standard: Identification service (is), release 1. http://www.hl7.org/implement/standards/product_brief.cfm?product_id=87 (2014)
Healthcare Service Specification Project (HSSP): Hssp specifications. https://hssp.wikispaces.com/specs
Herbold, S., Bünting, U., Grabowski, J., Waack, S.: Deployable capture/replay supported by internal messages. Adv. Comput. 85, 327–367 (2012)
Herbold, S., Grabowski, J., Waack, S.: A Model for Usage-based Testing of Event-driven Software. In: 3rd International Workshop on Model-Based Verification & Validation From Research to Practice. IEEE Computer Society (2011)
Herbold, S., Harms, P.: AutoQUEST—automated quality engineering of event-driven software. In: Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (2013). doi:10.1109/ICSTW.2013.23
International Software Testing Qualitifications Board (ISTQB): Standard glossary of terms used in Software Testing, Version 2.1 (2010)
Kosala, R., Blockeel, H.: Web mining research: a survey. ACM SIGKDD Explor. Newsl. 2(1), 1–15 (2000). doi:10.1145/360402.360406
Le Guen, H., Marie, R., Thelin, T.: Reliability estimation for statistical usage testing using markov chains. In: Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE) (2004). doi:10.1109/ISSRE.2004.33
Littlewood, B.: A reliability model for systems with Markov structure. J. R. Stat. Soc. Ser. C (Applied Statistics) 24(2), 172–177 (1975)
MIDAS Consortium: Model and Inference Driven Automated testing of Servicesarchitectures (MIDAS). http://www.midas-project.eu (link checked June 2nd, 2014)
Motahari-Nezhad, H.R., Saint-Paul, R., Casati, F., Benatallah, B.: Event correlation for process discovery from web service interaction logs. VLDB J. 20(3), 417–444 (2011). doi:10.1007/s00778-010-0203-9
Object Management Group (OMG): Retrieve, locate, and update service (rlus). http://www.omg.org/spec/RLUS/ (2011)
Rumbaugh, J., Jacobson, I., Booch, G.: Unified modeling language reference manual, the (2nd edition). Pearson Higher Education, New York (2004)
Srivastava, J., Cooley, R., Deshpande, M., Tan, P.N.: Web usage mining: discovery and applications of usage patterns from Web data. ACM SIGKDD Explor. Newsl. 1(2), 12–23 (2000). doi:10.1145/846183.846188
Testing Technologies: Ttworkbench. http://www.testingtech.com/products/ttworkbench.php
Tonella, P., Ricca, F.: Dynamic model extraction and statistical analysis of web applications. In: Proceedings of the 4th International Workshop on Web Site Evolution (WSE) (2002)
Tonella, P., Ricca, F.: Statistical testing of web applications. J. Softw. Maint. Evol. Res. Pract. 16(1–2), 103–127 (2004). doi:10.1002/smr.284
Tonella, P., Ricca, F.: Dynamic Model extraction and statistical analysis of web applications: Follow-up after 6 years. In: Proceedings of the 10th International Symposium on Web Site Evolution (WSE) (2008)
Tonella, P., Tiella, R., Nguyen, C.D.: Interpolated n-grams for model based testing. In: Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, pp. 562–572. ACM, New York, NY, USA (2014). doi:10.1145/2568225.2568242
Walton, G.H., Poore, J.H., Trammell, C.J.: Statistical testing ofsoftware based on a usage model. Softw. Pract. Ant Exp. 25(1), 97–108 (1995). doi:10.1002/spe.4380250106
Wendland, M.F., Schneider, M., Hoffmann, A.: A model-driven approach to test automation for soa systems. Software Tools for Technology Transer (submitted) (2015)
Wesslén, A., Wohlin, C.: Modelling and generation of software usage. In: Proceedings of the 5th International Conference on Software Quality (1995)
Whittaker, J.A., Poore, J.H.: Markov analysis of software specifications. ACM Trans. Softw. Eng. Methodol. 2(1), 93–106 (1993). doi:10.1145/151299.151326
Whittaker, J.A., Thomason, M.G.: A Markov chain model for statistical software testing. IEEE Trans. Softw. Eng. 20(10), 812–824 (1994). doi:10.1109/32.328991
Woit, D.M.: Specifying operational profiles for modules. SIGSOFT Softw. Eng. Notes 18(3), 2–10 (1993). doi:10.1145/174146.154187
Woit, D.M.: Conditional-event usage testing. In: Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research, CASCON ’98, p. 23. IBM Press (1998)
World Wide Web Consortium (W3C): Web services addressing (ws-addressing). http://www.w3.org/Submission/ws-addressing/ (2004)
Yufang Dan Nicolas Stouls, S.F.C.C.: A Monitoring approach for dynamic service-oriented architecture systems. In: SERVICE COMPUTATION 2012: The Fourth International Conferences on Advanced Service Computing, pp. 20–23. XPS (Xpert Publishing Services) (2012)
Acknowledgments
This work was done in the context of the “Model and Inference Driven-Automated testing of Services architectures” (MIDAS) European project (project number 318786). We would like to thank Testing Technologies for their support in terms of licensing as well as feedback to support requests regarding TTworkbench; Fraunhofer FOKUS for the creation and maintenance of the MIDAS DSL and TTCN-3 generation; and our pilot partners from ITAINNOVA and Dedalus S.p.A. for their support in conducting the pilot studies.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Herbold, S., Harms, P. & Grabowski, J. Combining usage-based and model-based testing for service-oriented architectures in the industrial practice. Int J Softw Tools Technol Transfer 19, 309–324 (2017). https://doi.org/10.1007/s10009-016-0437-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10009-016-0437-y