Time-Aware Test Case Execution Scheduling for Cyber-Physical Systems

  • Morten Mossige
  • Arnaud Gotlieb
  • Helge Spieker
  • Hein Meling
  • Mats Carlsson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10416)

Abstract

Testing cyber-physical systems involves the execution of test cases on target-machines equipped with the latest release of a software control system. When testing industrial robots, it is common that the target machines need to share some common resources, e.g., costly hardware devices, and so there is a need to schedule test case execution on the target machines, accounting for these shared resources. With a large number of such tests executed on a regular basis, this scheduling becomes difficult to manage manually. In fact, with manual test execution planning and scheduling, some robots may remain unoccupied for long periods of time and some test cases may not be executed.

This paper introduces TC-Sched, a time-aware method for automated test case execution scheduling. TC-Sched uses Constraint Programming to schedule tests to run on multiple machines constrained by the tests’ access to shared resources, such as measurement or networking devices. The CP model is written in SICStus Prolog and uses the Cumulatives global constraint. Given a set of test cases, a set of machines, and a set of shared resources, TC-Sched produces an execution schedule where each test is executed once with minimal time between when a source code change is committed and the test results are reported to the developer. Experiments reveal that TC-Sched can schedule 500 test cases over 100 machines in less than 4 min for 99.5% of the instances. In addition, TC-Sched largely outperforms simpler methods based on a greedy algorithm and is suitable for deployment on industrial robot testing.

References

  1. 1.
    Aggoun, A., Beldiceanu, N.: Extending CHIP in order to solve complex scheduling and placement problems. Math. Comput. Modell. 17(7), 57–73 (1993)CrossRefGoogle Scholar
  2. 2.
    Baptiste, P., Le Pape, C., Nuijten, W.: Constraint-Based Scheduling: Applying Constraint Programming to Scheduling Problems, vol. 39. Springer Science & Business Media, Berlin (2001)MATHGoogle Scholar
  3. 3.
    Beck, J.C., Feng, T.K., Watson, J.P.: Combining constraint programming and local search for job-shop scheduling. INFORMS J. Comput. 23(1), 1–14 (2011)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Behnke, D., Geiger, M.J.: Test instances for the flexible job shop scheduling problem with work centers. Technical report RR-12-01-01, Helmut-Schmidt University, Hamburg, Germany (2012)Google Scholar
  5. 5.
    Beldiceanu, N., Carlsson, M.: A new multi-resource cumulatives constraint with negative heights. In: Hentenryck, P. (ed.) CP 2002. LNCS, vol. 2470, pp. 63–79. Springer, Heidelberg (2002). doi:10.1007/3-540-46135-3_5 CrossRefGoogle Scholar
  6. 6.
    Brandimarte, P.: Routing and scheduling in a flexible job shop by tabu search. Ann. Oper. Res. 41(3), 157–183 (1993)CrossRefMATHGoogle Scholar
  7. 7.
    Brucker, P., Knust, S.: Complex Scheduling (GOR-Publications). Springer-Verlag New York Inc., Secaucus (2006)MATHGoogle Scholar
  8. 8.
    Brucker, P., Drexl, A., Möhring, R., Neumann, K., Pesch, E.: Resource-constrained project scheduling: notation, classification, models, and methods. Eur. J. Oper. Res. 112(1), 3–41 (1999)CrossRefMATHGoogle Scholar
  9. 9.
    de Campos, J., Arcuri, A., Fraser, G., de Abreu, R.: Continuous test generation: enhancing continuous integration with automated test generation. In: ASE 2014, Västerås, Sweden, pp. 55–66 (2014)Google Scholar
  10. 10.
    Carlsson, M., Ottosson, G., Carlson, B.: An open-ended finite domain constraint solver. In: Glaser, H., Hartel, P., Kuchen, H. (eds.) PLILP 1997. LNCS, vol. 1292, pp. 191–206. Springer, Heidelberg (1997). doi:10.1007/BFb0033845 CrossRefGoogle Scholar
  11. 11.
    Carlsson, M., et al.: SICStus Prolog user’s manual, release 4. Technical Report, SICS - Swedish Institute of Computer Science (2007)Google Scholar
  12. 12.
    Do, H., Mirarab, S., Tahvildari, L., Rothermel, G.: The effects of time constraints on test case prioritization: a series of controlled experiments. IEEE Trans. Soft. Eng. 36(5), 593–617 (2010)CrossRefGoogle Scholar
  13. 13.
    Duvall, P.M., Matyas, S., Glover, A.: Continuous Integration: Improving Software Quality and Reducing Risk. Pearson Education, London (2007)Google Scholar
  14. 14.
    Elbaum, S., Rothermel, G., Penix, J.: Techniques for improving regression testing in continuous integration development environments. In: FSE 2014 (2014)Google Scholar
  15. 15.
    Fowler, M., Foemmel, M.: Continuous integration (2006). http://martinfowler.com/articles/continuousIntegration.html
  16. 16.
    Gotlieb, A., Marijan, D.: Flower: optimal test suite reduction as a network maximum flow. In: ISSTA 2014, San José, CA, USA, pp. 171–180 (2014)Google Scholar
  17. 17.
    Hao, D., Zhang, L., Wu, X., Mei, H., Rothermel, G.: On-demand test suite reduction. In: ICSE 2012, pp. 738–748 (2012)Google Scholar
  18. 18.
    Hartmann, S., Briskorn, D.: A survey of variants and extensions of the resource-constrained project scheduling problem. Eur. J. Oper. Res. 207(1), 1–14 (2010)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Hartmann, S., Kolisch, R.: Experimental evaluation of state-of-the-art heuristics for the resource-constrained project scheduling problem. Eur. J. Oper. Res. 127(2), 394–407 (2000)CrossRefMATHGoogle Scholar
  20. 20.
    Herroelen, W., De Reyck, B., Demeulemeester, E.: Resource-constrained project scheduling: a survey of recent developments. Comput. Oper. Res. 25(4), 279–302 (1998)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Kolisch, R., Hartmann, S.: Experimental investigation of heuristics for resource-constrained project scheduling: an update. Eur. J. Oper. Res. 174(1), 23–37 (2006)CrossRefMATHGoogle Scholar
  22. 22.
    Kreter, S., Schutt, A., Stuckey, P.J.: Modeling and solving project scheduling with calendars. In: Pesant, G. (ed.) CP 2015. LNCS, vol. 9255, pp. 262–278. Springer, Cham (2015). doi:10.1007/978-3-319-23219-5_19 Google Scholar
  23. 23.
    Lin, C., Tang, K., Kapfhammer, G.: Test suite reduction methods that decrease regression testing costs by identifying irreplaceable tests. Inf. Softw. Technol. 56, 1322–1344 (2014)CrossRefGoogle Scholar
  24. 24.
    Marijan, D., Gotlieb, A., Sen, S.: Test case prioritization for continuous regression testing: an industrial case study. In: ICSM 2013, Eindhoven, The Netherlands (2013)Google Scholar
  25. 25.
    Mossige, M.: CSPLib problem 073: test scheduling problem. http://www.csplib.org/Problems/prob073
  26. 26.
    Mossige, M., Gotlieb, A., Meling, H.: Using CP in automatic test generation for ABB robotics’ paint control system. In: O’Sullivan, B. (ed.) CP 2014. LNCS, vol. 8656, pp. 25–41. Springer, Cham (2014). doi:10.1007/978-3-319-10428-7_6 Google Scholar
  27. 27.
    Orso, A., Rothermel, G.: Software testing: a research travelogue (2000–2014). In: FOSE 2014, Hyderabad, India, pp. 117–132 (2014)Google Scholar
  28. 28.
    Orso, A., Shi, N., Harrold, M.J.: Scaling regression testing to large software systems. In: FSE 2014, pp. 241–251. ACM Press, Newport Beach (2004)Google Scholar
  29. 29.
    Schutt, A., Feydy, T., Stuckey, P.J., Wallace, M.G.: Why cumulative decomposition is not as bad as it sounds. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 746–761. Springer, Heidelberg (2009). doi:10.1007/978-3-642-04244-7_58 CrossRefGoogle Scholar
  30. 30.
    Schutt, A., Chu, G., Stuckey, P.J., Wallace, M.G.: Maximising the net present value for resource-constrained project scheduling. In: Beldiceanu, N., Jussien, N., Pinson, É. (eds.) CPAIOR 2012. LNCS, vol. 7298, pp. 362–378. Springer, Heidelberg (2012). doi:10.1007/978-3-642-29828-8_24 CrossRefGoogle Scholar
  31. 31.
    Schutt, A., Feydy, T., Stuckey, P.J.: Scheduling optional tasks with explanation. In: Schulte, C. (ed.) CP 2013. LNCS, vol. 8124, pp. 628–644. Springer, Heidelberg (2013). doi:10.1007/978-3-642-40627-0_47 CrossRefGoogle Scholar
  32. 32.
    Schutt, A., Feydy, T., Stuckey, P.J., Wallace, M.G.: Solving RCPSP/max by lazy clause generation. J. Sched. 16(3), 273–289 (2013)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Siala, M., Artigues, C., Hebrard, E.: Two clause learning approaches for disjunctive scheduling. In: Pesant, G. (ed.) CP 2015. LNCS, vol. 9255, pp. 393–402. Springer, Cham (2015). doi:10.1007/978-3-319-23219-5_28 Google Scholar
  34. 34.
    Simonis, H., O’Sullivan, B.: Search Strategies for rectangle packing. In: Stuckey, P.J. (ed.) CP 2008. LNCS, vol. 5202, pp. 52–66. Springer, Heidelberg (2008). doi:10.1007/978-3-540-85958-1_4 CrossRefGoogle Scholar
  35. 35.
    Stolberg, S.: Enabling agile testing through continuous integration. In: AGILE 2009, pp. 369–374. IEEE (2009)Google Scholar
  36. 36.
    Szeredi, R., Schutt, A.: Modelling and solving multi-mode resource-constrained project scheduling. In: Rueher, M. (ed.) CP 2016. LNCS, vol. 9892, pp. 483–492. Springer, Cham (2016). doi:10.1007/978-3-319-44953-1_31 CrossRefGoogle Scholar
  37. 37.
    Taillard, E.: Benchmarks for basic scheduling problems. Eur. J. Oper. Res. 64(2), 278–285 (1993)CrossRefMATHGoogle Scholar
  38. 38.
    Walcott, K.R., Soffa, M.L., Kapfhammer, G.M., Roos, R.S.: Time-aware test suite prioritization. In: ISSTA 2006, Portland, Maine, USA, pp. 1–12 (2006)Google Scholar
  39. 39.
    Zhang, L., Hou, S., Guo, C., Xie, T., Mei, H.: Time-aware test-case prioritization using integer linear programming. In: ISSTA 2009, Chicago, IL, USA, pp. 213–224 (2009)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Morten Mossige
    • 1
    • 3
  • Arnaud Gotlieb
    • 2
  • Helge Spieker
    • 2
  • Hein Meling
    • 3
  • Mats Carlsson
    • 4
  1. 1.ABB RoboticsBryneNorway
  2. 2.Simula Research LaboratoryLysakerNorway
  3. 3.University of StavangerStavangerNorway
  4. 4.RISE SICSKistaSweden

Personalised recommendations