Skip to main content
Book cover

International Symposium on Search Based Software Engineering

SSBSE 2020: Search-Based Software Engineering pp 58–72Cite as

Search-Based Testing for Scratch Programs

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 12420)

Abstract

Block-based programming languages enable young learners to quickly implement fun programs and games. The Scratch programming environment is particularly successful at this, with more than 50 million registered users at the time of this writing. Although Scratch simplifies creating syntactically correct programs, learners and educators nevertheless frequently require feedback and support. Dynamic program analysis could enable automation of this support, but the test suites necessary for dynamic analysis do not usually exist for Scratch programs. It is, however, possible to cast test generation for Scratch as a search problem. In this paper, we introduce an approach for automatically generating test suites for Scratch programs using grammatical evolution. The use of grammatical evolution clearly separates the search encoding from framework-specific implementation details, and allows us to use advanced test acceleration techniques. We implemented our approach as an extension of the Whisker test framework. Evaluation on sample Scratch programs demonstrates the potential of the approach.

Keywords

  • Search-based testing
  • Block-based programming
  • Scratch

Authors listed in alphabetical order.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-59762-7_5
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   64.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-59762-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   84.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.

Notes

  1. 1.

    https://scratch.mit.edu/statistics/, last accessed 9.6.2020.

  2. 2.

    https://pptr.dev/, last accessed 9.6.2020.

References

  1. Aivaloglou, E., Hermans, F.: How kids code and how we know: an exploratory study on the scratch repository. In: Proceedings of ICER, pp. 53–61 (2016)

    Google Scholar 

  2. Anjum, M.S., Ryan, C.: Seeding grammars in grammatical evolution to improve search based software testing. In: Hu, T., Lourenço, N., Medvet, E., Divina, F. (eds.) EuroGP 2020. LNCS, vol. 12101, pp. 18–34. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-44094-7_2

    CrossRef  Google Scholar 

  3. Arcuri, A.: It really does matter how you normalize the branch distance in search-based software testing. Softw. Test. Verif. Reliab. 23(2), 119–147 (2013)

    CrossRef  Google Scholar 

  4. Boe, B., Hill, C., Len, M., Dreschler, G., Conrad, P., Franklin, D.: Hairball: lint-inspired static analysis of scratch projects. In: Proceedings of SIGCSE, pp. 215–220 (2013)

    Google Scholar 

  5. Chang, Z., Sun, Y., Wu, T.Y., Guizani, M.: Scratch analysis tool (SAT): a modern scratch project analysis tool based on ANTLR to assess computational thinking skills. In: Proceedings of IWCMC, pp. 950–955. IEEE (2018)

    Google Scholar 

  6. Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39(2), 276–291 (2012)

    CrossRef  Google Scholar 

  7. Fraser, G., Arcuri, A.: Handling test length bloat. Softw. Test. Verif. Reliab. 23(7), 553–582 (2013)

    CrossRef  Google Scholar 

  8. Frädrich, C., Obermüller, F., Körber, N., Heuer, U., Fraser, G.: Common bugs in scratch programs. In: Proceedings of ITICSE, pp. 89–95 (2020)

    Google Scholar 

  9. Gross, F., Fraser, G., Zeller, A.: Search-based system testing: high coverage, no false alarms. In: Proceedings of ISSTA, pp. 67–77 (2012)

    Google Scholar 

  10. Hermans, F., Aivaloglou, E.: Do code smells hamper novice programming? A controlled experiment on scratch programs. In: Proceedings of ICPC, pp. 1–10. IEEE (2016)

    Google Scholar 

  11. Hermans, F., Stolee, K.T., Hoepelman, D.: Smells in block-based programming languages. In: Proceedings of VL/HCC, pp. 68–72. IEEE (2016)

    Google Scholar 

  12. Johnson, D.E.: Itch: individual testing of computer homework for scratch assignments. In: Proceedings of SIGCSE, pp. 223–227 (2016)

    Google Scholar 

  13. Korel, B.: Automated software test data generation. IEEE Trans. Softw. Eng. (TSE) 16, 870–879 (1990)

    CrossRef  Google Scholar 

  14. Mahmood, R., Mirzaei, N., Malek, S.: Evodroid: segmented evolutionary testing of android apps. In: Proceedings of ESEC/FSE, pp. 599–609 (2014)

    Google Scholar 

  15. Maloney, J., Resnick, M., Rusk, N., Silverman, B., Eastmond, E.: The scratch programming language and environment. TOCE 10(4), 1–15 (2010)

    CrossRef  Google Scholar 

  16. Mao, K., Harman, M., Jia, Y.: Sapienz: multi-objective automated testing for android applications. In: Proceedings of ISSTA, pp. 94–105 (2016)

    Google Scholar 

  17. Meerbaum-Salant, O., Armoni, M., Ben-Ari, M.: Habits of programming in scratch. In: Proceedings of ITICSE, pp. 168–172 (2011)

    Google Scholar 

  18. Moreno-León, J., Robles, G.: Dr. scratch: a web tool to automatically evaluate scratch projects. In: Proceedings of WIPSCE, pp. 132–133 (2015)

    Google Scholar 

  19. O’Neill, M., Ryan, C.: Grammatical evolution. IEEE Trans. Evol. Comput. 5(4), 349–358 (2001)

    CrossRef  Google Scholar 

  20. Panichella, A., Kifetew, F.M., Tonella, P.: Reformulating branch coverage as a many-objective optimization problem. In: Proceedings of ICST, pp. 1–10 (2015)

    Google Scholar 

  21. Price, T.W., Dong, Y., Lipovac, D.: iSnap: towards intelligent tutoring in novice programming environments. In: Proceedings of SIGCSE, pp. 483–488 (2017)

    Google Scholar 

  22. Stahlbauer, A., Kreis, M., Fraser, G.: Testing scratch programs automatically. In: Proceedings of ESEC/SIGSOFT FSE, pp. 165–175. ACM (2019)

    Google Scholar 

  23. Techapalokul, P., Tilevich, E.: Quality hound-an online code smell analyzer for scratch programs. In: Proceedings of VL/HCC, pp. 337–338. IEEE (2017)

    Google Scholar 

  24. Techapalokul, P., Tilevich, E.: Understanding recurring quality problems and their impact on code sharing in block-based software. In: Proceedings of VL/HCC, pp. 43–51. IEEE (2017)

    Google Scholar 

  25. Techapalokul, P., Tilevich, E.: Code quality improvement for all: Automated refactoring for scratch. In: Proceedings of VL/HCC, pp. 117–125. IEEE (2019)

    Google Scholar 

  26. Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Inf. Softw. Tech. 43(14), 841–854 (2001)

    CrossRef  Google Scholar 

Download references

Acknowledgements

This work is supported by EPSRC project EP/N023978/2 and DFG project FR 2955/3-1 “TENDER-BLOCK: Testing, Debugging, and Repairing Blocks-based Programs”.

Author information

Authors and Affiliations

  1. University of Passau, Innstr. 33, 94032, Passau, Germany

    Adina Deiner, Christoph Frädrich, Gordon Fraser, Sophia Geserer & Niklas Zantner

Authors
  1. Adina Deiner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christoph Frädrich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gordon Fraser
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sophia Geserer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Niklas Zantner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gordon Fraser .

Editor information

Editors and Affiliations

  1. Monash University, Melbourne, VIC, Australia

    Aldeida Aleti

  2. Delft University of Technology, Delft, The Netherlands

    Annibale Panichella

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Deiner, A., Frädrich, C., Fraser, G., Geserer, S., Zantner, N. (2020). Search-Based Testing for Scratch Programs. In: Aleti, A., Panichella, A. (eds) Search-Based Software Engineering. SSBSE 2020. Lecture Notes in Computer Science(), vol 12420. Springer, Cham. https://doi.org/10.1007/978-3-030-59762-7_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59762-7_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59761-0

  • Online ISBN: 978-3-030-59762-7

  • eBook Packages: Computer ScienceComputer Science (R0)