Enhanced Sketchnoting Through Semantic Integration of Learning Material

  • Aryobarzan Atashpendar
  • Christian GrévisseEmail author
  • Steffen Rothkugel
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1051)


Within the transition from pen-and-paper to digital solutions, many tablet-based note-taking apps have been developed. In addition, sketchnoting, a highly visual form of note-taking, has gained in popularity, especially in teaching and learning. While current solutions already reduce the burden of organizing and searching through stacks of paper, they provide little support in finding additional information related to the notes, and search capabilities are mostly limited to textual queries and content. In this paper, we present a novel solution for digital sketchnoting aimed at enhancing the learning experience of students. In addition to common capabilities such as handwriting recognition, our note-taking app integrates semantic annotation and drawing recognition. Handwritten notes are recognized and passed through concept recognition and entity linking tools to enable a knowledge graph-based integration of contextually relevant learning resources. We extend traditional search capabilities by including the semantic metadata from the related material as well as enabling visual queries to find recognized sketches. Finally, resembling the exchange of paper notes among students, our app allows to share the semantically enhanced notes with other devices.


Sketchnoting Learning material Handwriting recognition Semantic annotation Drawing recognition Visual search 


  1. 1.
    Ahn, R., Ingham, S., Mendez, T.: Socially constructed learning activity: communal note-taking as a generative tool to promote active student engagement. Transform. Dialogues Teach. Learn. J. 8(3), 1–15 (2016)Google Scholar
  2. 2.
    Chandler, P., Sweller, J.: The split-attention effect as a factor in the design of instruction. Br. J. Educ. Psychol. 62(2), 233–246 (1992). CrossRefGoogle Scholar
  3. 3.
    Chang, M., Kuo, R., Chang, M., Kinshuk: Online annotation system and student clustering platform. In: 2015 8th International Conference on Ubi-Media Computing (UMEDIA), pp. 202–207, August 2015.
  4. 4.
    Dimeo, R.: Sketchnoting: an analog skill in the digital age. SIGCAS Comput. Soc. 46(3), 9–16 (2016). Scholar
  5. 5.
    García-González, H., Gayo, J.E.L., Paule-Ruiz, M.: Enhancing e-learning content by using semantic web technologies. IEEE Trans. Learn. Technol. 10(4), 544–550 (2017). Scholar
  6. 6.
    Grévisse, C., Manrique, R., Marino, O., Rothkugel, S.: SoLeMiO: semantic integration of learning material in office. In: Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2018, pp. 1524–1532. Association for the Advancement of Computing in Education (AACE), Las Vegas, NV, United States, October 2018.
  7. 7.
    Guidera, S.: The digital sketchbook: perceptions of the effectiveness of tablet-based sketching apps for design drawing. In: Proceedings of 10th Annual International Conference of Education, Research and Innovation, ICERI 2017, IATED, pp. 8498–8504, November 2017.
  8. 8.
    Ha, D., Eck, D.: A neural representation of sketch drawings. In: Proceedings of the Seventh International Conference on Learning Representations (2018).
  9. 9.
    Krieger, K.: Creating learning material from web resources. In: Gandon, F., Sabou, M., Sack, H., d’Amato, C., Cudré-Mauroux, P., Zimmermann, A. (eds.) ESWC 2015. LNCS, vol. 9088, pp. 721–730. Springer, Cham (2015). Scholar
  10. 10.
    Kuznekoff, J.H., Munz, S., Titsworth, S.: Mobile phones in the classroom: examining the effects of texting, twitter, and message content on student learning. Commun. Educ. 64(3), 344–365 (2015). Scholar
  11. 11.
    Longcamp, M., Zerbato-Poudou, M.T., Velay, J.L.: The influence of writing practice on letter recognition in preschool children: a comparison between handwriting and typing. Acta Psychologica 119(1), 67–79 (2005). Scholar
  12. 12.
    Mendes, P.N., Jakob, M., García-Silva, A., Bizer, C.: DBpedia spotlight: shedding light on the web of documents. In: Proceedings of the 7th International Conference on Semantic Systems, I-Semantics 2011, pp. 1–8. ACM, New York (2011).
  13. 13.
    Paek, S., Fulton, L.A.: Elementary students using a tablet-based note-taking application in the science classroom. J. Digit. Learn. Teach. Educ. 32(4), 140–149 (2016). Scholar
  14. 14.
    Prat, N., Wattiau, I., Akoka, J.: Artifact evaluation in information systems design-science research - a holistic view. In: Proceeding of the 19th Pacific Asia Conference on Information Systems, PACIS 2014, June 2014Google Scholar
  15. 15.
    Schmeck, A., Opfermann, M., van Gog, T., Paas, F., Leutner, D.: Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings. Instr. Sci. 43(1), 93–114 (2015). Scholar
  16. 16.
    Seddati, O., Dupont, S., Mahmoudi, S.: Deepsketch 3. Multimedia Tools Appl. 76(21), 22333–22359 (2017). Scholar
  17. 17.
    Sweller, J., van Merrienboer, J.J.G., Paas, F.G.W.C.: Cognitive architecture and instructional design. Educ. Psychol. Rev. 10(3), 251–296 (1998). Scholar
  18. 18.
    Winter, J., Cotton, D., Gavin, J., Yorke, J.: Effective e-learning? Multi-tasking, distractions and boundary management by graduate students in an online environment. Res. Learn. Technol. 18(1) (2010). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Computer Science and Communications Research UnitUniversity of LuxembourgEsch-sur-AlzetteLuxembourg

Personalised recommendations