Keywords

1 Introduction

The electronic educational materials are indispensable for the increasingly widespread trend of the “flipped classroom” teaching model. They must be both attractive to keep the student’s attention and accessible to disabled students. Some groups of students, such as the blind, have limited access to electronic materials containing visualised spatial objects because the blind’s perception of the environment mainly focuses on sound and tactile senses. These problems are more apparent in subjects such as mathematics, physics, chemistry, that are rich in math formulas and drawings. The accessibility barrier is reduced or aligned by creating materials according to the WCAG (Web Content Accessibility Guidelines) and WAI-ARIA (WEB Reach Internet Application) standard [9, 10], under development by W3C consortium. The WCAG and WAI-ARIA specify additional elements of web pages that increase accessibility as well as a user interface based on keyboard shortcuts for exploring them. So, we can say that by interacting with pages created in this way and using the interface according to the WCAG and ARIA recommendations, the blind users are in virtual reality, which for them becomes an audible and tangible reality. However, the WCAG and ARIA recommendations do not include support for finding, recognising, and exploring specific objects such as mathematical formulas, function graphs, drawings of geometric figures, mathematical quizzes, and tasks of the type “join into pairs”.

The problem of access to mathematical content by visually impaired users, especially blind users is well known. Numerous scientific publications report further new solutions that help in recognising the mathematical content. The research on the latest AR devices has demonstrated their extreme usefulness for increasing the student motivation in the learning process [5,6,7]. In [3] an in-depth analysis of published research on AR and statistics on the use of AR in education and goals of their use is presented. The two most frequently indicated goals were additional explanations of the subject and extension of information. The latest book on the subject published in 2020 [2], a multi-author review of the state of AR applications in education, confirmed the effective impact of multi-sensory AR in early education, similar conclusions are in [8]. In early mathematics, as well as advanced mathematics, for the needs of sighted students, AR is used to visualise 3D mathematical expressions [1]. According to the authors’ knowledge, there are no reports on research on the multi-sensual AR in math education of blind and low vision students. Especially for this group of students using alternative interfaces, who instead of sight rely on the senses of hearing and touch, there can be effective solutions providing additional information about each mathematical object using the multi-sensual user interface.

The article describes information superstructure that is received multi-sensually by the student, beyond the information recommended by WCAG and ARIA. It allows for interactive recognition of mathematical content. In conjunction with the developed technique, accessible for editing formulas, quizzes with math content, and creating graphics, it gives the blind student the possibility of creative, interactive, independent math learning.

2 Method

In order to learn mathematics effectively, blind students should be as independent as possible, especially in the “flipped classroom” learning model. This means that they should have access to all elements of a mathematical multimedia document, e.g. a manual or worksheet. Visually impaired students must be able to analyse thoroughly, imagine, and understand each mathematical object. Otherwise, they are threatened with cognitive impairment.

In their daily work with the computer, blind users use only the keyboard and special software called a screen reader. That is why the blind persons’ exploration of the document is different from that of a sighted. For example, instead of clicking a mouse pointer, they will use the tab key to highlight the button and then will press the space bar.

The accessibility of different mathematical objects can be achieved by enriching pages with additional information elements forming the Augmented Reality (AR) layers in opposition to the cognitive reality resulting from the use of the WCAG and ARIA elements.

We propose a method of enriching a mathematical document with multi-sensory AR elements, applied by the user in the subsequent stages of more and more explicit recognition of mathematical objects. There is one particular AR information layer associated with each recognition stage. There are three AR information layers supporting the user in three subsequent stages of exploring a mathematical document. There is also the fourth optional layer used depending on the teacher’s decision and/or the student’s needs.

Additional AR information elements in the form of texts and sounds of various types are accessible through the multi-sensual user interface such as synthetic speech, touch gestures on the touch screen or by the touch of the haptic Braille display also called as a braille line (see Table 1). It is a device connected to the computer, presenting the characters in the form of protruding pins forming the letters of the Braille alphabet.

Table 1. An example of using the 1st and 2nd AR layers: finding formulas and learning the content of a selected formula
Full size table

2.1 First AR Layer: General Recognition of Mathematical Objects in the Document

The first general information layer (additional to WCAG and ARIA requirements), contains text elements that are conveyed by synthetic speech, and it informs about encountering object such a graphics, mathematical formulas, quizzes, questions, answers, pairing fields and links to comments recorded by a teacher (see Table 1). The first layer user interface (UI) is a set of keyboard shortcuts that allows for simple finding locations of initially recognised objects in a document. It is also enriched with the haptic interface for the Braille display. For quickly locating mathematical objects of a particular type in a document (formulas, graphics, quizzes and others), there is a set of shortcut keys, the so-called hotkeys for quick access.

An Example of Using the First AR Layer

When the student, while navigating a document, using arrow keys on his keyboard, puts the cursor on a mathematical formula, the screen reader, through a speech synthesiser, will read to them that it is a formula. Similarly, he/she will get information on other types of elements such as graphics, pairing fields, questions and answers in quizzes. Moreover, each of this message is sent simultaneously to the Braille display in a shortened form to accelerate the speed of haptic reading.

2.2 Second AR Layer: More Information About Math Objects

After encountering a mathematical object and recognising its type, the student can examine it more thoroughly. It is possible, thanks to the AR elements placed in the second layer (see Table 1). This layer contains information elements such as semantically readable formula text in Polish or English via speech synthesiser, graphic titles and descriptions, and content of pairing fields, questions or answers that can be a text, a formula or a graphic. The student can simultaneously read this information using the Braille display. UI of the second layer is a set of keyboard shortcuts used to obtain information about each math object in the document. Additionally, for purposes of haptic reading of formulas, this layer is equipped with converters of mathematical MathML web notation [11] to UEB - English mathematical Braille notation [12] and BNM - Polish mathematical Braille notation [13].

An Example of Using the Second AR Layer

When students encounter an object in a document and are informed that it is a mathematical formula, they can become familiar with its content. The text of semantic reading of the formula in Polish or English is automatically created and then read with synthetic speech. For example, for the formula 1/3 + 2/3 = 1, the text of the readout is: “one third plus two thirds equal one.” In addition to listening, the student can read the same formula haptic through touch using a Braille display. It is possible thanks to converters of MathML formula notation in HTML to Braille BNM (Polish) or UEB (English) notation.

2.3 Third AR Layer: Thorough Exploration of the Math Object

The third AR layer was developed for detailed exploration of formulas, function graphs and geometric drawings. It consists of the following items: texts of automatically generated or/and manually added descriptions of graphic elements; comments of the whole graph and its elements; texts of the formula elements; continuous sounds with constant and variable monotonicity describing graph elements; short technical sounds called audio-icons. UI access to the third layer is provided by four keyboard shortcuts, numeric keyboard keys, several on-screen touch gestures, a haptic touch of a braille display, synthetic speech and bolded lines outstanding graphic elements (see Fig. 1).

Fig. 1.
figure 1

Third AR layer: elements and UI for detailed exploration of a function graph

Full size image

The third informative layer enables detailed, interactive exploration of complex formulas and mathematical drawings by the following elements: 1. Reading parts of the structures of formulas and descriptions of drawings introduced during their editing (by synthetic speech and a haptic touch of the Braille display), 2. listening to the monotonic sound generated while the user is touching lines of geometrical figures and function graphs (through touch gestures), 3. listening to the sound of variable monotonicity describing the function graph (through touch gestures). Additionally, the user is informed with short beeps on zero’s points of a function and long beep about surpassing the drawing area during the exploration with touch gestures.

An Example of Using the Third AR Layer

The students need to analyse the complex formula more thoroughly (see Table 2). For this purpose, they can use the third AR layer, thanks to which the formula structure becomes fully accessible. The students use the keyboard shortcut shift + ctrl + 1 to enter the formula structure and navigate, like on a tree, through this structure while hearing the reading of the focused parts of the formula. They use the arrow keys or touch gestures to read the formula elements. In this way, the students can build a formula structure in their imagination.

Table 2. An example of using the 3rd AR layer: immersive learning about the elements of the formula (reading and modification)
Full size table

The thorough exploration of mathematical drawings is done with the help of sound signals generated when sliding one’s finger on the touch screen with displayed graphics on full screen. The student enters the drawing with two keyboard shortcuts, to interactively explore or to passively recognise it. By moving his finger on the screen, the student hears various sounds representing objects contained in the drawing.

Thanks to the feature automatically supporting graphics creation process, parameters given during its construction, such as coordinates of points, length and width of the polygon or radius of a circle are transformed into texts describing the given object. This automatically generated description is read after the tap gesture when the sound generated along the line or at the point is heard. This automated drawing creation mechanism is fully accessible to a blind student, thanks to which he/she can create or modify mathematical graphics on his own. When the student is “immersed” in graphics by touching its elements, then thanks to the 3rd AR layer, he/she can also modify it on his/her own. Thanks to this, immersive graphics learning is implemented. Next, we present a detailed description of the three methods of recognising graphics in the third AR layer.

Method 1

When the student hears a sound while sliding finger on the screen, then as a result of the finger tapping gesture he/she will hear information with synthetic speech, relating to the encountered element of the drawing: “axis x”, “axis y”, <coordinate values> of the touched graphics elements, <text describing the figure’s elements>, <text of a comment>, <the comments and additional descriptions> in addition to those generated automatically are entered by the teacher or by the student while editing the drawing.

Method 2

Listening to monotonic sound while sliding the finger over the bolded line of a figure or a function graph. The displayed drawing line is specially thickened to keep the finger moving within its area.

Method 3

Listening to automatically generated sound with variable monotonicity, describing the function graph. The pitch informs the student about the Y coordinate. Students, knowing that the graph is always played from the smallest to the largest values on the X-axis, can build in their minds an image of the function graph they are listening to. Additionally, with particular sounds, students are informed about zeros of a function and on finger movement outside the drawing area. The elements and UI of the part of the third AR layer (for exploration of the function graph) are shown in Fig. 1.

2.4 Fourth AR Layer: Recorded Teacher Comments

The fourth layer, optional, contains information recorded by the teacher as hints, tips, or opinions in the form of static video (just sound) embedded in the content as a link to a resource placed on YouTube. This layer is supported by standard keyboard shortcuts that allow for opening links in a browser. It is up to the teacher whether to post comments and up to the student whether to use this form of teacher’s hints.

Thanks to this combination of AR reality providing the right tools, the students in their cognitive reality can enrich their mathematical knowledge.

3 Results

In the proposed method, the mathematical content can be independently, interactively, explored by students with visual impairment due to the addition of several layered AR, which is formed by subsequent information layers. The presented multi-sensual AR was implemented in the online environment EuroMath, in English and Polish, for independent learning of mathematics for blind and visually impaired students, also for the creation of mathematical content by teachers and students. The EuroMath environment consists of a WEB application for creating and exploring interactive multimedia mathematical content and portal supporting the repository of open educational math resources (OER) in the form .epub files [4]. OER consists of math content accessible to students with visual impairments (blind, with low vision) and methodological materials for teachers. Math teachers from Poland, the Netherlands, and Ireland have already created over 300 multimedia mathematical documents using the WEB application and uploaded them to the OER repository.

Mathematical educational resources (OER) are described in detail with metadata that classifies resources by mathematical subject, type of school, level of education, visual impairment. Students can search for materials helpful in learning mathematics using the search engine available in the repository. Moreover, they can also support cooperation in a group of students on a discussion forum available on the EuroMath portal.

EuroMath is an intelligent system supporting the learning process of students with visual dysfunction as well as teachers in creating educational materials:

  • In the EuroMath system, instead of several versions of a given document, for example, a work card, intended for the sighted student, the blind student and the low vision student, the teacher creates one universal version, which is adapted to the needs of each group of students, as a result of the application of conversion algorithms;

  • The EuroMath is teacher-friendly because: a) shortens the time needed to create universal materials by automatically generating and supplementing information about mathematical objects contained in the document (types of objects, the content of formulas and their elements, coordinates of graphic elements, zero points, and function graphs), b) enables the teacher to choose tools for editing and reading formulas and graphics, in extreme cases a blind math teacher (there are some) can create formulas and mathematical graphics available for each student according to their needs;

  • EuroMath is useful for blind and visually impaired students who can independently read and edit documents with formulas figures drawings and function graphs. The capacity to read is facilitated by the three described AR layers and the special UI. The possibility of interactive immersion in the environment of mathematical objects, which a student with visual impairment can “touch” and cause their change, is made by a set of developed algorithms for converting mathematical notation (such as UEB/MathML/UEB, AsciiMath/MathML/UEB, AsciiMath/MathML/AsciiMath), translation of MathML notation into semantic texts of the readout, translation of vector graphic inscriptions into sounds of constant and variable monotonicity and special UI adapted to the needs of the student;

  • Mathematical resources in the EuroMath repository allow each student to follow the appropriate educational path. So that all material from this path is tailored to the needs of the student in terms of methodology and technical availability.

EuroMath was developed as part of a project (2017–2020) under the UE Erasmus + program.

4 Conclusion

The presented multi-sensual AR was designed together with blind students and math teachers of the final technical classes of secondary schools. While this method has been developed with the participation of blind and visually impaired students and their math teachers, we can conclude that it has been accepted by potential users. It is currently being evaluated by mid-2020 by teachers with students in Poland, Ireland, and the Netherlands.

Due to the narrow target group, we can only conduct qualitative research. EuroMath valorisation was undertaken by mathematics teachers with their students from the only special centre for children with visual impairment in Ireland, from two, out of nine existing, special centres in Poland and from five schools run in the Netherlands by an expert centre for blind and low vision children. Since the outbreak of the COVID19 epidemic, when schools and universities have been closed, the interest in ICT learning supporting tools in the reverse class model has increased as well as the interest in the EuroMath system as a tool that could be useful in this model. The third special centre in Poland and one of the universities of technology in Ireland will also partake in the valorisation of EuroMath. Currently, EuroMath is being tested by 13 mathematics teachers and students: four from the Netherlands, four from Ireland, including three academic teachers, five teachers from Poland, altogether 12 educational units. The effectiveness of the AR + UI method increasing the availability of mathematical content implemented in EuroMath has been confirmed by teachers from Ireland and Poland. One centre in Poland is already using EuroMath in its ongoing operations. The above-mentioned University of technology has expressed interest in implementing the EuroMath system in its infrastructure. Teachers from the Netherlands also sent a preliminary positive opinion about the method of sharing mathematical content described in this paper.

In connection with the closing of schools, blind students from special centres began remote education, in which a visually impaired student is forced to become more independent in the learning process. To what extent our method has increased this independence in practice. We will be able to examine after the COVID 19 epidemic. It is worth noting that the students’ self-reliance is influenced not only by the AR + UI method itself but also by the quality and attractiveness of the materials prepared in the EuroMath application - work cards, exercises and tests available in the repository. To facilitate teachers’ work in this area, help materials in the form of instructional videos have been made available on the You-Tube channel dealing with the EuroMath system.