Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation, Healthcare, and Inclusive Well-Being
A sensing and response system that sources physiologic human data (typically using sensors on- or off-body) mapped to selectable content (typically digital) as feedback to self-inform participants of their input consciously, subconsciously, or subliminally. Often used in treatments to teach patients to modify specific physiologic functions.
Application of game elements within a typically nongame context. Contextually, in this case [re]habilitation, healthcare, and interactive performance art, targeting participant motivation, immersion/engagement, and playfully rewarding positive experiences offering inclusive well-being. (Regaining skills, abilities, or knowledge that may have been lost or compromised as well as helping disabled people attain, keep, or improve skills and functioning for daily living – in line with the Convention on the Rights of Persons with Disabilities (CRPD).).
- Gesture Motion
Movement of the limb, torso, or whole body within a sensing space.
Contextually specific referring to an author-conceived bespoke accessible, inclusive, and interactive multimedia environment targeting, through adaptive personalization, creative expression, and playful interactions alongside user experiences of enjoyment, fun, and entertainment. Utilized as an alternative intervention concept in healthcare and rehabilitation to improve participation in treatment programs. The concept is informed from the author’s audiovisual art.
- Video Games
Computer graphics and sound elements comprising a “virtual environment” within which players interact (via an input device) with typically animated objects displayed on a monitoring device for the sake of entertainment.
As Tolstoy stated in “What is Art?” (Tolstoy 1995 ) – Art is a human activity consisting in this, that one man [or woman] consciously by means of certain external signs, hands on to others feelings he has lived through, and that other are infected by these feelings and also experience them. In the text he states how art is a form of consciousness, framing in so doing the essential role of art as a vehicle of communication and empathy.
On a more recent note than Tolstoy, Grau (2003) posited: …ultimately, it is the intellectual vision, transposed into the work step by step with technology as its reference, that remains the core of a virtual work of art.
This article introduces a body of work where the catalyst is creative expression and playful interactivity. The author’s background as an artist is prevalent in how empowerment via embodied interaction utilizing digital technologies (predominantly sensor-based [on-body/off-body] biofeedback mapped to digital multimedia [auditory, visual, robotic stimuli, etc.] and analogue content [video feedback, vocals, etc.]) was identified as a means to supplement traditional intervention in specific healthcare treatment programs and (re)habilitation.
Within the work a commercial industry start-up was realized from the author’s research, as well as international and national funded projects, and global acclaim as, e.g., plenary keynotes at leading international conferences, and more.
This contribution is focused upon sharing how in the 1990s, for approximately a decade, computer graphics were created as gesture-based interactive games under the author’s gamification (including social interaction, creative expression, and enjoyable play) approach to healthcare and rehabilitation intervention. The core of the strategy is a catalyst fun experience from within an openly adaptive interactive environment that can be tailored for each participant profile and the treatment program goals. Most recently the work has realized a series of publications under the theme of “Technologies for Inclusive Well-Being.”
The work originated as an alternative contemporary avant-garde “human-at-center” art-related creative expressive form (i.e., “human-as-art”). Following numerous proof-of-concept and feasibility testing, early apparatus and method were realized in performances and exhibitions with positive reactions. In other words, the author-developed sensor-based bespoke systems were primarily explored within the author’s work of own stage performances at numerous national and international festivals (including televised performances). These explorations included instances where he directed and produced (e.g., Cultural Paralympiad [Atlanta 1996 at Rialto Theatre]; Cultural Olympiad and Paralympiad [Sydney 2000 at Homebush arena]; European City of Culture [Copenhagen 1996 at Arken MoMA and Avignon 2000 at Cafe Nine]; Danish NeWave New York at Gershwin, 1999; Scandinavian Museums of Modern Art exhibition tours 1995–1999; and other “art-related” settings, e.g., Roskilde Festival 2000 and more). All instances were targeted as research resulting in ongoing learning of system limitations, potentials, and possibilities toward the envisioned applications beyond solely traditional “art” forms. In other words, in the majority of cases – for example, inclusive or adjacent to the above listed events – demonstration workshops, hands-on tutorials/seminars/symposiums, or other accessible showcase forms were arranged at the author’s initiative to present the “alternative art” application, i.e., in healthcare, rehabilitation, and therapeutic training intervention. Such additional events offered increased research and learning opportunities including reviews, appraisals, assessments, and evaluations across disciplines, nationalities, and end users.
Bespoke Systems Overview: Leading to Patent - see Brooks and Sorensen (2005)
Overviewing and simplifying, the systems consisted of on-body and off-body systems that were experimented having differing biosensing profiles. Thus input was sourced ranging, for example, from inner-body micro-electrical signals, through limb or whole-body gestural position and motion dynamics, to spatial environmental signals where human occlusion or signal generation results in system input. Thus, human gesture attributes (proprioceptive, kinaesthetic, and related dynamics) and human state (being, emotions, etc.) acted as input. Sourced input signals are routed to selectable software for mappings (scaling, filtering, etc.) to impact feedback responsive content (typically digital). This process routing of the data signals is managed to align desired relationships, i.e., interactive/reactive toward a goal of achieving flow, aesthetic resonance, self-agency, efficacy, and related idiosyncratic human attributes via afferent/efferent neural feedback loop closure. This causal loop closure is achieved through the process of optimally tailoring system attributes to human attributes, e.g., where the designed challenge is matching user satisfaction and sense of achievement. Individual end-user profile assessment can be either formal (with therapeutic input to realize targeted preset steps) or improvised (through system operator’s – usually the author – experiences, so more impromptu adjustment of change parameters) to impact the system session design as experienced by a user. See Brooks Patent US6893407B1 on method and apparatus
The designed embodied interaction considered intent and non-intentional input as well as conscious and unconscious innate attributes. Through these means control and non-control have been experimented Brooks (2004, 2018).
Self-reflections and self-critique included from first-person and third-person experiences through which the system developed as a substantiated open and adaptive entity. As such the system (apparatus and method) enabled adoptions of various technologies as they appeared as both input interface apparatus and content toward optimizing a used model for treating a range of patients including those “born with” impairment or those who had “acquired” impairment either through accident, incident, or disease.
Experience as Product
A goal behind the “human-as-art” performative inquiries with the system was to realize a new instrument or tool that could supplement in and across rehabilitation and healthcare contexts. The concept was to explore creativity and play as motivational human modes attempting to make the experience of treatment/training more enjoyable, fun, and stimulating to participate within and less mundane, tedious, and boring. Once the system had reached certain maturity, further reflections and critiques resulted in system improvements that aligned with external professionals who evaluated from a formal and professional therapeutic perspective. Over the many years, the family and friends of users also evaluated – albeit in a more non-formal/informal context.
A motivational intervention (in-action) model and an (on-action) evaluation model were developed and published to support practicing professionals and/or home use by families and carers or even in self-use Brooks and Petersson (2005).
Exploring Nuances of Differences
An ongoing vision was of creating, utilizing, and exploring digital technologies to explore aspects of interactions such as nuances of differences that may be apparent through dysfunction compensatory requirements (e.g., an augmented sense of hearing/touch if a person cannot see). An evolving interest from this vision is in the exemplification and integration of such finite sensorial differences and how they can be represented and utilized within the art-related works for wide-audience education and enjoyment. In other words, and for example, how an “artist” with heightened sensory attribute (through loss of other sense) can represent such a sensory nuance so that audience members who are fully sensory loaded (thus, potentially, not as highly nuanced in a specific sense) may appreciate the art and be provoked to reflect on their educating of their evoked sense via the artwork.
This compensatory approach questioning comprehension of alternative channelling to augment training and other benefits from creatively expressing was apparently original within rehabilitation fields. For example, the applied research with acquired brain-injured patients in a clinical setting questioned how a sense of proprioception (a participant’s body sense of its parts and relative location to its other parts and the effort exerted in motion often related to balance and/or neglect) that was damaged could be “trained” through a patient listening to or seeing where their relative position of their own parts of the body are instead of feeling it. Additionally, patient dynamics of kinesthetic awareness of the position and movement of their parts of the body by means of their proprioceptors is targeted through programmable thresholds in the digital content algorithm according to a patient profile. Interactive computer graphics were thus engaged, alongside other digital content, as a means to visually inform a user of system input.
Afferent/Efferent Neural Feedback Loop Closure
Afferent/Efferent Neural Feedback Loop Closure: Proprioception and kinesthetic awareness are key aspects of the concept presented herein.
Literature informs that the central nervous system (CNS) receives sensory stimuli as (afferent) impulses external to the body. It then sends appropriate (efferent) instructions to a person’s muscles and joints on how to react. The brain also receives some messages that cause the body to react unconsciously. Proprioception is a term referring to the internal messaging (the central nervous system) driving and controlling motion actions. Proprioceptors are sensors in human joints, muscles, and fascia, providing information needed to produce coordinated movement. Kinesthetic awareness refers to our ability to navigate space and the awareness of how we move. Kinesthetic awareness and proprioception work as partners to get us through the movements of our lives from the inside and the outside of the body. Muscle memory is a kinesthetic concept. So many things that we do without thinking – such as walking, whether we do it correctly or not – are kinesthetic experiences based on proprioception, which provides the awareness of our joints and body in space. Proprioception and kinesthetic awareness decrease after an injury – although, your brain will still have the skills stored within, so it is easier to relearn them. Even though your strength may come back easier, you will still need to spend time improving your proprioception and kinesthetic awareness so that you can fully recover.
By employing a system that responds to motion within an invisible space that can be controlled for data inhabitation and also mapping to digital content as stimuli offers opportunities in learning through feeling such as a sense of body position, muscle movement, and weight as felt through nerve endings.
Cognitive aspects are also involved.
Technologies for Inclusive Well-Being
Within the ongoing research, the art informs the design and intervention in rehabilitation in a cross correspondence such that the intervention also informs the art. This is aligned with reconceptualization, reframing, and cross-domain mapping as a bilateral approach that has been found effective in developing the research to the next level; however, that is not elaborated herein as it is a subject of another publication. Suffice to say that holistically, the research targets societal impact in (re)habilitation and healthcare under an umbrella titled “Technologies for Inclusive Well-Being” under which a number of publications have resulted with the author as lead editor. The next section introduces the author’s gamification intervention approach.
Computer Feedback Training Under Gamification Approach
Near the time when biofeedback was being explored via worn sensor systems, audiovisual computer feedback and a series of robotic light devices were applied under a gamification approach to training in (re)habilitation and healthcare as outlined in the previous section. The purpose of this approach to supplement traditional approaches in intervention within training treatment programs was to engage the participant to enjoy the training (vs. mundane, boring, and repetitive exercising without any self-reflective feedback that informs of progression).
Fun, Play, and Creative Expression Leading to Aesthetic Resonance
In this simple exercise, it was clear how a tangible outcome meant so much for the participants. When using auditory feedback, there is no tangible outcome unless a recording is made and given on USB stick or download. The design was thus a success for participants and the organizing staff who attended with the groups. On return 4 months later for a conference, the leader of the elderly home for mentally challenged/dysfunctional attendees of the workshop informed that the institute organized an exhibition and oral presentation event so that the elderly could tell their own stories about creating their paintings. She explained how their motivation and inspiration had been stimulated to a highly positive degree such that their stories, unusually remembered compared to their other living detail, detailed aspects that her staff did not realize. She explained that it was a moving experience for the families who attended.
Soon after the mid-1990s, investigations of interactive animations and gamification within the SoundScapes research were conducted. This preceded the Personics interactive computer graphic environment that is introduced next.
Experiments within the research led to a first simple “game” as an animation of an airplane that could take off, fly, and land with control via unencumbered gesture (i.e., no mouse, joystick, or keyboard). The reader should be aware that this experimental use of games controlled with gesture technology was before the pervasiveness of games and gesture control peripherals in the early 2000s.
The simple images on the following pages indicate the further explorations of basic interactive environments developed with the software Macromedia Flash (later Adobe Flash) as game content under the commercial company Personics as animated computer graphics as gamification in (re)habilitation therapy. The development was across the two funded aforementioned national and international projects having similar goals. In one case, the development followed input from the Humanics (the Danish government project) research team from the Center for Rehabilitation of Brain Injury [CRBI], this being where the clinical location of patient training was based at the University of Copenhagen. The team was working with acquired brain injury patients. Author-led roles in the project included conceptualizing, iterative design, leading intervention sessions, and communicating proposed designs and refinements with the development team, alongside testing and troubleshooting prior to sessions. The research personnel included a neuropsychologist, a psychologist, a cognitive psychologist, and a physiotherapist that worked under the Danish government Erhvervsfremme Styrelsen (Danish – translated as business development agency) who funded the research project titled Humanics for 9.5 million Danish Kroner (DKK). Another Danish government body, named Satspuljen, funded the project for an additional one million Danish Kroner (DKK).
Parallel to this project was the aforementioned international project. The background of this project is the research resulted in a European Union probe (under the European Network for Intelligent Information Interfaces – www.i3net.org). This was a funded project titled “The World Is As You See It” (TWIAYSI) – with the University of Bristol and a Swedish partner. TWIAYSI was developed into a European funded Framework V IST Key Action 1 project supporting the program for applications relating to persons with special needs including the disabled and elderly titled CAREHERE (Creating Aesthetic Resonant Environments for Handicapped, Elderly and Rehabilitation) Funding was approx. €2M.
Personics was invited by the author to participate in both projects.
Under the sponsorship of IBM at the World Congress for Physical Therapy (WCPT) in Yokohama, Japan, the author presented his research paper titled “Virtual Interactive Space (V.I.S.) as a Movement Capture Interface Tool Giving Multimedia Feedback for Treatment and Analysis” (Brooks, 1999).
Approximately a decade later, Hagedorn and Holm’s (2010) independent randomized intervention study questioned traditional training versus computer feedback training (system resulting from author’s research). This is reported in the European Journal of Physical and Rehabilitation Medicine where results state impact gains of up to 400% illustrate potentials from using selected games in Virtual Interactive Space as published in Brooks’ 1999 paper. The 2010 investigation is introduced after the next section that presents the simple interactive computer graphics used.
Personics Computer Graphics
The following images represent the gameplay graphics. Notes attempt to describe the gameplay and target in the therapeutic sessions with bmp screenshots depicting start screen and interactions.
The fourth game used in the training was where balance controlled the position of an animated empty basket to catch images of fruit falling from a tree. Healthy fruit was collected, while rotten fruit was not. Each incremental level had increasing speed of fruit falling.
Eber (1997) reflects how a work of art, according to Tolstoy (1995 ), is sincere, and it transmits feelings through lines, colors, sound, or words. The feelings embedded in the imagery start with the creator and the creative process. The work may take any form, but to be art, the object, idea, or installation goes beyond the physical and contain some form of human experience. Art may be created with any tool, as long as the artist rises beyond that tool into an experiential realm. Many have debated the existence of the creative domain with the computer art medium, especially virtual environments (VE). With the tools to create a VE, the artist will learn a new technology that may influence the nature of and how she reaches the creative level (see Eber 1997).
According to Eber (1997), in addition to the acquisition of new information, the artist who chooses to work with VEs also has a new set of aesthetics to consider, as the final work of art is wholly different from that using any other medium. Contrary to the concepts expounded in the popular media, a VE art installation can be more than a display arena for the art of others (e.g., Picasso) or a “shoot 'em up” computer game. It can be a work of art in and of itself, one that requires of the artist the same level of abstraction into the spirit of creativity as any traditional medium demands. How and at what point does the creative process happen for a VE artist in a world of computer peripherals and code?
Further Eber (1997) states how an art installation is a work of art that goes beyond an object that exists on a wall or behind a glass but encompasses an infinite number of artistic possibilities including alternative presentations, environmental constructions, multisensory stimulation, viewer interactivity, and theatrical performance.
This text presents a historic perspective of the research alongside a review of the basic computer graphics that resulted. The purpose of the text is to share the narrative while attempting to inspire next-generation researchers. The goal to inspire targets championing others to persevere against adversity, often when being mocked when pursuing one’s original concept, in this case a concept that resulted in national and international funded projects, a patented method and apparatus, commercial industry product, and company start-up.
Early studies explored solely auditory stimulus as a feedback to user input. Initially the means of input was hardware-based rocker control pedals as typically used by musicians for altering instrument output, e.g., guitar. Subsequently, user biofeedback signals were sourced either via on-body or off-body sensing interfaces with differing profiles. Such profiles have increasingly advanced over the years of the research such that original interfaces are no longer viable when compared to affordable and available computer game peripherals and camera-based solutions.
However, the “communication method and apparatus” patent has been referenced 16 times, including 12 by the patent examiner.
A further example of impact is an independent investigation of product resulting from this research in a randomized intervention study by Hagedorn and Holm (2010) comparing conventional balance training against computer feedback training with older people. The randomized controlled 12-week intervention trial was designed on pre- and post-training evaluations conducted on 35 outpatients of a geriatric falls and balance clinic. Responsive computer graphics, with a variety of selectable game themes, responded to input motion sensors.
Results were reported of 400% improvement in specific performance; however, in the author’s opinion, the industry is still lacking behind in training trainers to fully optimize such results within a wide range of interventions so as to benefit societally via transfer to activities of daily living (ADL).
This article shares insight of avant-garde art with societal impact that has been recognized by third-party researchers and educators as pioneering the use of digital technologies with differently abled. A focus has been on illustrating the simple computer graphics used as mappable content to give feedback selectable from an array of stimuli. This text informs how the healthcare rehabilitation informs the art and correspondingly how the art informs the healthcare rehabilitation.
The use of multimedia responsive feedback to human input indicates how art (creative expression) and play (enjoyment and fun) have an important part to play in both physical and cognitive therapy (re)habilitation. While extended range and dynamics of motion (body, limb, etc.) are quantifiable, it is also evident through the comprehension of designed interactions by profound and multiple learning disabled (PMLD) how the concept has potentials beyond what has already been reported.
To sum up and in accord with these conclusions, it is important to point out how it is increasingly evident that improved systematic evaluation is needed in this field to define the specific use benefits of (computer graphics as) video games in healthcare and rehabilitation intervention. Aspects of such evaluation are proposed as incorporating increased duration and control of trials with improved measurements (randomization, blinding, etc.) and consistency of measurement tools across investigations including beyond actual treatment programs to embrace impact on activities of daily living (ADL). However, innate to such a proposal are the ongoing challenges of individual human differences that many may consider immeasurable; yet, for improved research and impact comprehension, it is important to target optimized research validity and reliability in order to advance and educate.
The figures (5 – 14) are from the author’s own archive from employment. All efforts to get permission have not been responded upon and it is understood that the company was closed shortly following the author’s departure and copyright ownership is not listed for these images. Acknowledgement made in this chapter for the authors of the images created under the company Personics who do not name or credit authorship.
- Brooks, A.L.: Body electric and reality feedback loops: virtual interactive space and entertainment. In: Proceedings 14th International Conference on Artificial Reality and Telexistence (ICAT 2004), pp. 93–98. Advanced Institute of Science and Technology, Seoul (2004)Google Scholar
- Brooks, A.L.: SoundScapes: The Evolution of a Concept, Apparatus and Method where Ludic Engagement in Virtual Interactive Space is a Supplemental Tool for Therapeutic Motivation. (PhD) Institut for Arkitektur og Medieteknologi. Aalborg University, Denmark (AD:MT, Vol. 57) (2011)Google Scholar
- Brooks, A.L.: An HCI approach in contemporary healthcare and (Re)habilitation. In: Norman, K., Kirakowski, J. (eds.) The Wiley Handbook of Human Computer Interaction, vol. 2, pp. 923–943. Wiley, New York (2018)Google Scholar
- Brooks A.L., Petersson, E.: Recursive reflection and learning in raw data video analysis of interactive ‘play’ environments for special needs health care. In: Proceedings, IEEE HEALTHCOM 2005, Enterprise networking and Computing in Healthcare Industry, Busan (2005)Google Scholar
- Brooks, A.L., Sorensen, C.D.: Communication method and apparatus. US Patent 6893407 (2005)Google Scholar
- Grau, O.: Virtual Art: From Illusion to Immersion. MIT Press, Cambridge (2003)Google Scholar
- Hagedorn, D.K., Holm, E.: Effects of traditional physical training and visual computer feedback training in frail elderly patients. A randomized intervention study. Eur. J. Phys. Rehabil. Med. 46(2), 159–168 (2010)Google Scholar
- Tolstoy, L.: What is Art? (Translated by Richard Pevear and Larissa Volokhonsky). Penguin, London (1995 )Google Scholar