Keywords

1 Introduction

The distance we humans have extended our vision into space falls outside what we can see with our unaided eyes, as well as the sensory range of particular phenomena like sound, touch, and the many senses of the human body: the big and small, close and far, fast and slow, humid and dry, intense and soft, hot and cold, bright and dark… but we do, however, have the means to translate these non-experienceable things to the domain of our human senses through technologies and visualization. This is not that different from compensating for a missing capability within the human body. In the case of a disability, the need to interact with and navigate space may help adapt and compensate within the environment to orient and evaluate decisions and actions, yet the environment remains the same for people with and without disabilities.

Navigating and interacting with space to handle objects by changing place and position happens as a multimodal experience. Senses confirm and enhance each other to gain awareness and to reason and act on the information gathered by the senses. “Our world radiates out from our bodies, as perceptual centers from which we see, hear, touch, taste, and smell our world” [1]. From this perspective an experience is not about the senses themselves but rather the overall spatial construction of the experience enabled by the sensory signals that converge in the brain. Soundscapes, temperature variation, touch used to move about –to name a few–, all contribute to creating an internal model of the space, or objects in space around the body.

On the other hand, the natural experience of the world is the main referent to how we evaluate representation in any form: auditory, visual, tactile, and so on. Yet our representation devices have become such a big part of our daily lives that we forget how eerie they feel the first time we experience them with our senses, when the media used only partially recreates a multi-sensory experience: what the first people confronted with a silent film felt, for example, not having accurate sounds to accompany the images displayed on the screen, or even today, what people experiencing virtual reality devices describe as nausea, because of the contradictory information regarding motion being acquired by vision but not by the vestibular system: where the eyes see a landscape as if one was moving forward, yet the balance system in human ears do not sense any spatial displacement.

The difference between a purely visual printed piece or representation, and any experience that includes real time sensory input, calls attention to itself as a novelty. The challenge today is to evaluate sensory means of representation against natural experiences, where the senses’ bias are taken into account, like virtual architecture tailored to the senses, to fully utilize the capacity of human perception and attention to be meaningful. This biased experience of “(…) the nature of our bodies, the constrains on our perception, and the structure of our consciousness give prominence to the CENTER-PERIPHERY organization of our experienced reality” [1].

Working at the Adler Planetarium informs the insights mentioned here, since the experience of the Universe, derived from data collected by devices that extend our senses, is being tailored to our perceptual system in different ways, just like issues of perceptual disability are tailored to compensate for a given limitation. This approach is also related to the connection to nearby Electronic Visualization Laboratory, and the Virtual Reality work produced there which has evolved to consider the creation of virtual experiences as enhancers of perception in matters of extension of the human senses in courses such as Human Augmentics.

2 Perception and Cognition as a Personal Experience

Perception does not work as an isolated system within a person, but is part of a continuum of understanding that may trigger preconditioned automated responses. Perception may also increase a person’s awareness, prompting the need to elaborate new meaning from the experience. New meaning may be formed in order to produce a custom response, a future automated response, or even create new metaphorical constructs. In this continuum “Metaphor reaches down below the level of propositions into this massive embodied dimension of our being” [1]. The connection to the experience is thus not lost, but abstracted and revisited as needed.

When we think of the personal experience of space, whether it an experience of pose, displacement, or a motion pattern, perception works to acquire information to elicit a response that may be a physical action, or a concept, or both, that remains tied to that experience of space that allows us to “understand abstract purpose in terms of motion along a path” [1].

This relation to form connects to the posture of the person and is a way in which we internalize structures outside of our body, and bring the outside inside. But not all senses create the same separation between inside and outside. While visual perception is the main sense to negotiate decisions with the world outside of the body, sound, for one, has to come inside of the body to be acknowledged, and we can even hear the inside of our own body. In this manner, the “(…) sense of sight implies exteriority, whereas sound creates an experience of interiority” [2].

Taste may also suggest interiority, and other kinds of sensory input also have a conditioned assumption of spatial placement and distance as to how they can reach, besides a starting proximity that is located inside, in the boundary (i.e.: touch) or outside of the body as in the case of vision. The body, as the point of origin of perception, measures space and places an understanding of the structure of space in the experience. Memories of skiing, displacing the body at high speed in the soft snow, may be assembled from the perceived brightness, wind sensation including temperature, and vestibular tilt, whereas the experience of an earthquake may be assembled from rambling sounds that come from colliding objects at various distances and from various directions simultaneously, together with the body’s vestibular system struggling to aid in keeping balance against the moving floor.

3 Perception and Cognition as a Collective Experience

In contrast to personal experiences, what is standard about our perceptual system? Similar weight and reaction to gravity, similar leg height and arm’s length? There is a range of functionality of the human body with its senses and its dependence on the environment that can be shaped onto a collective understanding of how such a body relates to space through the coordinated input from senses. In turn, there is difference between people who are better at certain modalities [3].

The evolution of senses in different species are all traceable to adaptations to different environmental variables that affect survival such as obtaining food, protection from dangerous conditions, and ensuring continuity [4]. As a species we align our senses into one process, and in doing so we develop interaction patterns for handling and navigating. We could say that there is no thought or action detached from the body and the mind is “(…) part of an ongoing evolutionary process in which organisms seek to survive, grow, and flourish within various environments.” Moreover, mind is but an “emergent process, never separate from body. Thus, experience is a series of purposive bodily activities immersed in the ongoing flow of organism-environment interactions” [1].

It is in this series of interactions and emergence of the mind where there are no ontological gaps between various levels of functional complexity [1] From this perspective, organisms “develop what we call mind when they achieve levels of functional organization that make communication and shared meaning possible for them, thereby opening up a host of unprecedented possibilities for dealing with the life problems they encounter.” At this level the capability of shared meanings arise to “engage in various modes of inquiry and reasoning, and coordinate activities with other creatures who have minds, using symbols that have meaning for us” [1]. Thus, the mind is also externalized to connect with others as a collective mind or thinking process. But how similar is perception from one person to another? Perceptual development has been averaged in terms of defining learning disorders [5]. It is in those disorders that tracing missing meaning points to perception that is lacking or perhaps varies too much to fall within what constitutes collective understanding.

The design of interactive spaces that are tailored to an approximation of the collective understanding built from perception requires an understanding of this continuum. The collective understanding that emerges from the interaction of independent perceptual systems centered on emergent minds is required for the emergence of the collective. Automatic responses at the collective level are what we know as generic, that is, devoid of specific meaning that corresponds to a specific situation, but rather, a response that is general and remains hidden from consciousness.

A learning disability rooted in perceptual differences may also have parallels in social learning. What would account for a perceptual problem at that scale? The collective experience, average or not, may reflect back to the individual as a perceptual modifier of the understanding of space navigation and handling.

4 Disability and the Limits of Perception: A Comparison

We can learn about our capabilities from our limitations. Compensating for a loss may imply a translation of something usually sensed in a particular way to another available sense. In scientific visualization the translation of data to a perceivable form is a way to expand our existing perceptual system, and also a way of compensation. What part of one sense cannot be replaced by another sense? What is beyond the senses altogether? The translation of what is missing from the capabilities of the average person, or limited by the average set of senses, is nothing new. In both cases, a person needs to train her or himself to recognize the translated signal, correlate it with other sensory cues, and accept it as natural. In this regard the concept of Human Augmentics considers senses as an ecosystem where perceptual cues can communicate with each other to coordinate necessary assistance [6].

In the case of astronomy, the farther one sees, the more we see the same thing. Perception of distant phenomena causes the point of view of humans to converge, since we all humans are looking out from the same planet, albeit in different directions depending on where we are on the planet’s surface. Our experience of the Universe is a collective experience, enabled by instruments such as ground and space telescopes. These instruments collect data that is then scaled and/or retimed to show forms where constitutive parts relate to each other, and change, and become part of our human experience.

There is also the question of whether vision is defined by the eyes or the brain. Research in sensory integration reveals that when there are issues regarding how the senses connect to each other, spatial perception can be impaired. When senses that work in tandem to assess space cannot relate to each other, neglect may result because of this impairment in integration. In a situation of neglect a person may not see things on the right side of the visual field, for example [7]. Another visual issue is the inability to see stereoscopically. It is common knowledge that about 12 % of the population have issues with binocular vision where the sight of both eyes does not get integrated into an understanding of visual depth, although other perceptual cues for visual depth such as motion parallax are still available.

Senses support each other to extend the experience. This is an important aspect of disability compensation where ultimately we seek to handle spatial interaction, no matter the combination of senses that inform it. A well-known issue with VR devices is that when a user moves through virtual worlds there may be a conflict with the vestibular system. As mentioned before, the vestibular system may not be able to match the speed at which the image is changing due to the lack of motion expected in the balance system located in the ears. Similarly, in planetarium domes, tilting imagery has to be done at a discrete speed so as to not unbalance viewers.

Lastly, information about the Universe that is being translated to visible light, to noticeable speed, and in a scale that matches the range of human perception, is not direct, but also a translated or mediated experience. We notice the edges of perception when we cannot sense anything anymore, but require aids such as telescopes, cameras, or filters that can capture what human vision, scale and timeframe cannot. Yet the need for this spatial and temporal translation is not perceived as a disability, but a gain, because by extending our senses, we are extending human understanding. This gain, however, may still reflect into an experience of spatial and temporal reasoning, just as a natural experience would.

5 Sensory Integration to Understand and Interact with Space

The perception and experience of reality and designed reality is multi-sensory, and there is a premise in that triggers of involuntary memory are produced by a minimum of two senses [8]. The design of graphical interfaces and visualization development relies, whether consciously or not, on saccadic eye motion (the way human eyes wander about an image to build a model of it in the mind) all the way to proprioception in literally navigating space, for the purpose of visual flow. Conditioned areas in the brain that are devoted to multimodal perception affect the integration of these kinds of motion since visual navigation is integrated with other senses for assessment. For example, parts of the brain triggered by multimodal perception are slightly different than the sum of the independent sensory modalities [3].

On the other hand, sensory integration relies on attentional modulation, that is, memory or conditioning from previous experiences that inform how worthy an event is of attention [9]. Senses interact with biases recorded from past experiences in the brain. So in a way, the human senses not only negotiate among each other to process what is perceived, but sensory information is itself processed against memories [10].

Ultimately, these experiences of managing visual flow in the actual world model the experience of interaction with visualization devices. In turn, devices are taking over part of the role of memory in dealing with the environment, which affects the decision on what is worth thinking about and what is just a cue for resource acquisition [11]. In scaling spatial and temporal data, the cues for understanding space and duration can be greatly enhanced by engaging as many senses as possible. Spatial depth already suggests the understanding on how proprioception would work if one were able to move around a given environment. There are connections among our senses that inform the understanding of what we see, and incorporating other senses, even those that have no current stimulus, would reinforce the existence of what is being perceived.

6 Ranges of the Scale in the Human Body

For the human body, our senses have evolved in particular ways to relate to space and time, and also to extend its spatial reaches and understand the passage of time, invest in memory and make predictions, based on information. This is the human Umwelt, the world as sensed by the human perceptual system [4].

Devices such as telescopes and microscopes extend human vision to the very distant and the very small, but in scientific visualizations that information can be stretched in terms of range in order to see the large and the small together, slowed down to see what happens too fast, and compressed in order to see what happens too slowly. In other words, in order to see patterns, not only size and speed matters but range. Some patterns are out of range because the things being connected may be too small and too far from each other to be seen at once. Not only things are bigger and smaller than human senses can experience, but the thresholds where they die off –if we assume that the perceptual human range is maintained– may result in nothing to be perceived. This because salient qualities may have become too small and their distance from each other too large, perhaps, to the point nothing is visible or perceived by other senses either.

The human body has evolved to structure imagination within the ranges of its senses, although imagination can extend those senses through associative and combinatorial means. As Johnson puts it, “Imagination is our capacity to organize mental representations (especially percepts, images, and image schemata) into meaningful, coherent unities” [1]. Johnson accounted from Kant’s work on the role of functional imagination to achieve meaning, and being able to understand, reason and communicate. The development of scientific visualizations fits within this task like an operation as described by Aristotle, where imagination is more of a mechanical operation tied to the senses rather than a creative process [1].

Furthermore, besides association, “Kant describes this ultimate unifying structure of consciousness as an operation of imagination, because it is a synthesizing activity that gives the general structure of objective experience as such” [1]. This synthesizing activity used in the development of representations or scientific visualizations belongs to the discipline of an artist who can imagine multiple scales simultaneously, and understand structure at various scales and ranges: a composition at many levels where weight can also be distributed and modulated in all of them to converge in the human perceptual system.

7 Extending Ranges of Experience

We live in extended bodies today, from communication devices that allow us to reach distant people and places, and information from the past and the future, to air conditioned cars that enhance speed while keeping us in a comfortable climate bubble, and virtual experiences that allow us to explore what is beyond our natural ability to perceive what exists beyond our senses, sensory ranges, and our spatial and time constraints. Indeed any media extends: books as assistive memory devices, photography as visual record, film as spatial and temporal memory as well. Remote cameras extend real time presence. Virtual Reality devices and spaces for collaborative and remote endeavors are becoming more common as the need to coordinate and extend experiences involving scientific or fantastic data increase.

“Our lives are filled with paths that connect up our spatial world (…) certain paths exist, at present, only in your imagination, such as the path from Earth to the nearest star outside of our solar system” [1]. Interacting with multi-scale datasets creates new paths and new challenges for multisensory integration. The very action of changing scales is in itself non intuitive, either because a constant speed tends to appear too slow in the larger end of the scale, and too fast in the shorter end of the transformation. But increasing size and speed together help keep the experience consistent among the senses. Perhaps they work like “Preconceptual gestalt forces as constraint of coherence” [1] in the realm of visualization as well.

Representations of scale in science fiction cartoons or movies may make us forget that our perceptual system has limits. When giant monsters roam about, their ability to even pay attention to small humans tends to be over stated. In the movie “Monolith Monsters” there is no reason to run from rocks if they are not falling or even moving the slightest. What does that mean for metaphors of outer space? Of the unreachable? Of course movies seek to elicit emotions rather than spatiotemporal paths per se. This is where interaction capabilities have been utilized to fill in the evoking of cross-sensory relationships that fit a change of orientation or speed. Like target practice with arrows, the general orientation of the head serves to calibrate target. This requires multimodal integration. Multimodal relationship bias respond to tracking because space needs to be handled, especially space in the universe. Interaction is important in evoking senses other than visual, in this case, touch evoked by handling, since non-informative vision affects haptic performance [12]. Building a multi scalar visualization is an interesting process where what was presented before as a graph or an abstract model is now being designed as an immersive spatial experience bound to its own set of constraints both defined and evoked by the medium.

8 Reconfiguring the Perceptual System

Looking beyond our human senses has afforded humanity knowledge about phenomena that does not appear immediate to the human perceptual system. Extended senses still evoke the connection to other senses that have been conditioned by the life we live in the environment of our planet however, reconstructing a multimodal experience that reinforces existing structures of the mind or perhaps creates new ones. The ultimate goal remains the creation of a spatial model of being in space that ‘makes sense’ of all sensory information available. This model would rely not only on each sensory input, but on the synergy created over time by simultaneously triggered senses in such a way that they can reinforce each other. This synergy is what Mark Johnson described as where “there is an inferential structure in the epistemic domain that is tied to gestalt features of our experience of physical force and barriers” [1].

9 Conclusion

The experience of the Universe, derived from data collected from devices that extend our senses, is being tailored to our perceptual system in different ways, just like issues of perceptual limitations are tailored to compensate for missing capabilities.

Perception does not work as an isolated system within a person, but is part of a continuum of understanding that may trigger preconditioned automated responses. Perception may also rise to the person’s awareness, prompting the need to elaborate new meaning from the experience. New meaning may be formed in order to: produce a custom response, a future automated response, or even create new metaphorical constructs. There is a range of functionality in the human body with its senses that can be shaped into a collective understanding of how the body relates to space through the coordinated input from various senses.

Perceptual development has been averaged in terms of defining learning disorders. The collective experience, average or not, may reflect back to the individual as a perceptual modifier of the understanding of space navigation and object handling. Senses also support each other to extend the experience. This is an important aspect of disability compensation where we ultimately seek to understand the environment, with any available senses.

Sensory integration relies on attentional modulation, that is, memory or conditioning from previous experiences that inform how worthy an event is of attention [9]. Senses interact with biases recorded from past experiences in the brain. In a way, senses not only negotiate among each other to process what is perceived, but sensory information is itself processed against memories [10]. Ultimately, these experiences of managing visual flow in the actual world model the experience of interaction with visualization devices.

For the human body, senses have evolved in particular ways to relate to space and time, and also to extend their spatial reaches. This evolution has shaped us up to understand the passage of time and invest in memory and manage prediction models based on this information. The human body has also evolved to structure imagination within the ranges of its senses, although imagination can extend those senses through associative and combinatorial means. This synthesizing activity used in the development of representations or scientific visualizations belongs to the discipline of an artist who can imagine multiple scales simultaneously, and understand structure at those various scales and ranges: a composition at many levels where weight can be distributed and modulated across all of these levels to converge in the human perceptual system.

We live in extended bodies today, from communication devices that allow us to reach distant people and places along with information from the past and the future, to air conditioned cars that enhance speed while keeping us in a comfortable climate bubble, and virtual experiences that allow us to explore what is beyond our natural ability to perceive what exists beyond our senses, sensory ranges, and our spatial and time constraints.

Cinematographic techniques seek to elicit emotions rather than spatiotemporal paths per se. Beyond movies, interaction capabilities have been developed to evoke relationships that fit a change of orientation or speed that can be used to enhance understanding of information. In designing interactive experiences, multimodal relationships respond to tracking because the act of maneuvering objects or the environment greatly aids in orienting ourselves in space, especially space in the universe. For example, the use of various interactive devices including 3D prints of asteroids changes the appreciation people have of models seen in movies. Interaction is also important in evoking senses other than vision, for example, touch evoked by handling, since non-informative vision affects haptic performance [12]. Building a multi scalar visualization of the Universe that we know about from data is a complex process where what was presented before as a graph or an abstract model is now being designed as an immersive sensory experience that compensates for our human constraints.