Skip to main content
SpringerLink
Log in

Fundamentals of All the Realities: Virtual, Augmented, Mediated, Multimediated, and Beyond

  • Chapter
  • First Online:
Springer Handbook of Augmented Reality

Part of the book series: Springer Handbooks ((SHB))

Abstract

This chapter introduces the different “realities”: virtual, augmented, (deliberately) diminished, mixed, mediated, multimediated, and phenomenological. The contributions of this chapter include (1) a taxonomy, framework, conceptualization, etc., of all of the “realities”; (2) a new kind of “reality” that comes from nature itself, which expands our notion beyond synthetic realities to include also phenomenological realities; and (3) methods of using phenomenological reality as a means of visualizing as well as understanding hidden phenomena present in the world around us. VR (virtual reality) replaces the real world with a simulated experience (a “virtual” world). AR (augmented reality) allows the real world to be experienced while at the same time, adding to it, a virtual world. Mixed reality provides blends that interpolate between real and virtual worlds in various proportions, along a “virtuality” axis, and extrapolate to an “X-axis” defined by “X-Reality” (eXtended reality). Mediated reality goes a step further by mixing/blending and also modifying reality, including, for example, deliberate diminishing of reality (e.g., a computerized welding helmet that darkens bright subject matter while lightening dark subject matter). This modifying of reality introduces a second axis called “mediality.” Mediated reality is useful as a seeing aid (e.g., modifying reality to make it easier to understand) and for psychology experiments like George Stratton’s 1896 upside-down eyeglasses experiment. Multimediated reality (“all reality” or “All R”) is a multidimensional multisensory mediated reality that includes not just interactive multimedia-based “reality” for our five senses but also includes additional senses (like sensory sonar, sensory radar, etc.), as well as our human actions/actuators. These extra senses are mapped to our human senses using synthetic synesthesia. This allows us to directly experience real (but otherwise invisible) phenomena, such as wave propagation and wave interference patterns, so that we can see radio waves and sound waves and how they interact with objects and each other, i.e., phenomenological reality. Moreover, multimediated reality considers not just multiple axes in addition to the X-Reality axis but also that the origin of the axes exists at zero sensory stimuli. In this way, we can account for various (virtual, augmented, etc.) realities in a sensory deprivation float tank. Consider, for example, wearing an underwater VR headset while floating in a sensory deprivation tank. Then consider an augmented reality headset while floating in an immersive multimedia pool. Consider Internet-connected immersive multimedia water therapy and music therapy pools that allow multiple users to play a hydraulophone (underwater pipe organ) while meditating together in a hydraulically multimediated collective. These are examples of what we call “fluid (user) interfaces,” and they fall outside the range of any previously existing “reality taxonomy” or conceptualization.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 309.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 399.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Albright, W.F.: The Proto-Sinaitic inscriptions and their decipherment. Harvard Theological Studies (1966)

    Google Scholar 

  2. Anstis, S.: Visual adaptation to a negative, brightness-reversed world: some preliminary observations. In: Carpenter, G., Grossberg, S. (eds.). Neural Networks for Vision and Image Processing, pp. 1–15. MIT Press, Cambridge (1992)

    Google Scholar 

  3. Artaud, A.: The Theater and Its Double, 159 p (1958). First published reference to “la realite virtuell” (“virtual reality”)

    Google Scholar 

  4. Azuma, R.T.: A survey of augmented reality. Presence Teleop. Virt. 6(4), 355–385 (1997)

    Article  Google Scholar 

  5. Coleman, B.: Using sensor inputs to affect virtual and real environments. IEEE Pervasive Comput. 8, 3 (2009)

    Article  Google Scholar 

  6. Cytowic, R.E.: The Man Who Tasted Shapes: A Bizarre Medical Mystery Offers Revolutionary Insights into Emotions, Reasoning, and Consciousness. G. P. Putnam’s Sons, New York (1993)

    Google Scholar 

  7. Dolezal, H.: Living in a world transformed: Perceptual and performatory adaptation to visual distortion. Academic Press, Chicago (1982)

    Google Scholar 

  8. Euler, L.: Tentamen explicationis phaenomenorum aeris. Commentarii academiae scientiarum Petropolitanae, 347–368 (1729)

    Google Scholar 

  9. Ganascia, J.-G.: The generalized sousveillance society. Soc. Sci. Inf. 49(3), 489–507 (2010)

    Article  Google Scholar 

  10. Gershon, N., Kapralos, B., Mann, S.: Games, entertainment, and media for life, health, and community: IEEE GEM 2014. IEEE Consum. Electron. Mag. 4(3), 67–72 (2015)

    Article  Google Scholar 

  11. Haller, M., Landerl, F., Billinghurst, M.: A loose and sketchy approach in a mediated reality environment. In: Proceedings of the 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, pp. 371–379. ACM, New York (2005)

    Google Scholar 

  12. Janzen, R., Yasrebi, S.N., Bose, A.J., Subramanian, A., Mann, S.: Walking through sight: Seeing the ability to see, in a 3-D augmediated reality environment. In: 2014 IEEE Games Media Entertainment, pp. 1–2 (2014)

    Google Scholar 

  13. Kress, G.R.: Multimodality: a social semiotic approach to contemporary communication. Routledge, London (2010)

    Google Scholar 

  14. Luce, H.R. (ed.): Special Double Issue: Photography. Life Magazine, pp. 112–113 (1966)

    Google Scholar 

  15. Mann, S.: ‘Mediated Reality’. TR 260. M.I.T. M.L. vismod, Cambridge (1994). http://wearcam.org/mr.htm

  16. Mann, S.: Humanistic computing: “WearComp” as a new framework and application for intelligent signal processing. Proc. IEEE 86(11), 2123–2151 (1998)

    Article  Google Scholar 

  17. Mann, S.: Can humans being clerks make clerks be human? Können Menschen, die sich wie Angestellte benehmen, Angestellte zu menschlichem Verhalten bewegen? Informationstechnik und Technische Informatik 43(2), 97–106 (2001)

    Google Scholar 

  18. Mann, S.: Intelligent Image Processing. John Wiley and Sons, New York (2001)

    Book  Google Scholar 

  19. Mann, S.: Mediated Reality with implementations for everyday life. Presence Connect (2002). wearcam.org/presenceconnect

    Google Scholar 

  20. Mann, S.: Telematic tubs against terror: immersive interactive media of the post-cyborg age. Leonardo 37(5), 372–374 (2004)

    Article  Google Scholar 

  21. Mann, S.: “fl Huge UId Streams”: fountains that are keyboards with nozzle spray as keys that give rich tactile feedback and are more expressive and more fun than plastic keys. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 181–190 (2005)

    Google Scholar 

  22. Mann, S.: Phenomenal augmented reality: advancing technology for the future of humanity. IEEE Consum. Electron. Mag. 4(4), 92–97 (2015)

    Article  Google Scholar 

  23. Mann, S.: Surveillance (oversight), sousveillance (undersight), and metaveillance (seeing sight itself). In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1408–1417 (2016)

    Google Scholar 

  24. Mann, S.: Rattletale: phase-coherent telekinetic imaging to detect tattletale signs of structural defects or potential failure. In: 2017 13th International Conference on Signal-Image Technology Internet-Based Systems (SITIS), pp. 241–248 (2017)

    Google Scholar 

  25. Mann, S.: Phenomenological augmented reality with the sequential wave imprinting machine (SWIM). In: 2018 IEEE Games, Entertainment, Media Conference (GEM), pp. 1–9 (2018)

    Google Scholar 

  26. Mann, S., Picard, R.W.: Being ‘undigital’ with digital cameras: extending dynamic range by combining differently exposed pictures. In: Proceedings of the IS&T 46th Annual Conference, pp. 422–428 (1995)

    Google Scholar 

  27. Mann, S., Fung, J., Moncrieff, E.: EyeTap technology for wireless electronic news gathering. SIGMOBILE Mob. Comput. Commun. Rev. 3(4), 19–26 (1999). https://doi.org/10.1145/584039.584044

    Article  Google Scholar 

  28. Mann, S., Manders, C., Fung, J.: Painting with looks: photographic images from video using quantimetric processing. In: Proceedings of the Tenth ACM International Conference on Multimedia, pp. 117–126 (2002)

    Google Scholar 

  29. Mann, S., Janzen, R., Aimone, C., Garten, A., Fung, J.: Performance on physiphones in each of the five states-of-matter: underwater concert performance at DGI-Byen Swim Centre (Vandkulturhuset). In: International Computer Music Conference, ICMC ’07, August 27–31, Copenhagen (2007)

    Google Scholar 

  30. Mann, S., Lo, R.C.H., Ovtcharov, K., Gu, S., Dai, D., Ngan, C., Ai, T.: Realtime HDR (High Dynamic Range) video for eyetap wearable computers, FPGA-based seeing aids, and glasseyes (EyeTaps). In: 2012 25th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 1–6 (2012)

    Google Scholar 

  31. Mann, S., Janzen, R., Ali, M.A., Scourboutakos, P., Guleria, N.: Integral kinematics (time-integrals of distance, energy, etc.) and integral kinesiology. In: Proceedings of the 2014 IEEE Games, Entertainment, Media Conference (GEM), Toronto. Citeseer, pp. 22–24 (2014)

    Google Scholar 

  32. Mann, S., Kitani, K.M., Lee, Y.J., Ryoo, M.S., Fathi, A.: An introduction to the 3rd workshop on egocentric (first-person) vision. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 827–832 (2014)

    Google Scholar 

  33. Mann, S., Hao, M.L., Warner, J.: Virtual reality games in sensory deprivation tanks. In: 2018 IEEE Games, Entertainment, Media Conference (GEM), pp. 1–9 (2018)

    Google Scholar 

  34. Mann, S., Defaz, D., Pierce, C., Lam, D., Stairs, J., Hermandez, J., Li, Q., Xiang, Y.X., Kanaan, G., Chen, H., et al.: Keynote-eye itself as a camera: sensors, integrity, and trust. In: The 5th ACM Workshop on Wearable Systems and Applications, pp. 1–2 (2019)

    Google Scholar 

  35. Mann, S., Do, P.V., Garcia, D.E., Hernandez, J., Khokhar, H.: Electrical engineering design with the subconscious mind. In: 2020 IEEE International Conference on Human-Machine Systems (ICHMS), pp. 1–6 (2020)

    Google Scholar 

  36. Mann, S., Do, P.V., Lu, Z., Lau, J.K.K.: Sequential Wave Imprinting Machine (SWIM) implementation using SDR (Software-Defined Radio). In: 2020 Seventh International Conference on Software Defined Systems (SDS), pp. 123–130 (2020)

    Google Scholar 

  37. Mann, S., Garcia, D.E., Do, P.V., Lam, D., Scourboutakos, P.: Moveillance: visualizing electric machines). In: 2020 22nd Symposium on Virtual and Augmented Reality (SVR), pp. 420–424 (2020)

    Google Scholar 

  38. Mann, S., Pierce, C., Bhargava, A., Tong, C., Desai, K., O’Shaughnessy, K.: Sensing of the self, society, and the environment. In: 2020 IEEE Sensors, pp. 1–4. IEEE, Piscataway (2020)

    Google Scholar 

  39. Mann, S., Wyckoff, C.: Extended reality, MIT 4–405 (1991). Available for download from http://wearcam.org/xr.htm

  40. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on the reality-virtuality continuum. In: Telemanipulator and Telepresence Technologies, vol. 2351. International Society for Optics and Photonics, pp. 282–292 (1995)

    Google Scholar 

  41. Minsky, M., Kurzweil, R., Mann, S.: The society of intelligent veillance. In: 2013 IEEE International Symposium on Technology and Society (ISTAS): Social Implications of Wearable Computing and Augmediated Reality in Everyday Life, pp. 13–17 (2013)

    Google Scholar 

  42. Mohl, R.: Cognitive space in the interactive movie map: an investigation of spatial learning in virtual environments. Ph.D. Dissertation. Massachusetts Institute of Technology (1981)

    Google Scholar 

  43. Paradiso, J.A., Landay, J.A.: Guest Editors’ introduction: cross-reality environments. IEEE Pervas. Comput. 8(3), 14–15 (2009)

    Article  Google Scholar 

  44. Rizzo, A.A., Galen Buckwalter, J., Neumann, U.: Virtual reality and cognitive rehabilitation: a brief review of the future. J. Head. Trauma. Rehabil. (1997)

    Google Scholar 

  45. Rizzo, A., Cukor, J., Gerardi, M., Alley, S., Reist, C., Roy, M., Rothbaum, B.O., Difede, J.A.: Virtual reality exposure for PTSD due to military combat and terrorist attacks. J. Contemp. Psychother. 45(4), 255–264 (2015)

    Article  Google Scholar 

  46. Robinett, W.: Synthetic experience: a proposed taxonomy. Presence Teleop. Virt. 1, 229–247 (1992)

    Article  Google Scholar 

  47. Rothbaum, B.O., “Skip” Rizzo, A., Difede, J.A.: Virtual reality exposure therapy for combat-related posttraumatic stress disorder. Ann. N. Y. Acad. Sci. 1208(1), 126–132 (2010)

    Google Scholar 

  48. Scourboutakos, P., Lu, M.H., Nerker, S., Mann, S.: Phenomenologically augmented reality with new wearable LED sequential wave imprinting machines. In: Proceedings of the Eleventh ACM International Conference on Tangible, Embedded, and Embodied Interaction, pp. 751–755 (2017)

    Google Scholar 

  49. Stanney, K.M., Kennedy, R.S.: The psychometrics of cybersickness. Commun. ACM 40(8), 66–68 (1997)

    Article  Google Scholar 

  50. Stratton, G.M.: Some preliminary experiments on vision without inversion of the retinal image. Psychol. Rev. 3(6), 611–617 (1896)

    Article  Google Scholar 

  51. Tan, L.: Mediated reality in bearable prosthesis: a tool for surveillance and intervention. In: Proceedings of the 1st Fascinate Conference. Falmouth University, Cornwall, pp. 1–17 (2013)

    Google Scholar 

  52. Tang, F., Aimone, C., Fung, J., Marjan, A., Mann, S.: Seeing eye to eye: a shared mediated reality using EyeTap devices and the VideoOrbits gyroscopic head tracker. In: IEEE International Symposium on Mixed and Augmented Reality, pp. 267–268 (2002)

    Google Scholar 

  53. Wiederhold, B.K., Wiederhold, M.D.: Virtual Reality Therapy for Anxiety Disorders: Advances in Evaluation and Treatment. American Psychological Association, Washington (2005)

    Book  Google Scholar 

  54. Wyckoff, C.W.: An experimental extended response film. Technical report No. B-321. Edgerton, Germeshausen & Grier, Inc., Boston (1961)

    Google Scholar 

  55. Wyckoff, C.W.: Silver halide photographic film having increased exposure-response characteristics. US Patent 3450536A (1961)

    Google Scholar 

  56. Wyckoff, C.W.: An experimental extended response film. SPIE Newsletter, 16–20 (1962)

    Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the many students at others at MannLab Canada, including Max, Sen, Jackson, Arkin, Alex, and Kyle. The authors appreciate the ongoing work on the Social Distancer project with Cayden Pierce, Christina Mann, and Chris Tong. Some of the work is supported, in part, by Mannlab Shenzhen.

Author information

Authors and Affiliations

  1. University of Toronto, Toronto, ON, Canada

    Steve Mann & Phillip V. Do

  2. University of Washington, Seattle, WA, USA

    Tom Furness

  3. IEEE Standards Association, 0xSenses Corporation, Piscataway, NJ, USA

    Yu Yuan

  4. IEEE Standards Association, Piscataway, NJ, USA

    Jay Iorio

  5. Sun Yat-sen University, Guangzhou, China

    Zixin Wang

Authors
  1. Steve Mann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Phillip V. Do
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tom Furness
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jay Iorio
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zixin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Steve Mann or Phillip V. Do .

Editor information

Editors and Affiliations

  1. Department of Mechanical Engineering, National University of Singapore, Singapore, Singapore

    Andrew Yeh Ching Nee

  2. Department of Mechanical Engineering, National University of Singapore, Singapore, Singapore

    Soh Khim Ong

Section Editor information

    Rights and permissions

    Reprints and permissions

    Copyright information

    © 2023 Springer Nature Switzerland AG

    About this chapter

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this chapter

    Mann, S., Do, P.V., Furness, T., Yuan, Y., Iorio, J., Wang, Z. (2023). Fundamentals of All the Realities: Virtual, Augmented, Mediated, Multimediated, and Beyond. In: Nee, A.Y.C., Ong, S.K. (eds) Springer Handbook of Augmented Reality. Springer Handbooks. Springer, Cham. https://doi.org/10.1007/978-3-030-67822-7_1

    Download citation

    • DOI: https://doi.org/10.1007/978-3-030-67822-7_1

    • Published:

    • Publisher Name: Springer, Cham

    • Print ISBN: 978-3-030-67821-0

    • Online ISBN: 978-3-030-67822-7

    • eBook Packages: Computer ScienceComputer Science (R0)

    Publish with us

    Policies and ethics

    Search

    Navigation