Keywords

1 Introduction

The visualization of scientific data and large network data have been evolving with the arrival of computer graphic techniques. Most recently key developments have involved the ability to process large amounts of data and render them in new and different graphical formats. The array of information remains primarily constrained to 2D dimensional projection (e.g., Tableau) or 2D renderings of 3-dimensional arrays.

Data visualizations are used to access the powers of human cognition. Data visualizations leverage the powerful capacities of human spatial, object, and color perception to detect patterns, trends, and anomalies in physical forms.

The introduction of virtual reality and especially augmented reality has enabled scientists, data analysts, and the general public to interact with scientific and big data visualizations in more intuitive ways. Augmented reality and virtual reality array data in a simulated physical space around the body. In this way, the leverage all of the visualization capabilities of traditional visualization but are further augmented by human spatial cognition and tangible, full body, physically interaction with the data. Virtual and augmented reality array the data around the body and engage in body-centric spatial awareness to detect patterns and comprehend scale and relationships. Full body interaction with data visualizations enables users given them new ways to engage, alter, and “interrogate” the data.

1.1 Visualizing Unseen Phenomena at Various Scales: Magnetic Fields

Magnetic fields are largely invisible phenomena exerting effects in the world. Magnetic fields are pervasive and can be modeled at different scales from the nanoscales to larger geospatial scales such as the interaction of sun and the earth’s magnetic fields. Magnetic fields lend themselves to the possibility of visualization at different scales including the superimposing on the real-world objects, the magnification of smaller scale magnetism, and the visualization of large scale early magnetism at an experiential scale of the body. In the project we start the interactive visualization of Environmentally Mapped View; these interactive visualizations view data mapped directly at one-to-one scale on physical objects or the surrounding environment.

2 Specific Objectives

The specific objectives are:

  • Develop a general-purpose interactive data visualization platform.

  • Visualize, simulate, and overlay natural physical processes on physical objects and natural spaces.

  • Develop and demonstrate the approach via the visualization of magnetic and electromagnetic phenomena.

  • Transform visualization to body-centric, tangible, and interactive scientific visualization

  • Visualize and simulate magnetic fields at different scales.

  • Support improved understanding of physical phenomena for scientific data exploration and education.

3 Methods and Procedures

3.1 Overview

Our development of a graphics and interactive design pipeline for the creation of interactive visualizations of time-based and environmental phenomena rendered in the physical environment to an embedded 3D observer. The procedure for body-centered, tangible experience using a multi-user augmented reality environment we call AR Vis.

3.2 Data Sources

We are building a general-purpose platform and a set of procedures for interactive visualization suited to display spatial and environmental data varying over time and different scales. For the prototype study, we demonstrate the visualization using magnetic data from the Center for Solar-Terrestrial Research at NJIT.

Magnetic Field Data: From the Inviable to the Tangible.

At the beginning of project work, magnetic field data that are used for the study of solar and terrestrial sciences are visualized (Fig. 1).

Fig. 1.
figure 1

Space magnetometer collecting terrestrial data at different Scales (Source: NASA)

Full size image

They are typically acquired by the instrument called “magnetometer” that measures the background (ambient) or time-varying component(s) of magnetic fields. It has a wide range of applications and is one of the widely utilized instruments for geospace research. There are a number of science magnetometers in operation in ground-based observatories and aboard spacecraft.

Magnetic field data sets are publicly available either online (e.g., NASA) or upon request. They are provided in the text-based ASCII format or Common Data Format (CDF) which contains the three-axis components (vector) or scalar (absolute) values to represent magnetic fields around the Earth.

The Earth’s magnetic fields are never constant, being disturbed by various sources. One of them is the continuous flow of charged particles from the Sun called “solar wind.” Thus, monitoring the Earth’s magnetic fields provide critical information about the environment between the Sun and Earth.

A member of this project, Dr. Kim at NJIT Center for Solar-Terrestrial Research (CSTR), is currently in charge of the operation of magnetometers in the polar regions in both hemispheres including Canada, Norway, Greenland, and Antarctica and has extensive experiences in analyzing data from the magnetometers. We are also participating in a satellite program run by NASA.

Here, we visualize such data sets from the various magnetometers and thus to provide the general public with more tangible experience with the unseen forces that surround us.

Data Mining, Reduction, and Culling.

Data mining and culling extract spatial and time-varying components. Data sets are modified and formatted to support 3D visualization tools.

3.3 Visualization and Interactive Simulation Model Building

The magnetic field visualization prototype was created in Unity and the scripts were written in C#. The Microsoft Mixed Reality ToolKit was used to implement the Hololens functionality of the Unity Project.

The Mixed Reality ToolKit was necessary for the project because it allows Unity projects to be directly built onto the Hololens. It also contains scripts and prefabs which allows for a Hololens scene to be setup easily. For example, the Mixed Reality Tool Kit Parent Prefab was used to represent the Hololens’ in the scene. In addition, the prefab contained scripts and other objects attached to it that help with gazing and head position. There are also other prefabs in the Mixed Reality Toolkit that help with input and other essential things needed for a Hololens’ project.

The grid was created using two geometry shaders. The shader language used was Cg, which is a variant of HLSL. A series of vertices are passed to the GPU, the first geometry shader draws a cube at each of these points, while the second geometry shader draws lines connecting neighboring points. In addition, a script calculates the distance between the magnetic object in the scene and the player, as well as the distance between the player and each vertex. The sum of these calculations is adjusted by a manually tweaked value and applied to the vertices’ positions to make them move rapidly or slowly. The number is also used to determine the color of the grid vertices (Fig. 2).

Fig. 2.
figure 2

Environment sensors and optical displays within the augmented reality platform to be integrated into AR Vis interactive visualizations. (Source: AR_Vis interactive Visualization)

Full size image

Platform Porting and Interactivity.

The interactive visualizations and simulations are ported for interaction in an NJIT multiuser environment using Hololens and hardware. Hololens is an augmented reality system using optic see-though HMD for blending virtual objects into the physical environment. We begin with the networked dual-use platform (Fig. 3).

Fig. 3.
figure 3

AR Vis interactive visualization platform using Hololens augmented reality head-mounted displays. (Source: Science & Art Exhibition at CoAD Library, NJIT)

Full size image

Our visualizations Micro Software, Hololens’s application framework. We are able to support multiple spatial computing and visualization experiences to run at the same time.

Multi-observer Environment.

Because users and scientists often interact with data in teams, the interactive simulations design for two or more observers of the environment. Both are able to simultaneously view shared visualization. Models are shared across devices in real time (Fig. 4).

Fig. 4.
figure 4

Example of field data rendered for multi-user augmented reality interaction. (Source: MS web.)

Full size image

Augmented Reality Views.

The platform yields visualizations at different views using augmented reality. The current prototype supports the visualization of body-centered magnetic fields. Users are instrumented with a magnetometer and move through a real-time dynamic visualization of the magnetic fields in the room or space around their body. Actions include selection or even physical motion modify the data or support interaction.

The platform envisions different types of augmented reality views of magnetic data described below and include an object or environmentally mapped views and observer or “Gods eye” views of very large planetary scale static or real-time magnetic field activity.

figure a

4 Future Plans

This project supports the development of a prototype and development pipeline for an augmented reality, visualization interface.

Preliminary user studies examine the usability, interactivity, of the interactive visualization. Learning and discovery outcomes are also be assessed by comparing the spatial augmented reality visualizations to traditional visualizations of the same data.

The multi-user system will be generalizable to the visualization and augmentation of other information including for example architectural and structural visualization, embedded visualization of urban and physical environments, electrical grid and infrastructure visualization, medical patient data visualization, big data visualizations, radiation, and environment mapping.