The Intel® RealSense™ camera utilises a variety of sensing technologies to achieve depth perception, 3D imaging, interior mapping, and feature tracking. Intel present a variety of uses for the Intel® RealSense™, which considers virtual reality, robotic vision development, drones, security, 3D scanning and tracking, amongst others. Intel have developed a range of camera systems which can be integrated into a variety of platforms including PC’s, laptops, 2-1 Laptop/PC’s, external camera systems, smartphones and tablets.
A number of the capabilities of these camera systems, and those produced by other manufacturers are particularly important to consider when creating applications for clinical research; a selection of these are described in Table 1.
All of the 3D camera’s that utilise Intel's RealSense software development kits (SDKs) enable the development of natural, immersive, and intuitive software applications utilising routines that include but that are not limted to, facial recognition, hand gesture, background removal, 3D scanning and skeletal tracking. Another recent addition to the market is the Intel Euclid™ Development Kit. This utilises an Ubuntu® (Canonical Ltd., London, UK) operating system integrated with Intel RealSense depth camera technology in combination with an Intel® Atom™ ×7-Z8700 Quad Core CPU providing a compact and sleek all-in-one computer and depth camera system .
The Intel RealSense camera range can be used to scan and track facial features and gestures with and without facial hair and glasses. It has the ability to 3D track up to 78 facial landmark points that can support avatar creation, emotion recognition and facial animation . The Intel RealSense can also detect head orientation along 3D axes for yaw, pitch and roll. Depending on the camera in use the ability to track up to four faces with marked rectangles for face boundaries is achievable. For systems where the environment cannot be controlled this is extremely useful as it would allow the individual patient to be identified and solely tracked. The Intel RealSense camera range can offer greater resolution and sampling rates compared to some of the current market leaders, including the Microsoft Kinect 2.0. The greater resolution and sampling rate offer advantages when tracking fine or fast movements.
The Intel RealSense can also perform skeletal tracking. By tracking the 3D coordinates of body joints, the Intel RealSense can create a virtual skeleton allowing the calculation of patient movement parameters and joint angles. This real-time tracking enables the development of applications in a number of key areas. Tracking a user’s skeletal movement during a routine exercise or while playing a game opens up the ability to detect subtle improvements during rehabilitation from injury or stroke. In the past this has been done by placing sensors on the user’s body at key locations, or using complex multi-camera systems, and the transmitting data to a connected machine for analysis. These setups are often expensive, inconvenient and difficult to use in non-specialist settings. In addition, joint tracking can enable the measurement of body movement attributes such as gait parameters, body sway and balance while the patient is conducting simple performance tests in front of the camera. The ability to conduct these tests simply in small spaces and without attaching sensors or instruments to the patient increases the utility of this approach.
Intel RealSense camera SR300, R200 & F200
The Intel RealSense SR300 is a short range, coded light 3D imaging system and is one of the smallest 3D depth and 2D camera modules currently available on the market. The Intel RealSense SR300 combines depth sensing with a 1080p RGB camera, this provides users the opportunity to work with dynamic background segmentation, 3D scanning, facial recognition and hand gesture recognition. The Intel RealSense SR300 camera is ideal for face analytics and tracking, scanning and mapping, scene segmentation, hand and finger tracking and augmented reality. This is completed through the utilisation of an infrared (IR) projector and IR camera in tandem using coded light patterns .
The Intel RealSense Camera R200 is a USB 3.0 world-facing camera system that can provide colour, depth, and infrared video streams which can be utilised alongside various supported systems such as an Ultrabook™, 2-in-1, All-in-One, or mobile platforms (i.e. smart phones and tablets) . The R200 has a full HD colour camera and IR depth sensing features. Its three cameras provide RGB (colour) and stereoscopic IR to produce depth. The R200 consists of an infrared laser projection system, two infrared and full HD colour imaging sensors. The depth video stream is generated using stereo vision technology which is assisted by the infrared laser projector and the two infrared imaging sensors . The R200 system provides the capability to manipulate filters through varied options including parallax, dolly zoom, motion affects, object segmentation, colour popping, and others . Digitally capturing people or objects in a 3D world by utilising body scans provides the developer with the option to build 3D printable object and components, including the ability to create and manipulate personalised avatars. The creation of 3D objects or avatars provides objects that can be incorporated into real-world applications and spaces which can aid visualisations . The use of avatars has often been used in rehabilitation systems where patients may not like to see themselves after suffering a temporary or permanent physical disability.
The Intel RealSense range has progressively improved over recent years, and this is noticeable especially in the quality of motion capture, camera capabilities and field of view ranges. A full comparison of the general features of the SR300, R200 and F200 systems can be found in Table 2.
The Intel RealSense SDK, SDK components and depth camera managers for the F200, SR300, and R200 versions are no longer being updated due to the introduction of newer versions of the camera range , as described below. A full comparison of the SDK algorithm operating ranges and applications can be found in Table 3.
The 3D facial tracking capabilities of the Intel RealSense SDK have some important properties that assist with the development of robust and reliable applications. These relate to the ability to provide robust measurements across a range skin tones, and in the presence of additional objects such as piercings or glasses. Facial hair is another factor which can drastically alter the tracking capabilities, however the Intel RealSense SDK can combat this. Obstructions can also greatly affect the tracking capabilities of numerous camera tracking technologies; common movements such as yawning, scratching of the face etc., can affect tracking and data acquisition; the Intel RealSense camera range can also take this into consideration and continue to accurately track.
Variable lighting conditions can adversely affect the quality of the tracking and has been noted in the past to seriously affect the tracking of facial landmarks; this has been especially noted with the Microsoft Kinect . Vision based tracking methods often perform better in controlled environments with limited variables such as skin tone, clothing colour and shape, lighting level, position and restricted background clutter. Utilising this technology in the home for health measurement can be problematic for remote users where the environment may not be ideal . The Intel RealSense, however performs well in low light environments where other solutions continue to struggle but with huge variances in light intensity still being problematic.
Wearing glasses, in some situations can have an adverse effect on the tracking of the face. This is especially noted with 2D tracking where glasses can significantly affect the size of the eyes which often become amplified by the lenses, thus providing false tracking data (Fig. 1). In the presence of glasses, 2D images may result in loss of landmark tracking, as shown by the black tracking points on the 2D camera image in Fig. 1. This affects the ability to measure and detect actions such as blinking and winking. The use of 3D tracking utilising the SR300 camera in this example is unaffected by the lenses and the frames of the glasses; the tracking points clearly plot the position of the eyes which adjusts based on whether the eyes are open or closed.
Variance in lighting and colour can also have an effect on facial tracking. A good example of this is when a subject wears tinted glasses. When utilising 2D tracking the camera is no longer able to track the subject’s eyes and tries to predict where the eyes would be located as a result of the other landmark points. Yet when using 3D camera tracking, the system is able to calculate the location of the eyes accurately thus tracking is unaffected (Fig. 2).
Intel RealSense camera ZR300, D415 & D435
As the demand for the Intel RealSense and SDK development has advanced, there has become a greater demand for both short and long range depth sensing cameras resulting in the development of the Intel RealSense Cameras ZR300, D415 & D435.
The ZR300 includes a high resolution depth camera which has an integrated robust motion tracking module. The ZR300 has scope for various applications due to its high resolution depth sensing, highly robust tracking, long-range support, indoor and outdoor use, and low power usage. The indoor range of the camera is approximately 0.5-3.5 m with an outdoor range capable of reaching significantly greater distances. However the range supported does vary depending on the light conditions and environment . The Intel RealSense Development Kit ZR300 is seen as an ideal solution for education platforms, hardware prototyping and software development amongst other possible solutions. The ZR300 also has the capability to enable 3D spatial perception using motion tracking, depth perception, and area learning. Most importantly, the ZR300 has a high frame rate which enables tracking performance for 3D scanning applications and rapid motor movements . Table 4 demonstrates that the ZR300 is superior to the SR300; however, the pre-release of the D415 & D435 (Available in Q1 2018) also demonstrates further advancements.
Intel have recently launched a new D400 camera range that presents the most advanced depth perception capabilities to date. The D415 & D435 depth sensing cameras contain the latest Intel RealSense vision processor and module. The addition of the rolling image shutter and standard field of view, the Intel RealSense Depth Camera D415 offers a general-purpose solution with simple depth stream data capture. However, the D435 offers a global image shutter and wider field of view with the capability to capture and stream the depth data of moving objects thus, providing high depth perception accuracy in motion . In comparison to earlier versions of the Intel RealSense Camera range the optimal environment for the D400 series has been significantly increased with data capture possible at distances up to and in excess of 10 m in both indoor and outdoor environments .