1 Introduction

Forensic firearm identification, sometimes called ballistics, is the process of examining the characteristics of firearms, fired bullets and cartridge cases [11]. It is often applied on fired bullets and cartridge cases left at a crime scene to identify the particular firearm that was used in that crime, and so it can have a tremendous impact on the forensic investigations. This identification can be made thanks to the analysis of the microscopic striations, impressions and markings left on fired bullets and cartridge cases from the firearm’s main components such as firing pin, breech face, extractor, ejector and the rifling inside the barrel [11]. Figure 1 shows the firearm main components. The theory behind firearm identification is that these markings are unique, reproducible, and therefore, like “ballistic fingerprints” can be used to determine the actual weapon that shot.

Fig. 1
figure 1

Main components of a firearm that leave traces on cartridge cases (Breech Face, Firing Pin, Ejector, Cartridge, Barrel) and projectiles (Barrel, Barrel Rifling) [11]

Full size image

One of the first article in this field, written by Dr A.L. Hall, was published in the June 1900 issue of the Buffalo Medical Journal [10]. It dealt with the difference between bullets fired through different makes and types of weapon, by saying that they were impressed with unique rifling marks and explaining the procedure to measure land and groove markings. A milestone in the history of ballistics examination was the introduction of the optical comparison microscope. It consists in an optical bridge and two arms which fit over the vertical tubes of two microscopes, first with a monocular eyepiece and after with a binocular eyepieces. Thus allowing the simultaneous viewing of magnified images of two cartridge cases or bullets. Nowadays, the visual examination on a comparison microscope constitutes the most employed method [11], while new 3D and virtual methods are being slowly introduced.Footnote 1 Nevertheless the optical comparison microscope has some limitations such as the incapability to effectively represent three-dimensional images: it also is hugely dependent on the correct lighting balance.

Mathematical approaches could help providing solutions, e.g. by providing, during the ballistics comparison, the same/similar geometric regions between two cartridge cases under examination. This is achieved through the measurement of the distance between points in the geometric space. The possibility to represent shells as 2D images, topography images and 3D point-clouds, has opened up to new research challenges, aimed to create more sophisticated semi or fully-automatic digital techniques to improve forensic ballistics analysis. Three-dimensional data can help obtain more accurate judgments, compared to the equivalent 2D representations, thanks to the larger amount of information available. The Virtual Comparison Microscopy (VCM) has been studied and tested in [4] for 3D surface topography digital examination, in which it is possible to review and compare microscopic toolmarks of cartridge cases and bullets, left by the firearms.

This work proposes an innovative approach and tools for cartridge cases comparison, based on three-dimensional graphical reconstruction and immersive observation. It allows the reconstructed 3D scenario to be observed through intuitive natural gestures, with an unprecedented level of detail, from any vantage viewpoint and exploiting visual aids. This opens up to a new way to effectively examine cartridge cases shapes, deformations and cavities. The overall framework is described in terms of: user interface, interaction within the proposed virtual environment and data analysis. Our implementation is also presented, which makes use of the developed software and Oculus Virtual Reality (VR) headset. Also, we describe our experimentation aimed at assessing viability and performance of the approach and involving 14 test-users.

“The Handbook of firearms and ballistics: examining and interpreting forensic evidence” [11] and the AFTE Theory of Identification and Range of Conclusion [18] being a main references for the qualified firearms examiner community, was taken into account throughout this paper not only in the design phase of the software but also for all the consideration/comparison reported.

The proposed software, which has been created and tested under the supervision of the Italian law enforcement agency Carabinieri Corps, represents an innovative software in the forensic ballistics field, due to the functionality and, in general, the overall structure and organization of the virtual environment. In addition, the firearms examiners highlighted the importance of forensic ballistics examination carried out even with this proposed approach. It allows manipulating and performing in-depth and more detailed analysis, for example, by entering inside the shell casings (scaling the selected object without losing any detail of the 3D point cloud using only virtual hands) can lead not only to a more objective and precise analysis but also many limitations present in the use of traditional optical tools currently used by law enforcement can be overcome.

The remainder of this paper is organized as follows: Section 2 describes two ballistics toolmark datasets; Section 3 reviews the state-of-the-art; Section 4 presents the proposed approach; Section 5 describes our system and implementation; Section 6 presents the experiments and results analysis; Section 8 draws the conclusions.

2 Ballistics toolmark datasets

Since 1925 Forensic Firearm experts have analyzed the fine scratches (striae) and impressions found on the surfaces of ballistic evidence taken from a crime scene [11] to make an identification with a given weapon. The breech face and firing pin markings (highlighted in Figure 2(b)) are very relevant and their comparisons are often crucial to resolve a forensic firearm identification. Figure 2(a) shows the position of the breech face markings and firing pin markings on the surface of a cartridge case. The main datasets containing the breech face impression and firing pin impression as 2D and 3D scanned images are shown and described below.

Fig. 2
figure 2

Bullet, case, powder and rim are the main components of a cartridge. The firing impression, breech face impression regions and the caliber number are analyzed by forensic experts to solve the firearm identification task

Full size image

2.1 NBTRD and RIS datasets

The NIST Ballistics Toolmark Research Database (NBTRD) clearly appears as one of the datasets most used in forensic ballistics comparison. The dataset is freely available,Footnote 2 and it allows users to choose different types of cartridge cases based on the calibre millimetre, brands and other features. It is very useful for developing and validating methods aimed for example at firearm identification. The dataset contains two different types of shell description: (1) Two-dimensional data consisting of 2D images in PNG format; (2) Three-dimensional data consisting of point clouds in .X3P format. The 3D topography of 3D data is a direct measurement of surface contours.

Figure 3 shows examples of NBTRD dataset’s 2D images and 3D point clouds. The R language is used to plot 3D data.Footnote 3 Although working with 3D data is more difficult than working with 2D images, 3D data provides high details and precision, which is an essential quality in ballistics comparison. Furthermore, 3D data are not sensitive to acquisition light conditions. This is a clear advantage as e.g. shadows may hide relevant details.

Fig. 3
figure 3

Examples of 2D images and 3D point clouds of NBTRD and RIS dataset

Full size image

An innovative dataset of bullets and cartridge cases has been developed in Italy by the law enforcement agency “Reparto Investigazioni Scientifiche” (RIS), which is part of Carabinieri Department in Messina. The dataset is composed by 2D images and 3D point clouds. The data have been acquired through the BalScan laser scanner, which allows the acquisition of a wide range of calibres of guns and other firearms type. As described in the official documentation,Footnote 4 bullets, cartridge cases and other firearm parts, are scanned in high 3μm resolution and include 3D information. BalScan is very suitable for the scanning and comparison of firearm’s deformed bullets, bullet fragments, and even breech faces and firing pins. Some features of this scanner are: high-end monochrome and colour digital camera; high-end telecentric lens developed especially for the BalScan devices; precise laser focus; segmented circular LED illuminator, LED linear side-light; and a 3μm/px resolution. 2D images are saved in JPEG format, whereas 3D point clouds are saved in STL format. The latter allows a simple 3D visualizer program to show the contained 3D data.

The RIS database contains a much richer 3D set of data than the NBTRD. The RIS database has the breech face and fire pin impressions in the same file, whereas the NBTRD database typically contains only the breech face impressions. In the few cases the NBTRD database also includes the firing pin impressions, these are saved in a file (.x3p) that is different from that of the breech face impression. It should also be noted that the RIS dataset is acquired only through the BalScan laser scanner, whereas data in the NBTRD dataset are acquired using with different laser scanners and quite often with lower resolution too.

Currently, the RIS dataset is composed of 318 2D images in JPEG format and 321 3D data in STL format. This last one is composed of different brands and different millimeters (9 and 7.65).

Examples of 2D images and 3D point-cloud of the RIS and NBTRD datasets are reported in Fig. 3.

3 Related works

Forensic firearms examination is based on the judgment of examiners expert in the field. Specific software can help the forensic expert to further confirm the result of the ballistics comparison through a probability ratio or estimated error rate between two or more cartridge cases under analysis. Different approaches have been proposed to the firearm model identification, through the analysis of traces left by different guns on shells and bullets. Tai and Eddy [32] proposed a fully automated technique based on 2D optical images for the comparison of the breech face impressions. The technique was composed by the following main steps: pre-processing, similarity metric computation and uncertainty quantification. The pre-processing phase included the process of the selection of the breech face marks, the image alignment, the removal of circular symmetry, outlier removal and filtering. The similarity computation was based on the maximum cross-correlation function between two images. Kara [12] proposed a method to identify similarities and differences between gun models. The related technique automatically detects traces on the shell images such as those related to the firing pin impression, capsule traces, and their intersection. The traces are then compared through image analysis and other identification processes, which result in a set of numeric values stating similarity significance.

Since 2015, several methods have been proposed in the literature, based on the use of the Congruent Matching Cells (CMC) algorithm proposed by Song [24] and developed by the National Institute of Standards and Technology (NIST). This is a standardized forensic ballistic comparison method based on the principle of discretization of cartridge case images. In particular, the cartridge case components (breech face impression and firing pin impression) are divided into cells and compared with cells from another bullet to identify valid and invalid correlation regions. Valid correlation regions describe the entire shape of a bullet that can be effectively used for ballistic comparison. Consequently, invalid correlation regions do not contain useful features and can be excluded for analysis. Cells are defined as congruent cells when there are: (a) similar registration angles for all related cells; (b) “congruent” x-y spatial distribution pattern for related cells; and (c) high similarity of surface topography. The latter foresees similarity calculated through the Area Cross Correlation Function maximum (ACCFmax). The main reason for using CMC to divide the surface to be analyzed into cells, is that of comparing complete surfaces in which several uncorrelated regions are present. The classification accuracy (e.g. of the firearm model identification task) in this way drastically reduced. Consider, for example, the brand of the cartridge. Two cartridges fired by the same weapon but of different brands, might have many uncorrelated regions and only a few regions precisely characterizing the weapon. It depends mainly on the type of material the cartridge case was made of. In this case they will not be able to achieve high classification results because several existing methods analyze the entire cartridge case surface. As a result, the algorithms will classify the two shells as having been fired by different weapons. The CMC method is currently one of the most powerful methods to achieve high classification values in ballistic firearms comparisons. This method has proven very useful in making the identification process more objective and in obtaining error estimates. Approaches [25, 26, 29, 41] used the CMC method to analyze the firing pin and breech face impression. A variant of the CMC algorithm has been later proposed [5], which led to high improvement in detection accuracy for breech face image correlations. In their work the CMC method was improved by considering a feature that they called ‘convergence’ representing how much cell pairs tend to converge to a specific angle for the overall alignment being effective. Song et al. [28] used the virtual image standard (VIS, originally proposed by NIST - [27, 30]) method to demonstrate and evaluate the performance of correlation accuracy and false positive (FP) error rate of five common CMC algorithms [5, 6, 24, 35, 36]. The experimental results showed that all CMC algorithms can correctly identify the firing weapon through the analysis of breech face impressions. In conclusion, the authors state that the convergence CMC algorithm should be considered the main method to be used for future work in the field of forensic ballistics comparison of firearms.

Recent works primarily consider 3D data to identify the weapon used in a crime scene. Typically, these approaches take into account an initial pre-processing step that includes, for example, alignment or extraction of key points. The representation of bullets and shells in 3D topographies or point clouds are used by modern algorithms in order to obtain comparable and more accurate measurements than the traditional analysis performed using comparative optical microscope.

Banno et al. [1] used 3D data to identify a firearm by comparing the impressions left on the surface of the cartridge case. They used the iterative closest point (ICP) method [3] for the alignment of the 3D shapes related to the two shells that must be compared. To establish the similarity or the difference between pairs of cartridge cases, a distance measure between the aligned 3D shapes was considered.

Riva and Champod [20] proposed a semi-automated method capable of analysing 3D measurements on shells. After a manual pre-processing phase, mainly based on image alignment and ad-hoc pre-processing, they computed the Likelihood Ratios (LR) to compare the firing pin impression and the breech face of two cartridge cases.

In late 2016 the NIST Ballistics Toolmark Research Database published an open-access dataset containing 2D and 3D acquisitions of bullets and shells fired by different firearms. Novel techniques for shells comparison and weapons identification could be valuated on the NBTRD dataset.

Morris et al. [19] exploited the NBTRD to evaluate the performance of the Integrated Ballistics Identification System (IBIS). This involved five standard cartridge cases and resulted in a good discrimination performance. The IBIS had previously been evaluated on the National Integrated Ballistics Information Network (NIBIN) dataset, which was composed by bullets and cartridge cases fired in a crime scene.

An interesting study was proposed by Zhang et al. [39], which exploited the combination of two algorithms: SIFT (Scale Invariant Feature Transform) and RANSAC (RANdom SAmple Consensus) applied to the breech face impression. The SIFT algorithm was used for the extraction key points and to compute its descriptors. The RANSAC algorithm was used for the key points matching to guarantee more robust results. Zhang et al. [40] showed that the RANSAC algorithm lacks robustness due to estimating a parametric model defined by a minimum number of randomly selected samples. Again based on features extracted through the SIFT algorithm to compare pairs of breech face impressions, the authors proposed an alternative to the RANSAC algorithm based on the use of the Support Vector Regression (SVR) method [23] in order to obtain a more robust approach. The results obtained, comparing breech face impressions fired from different weapons, demonstrate the robustness of the proposed method by overcoming the limitations of RANSAC, obtaining a clear distinction between known matching (KM) and non-matching (KNM) pairs of key points. There is not yet consensus on a shared benchmark of the various techniques. In Tai and Eddy [32] the results were presented by the TOP-10 list test (originally proposed in [37]). For each image I the TOP-10 list is composed by 10 images, available in the reference dataset, with the highest similarity scores when compared against I.

Giudice et al. [9] proposed a new neural network based framework called Siamese Ballistics Neural Network (SBNN) to identify whether two cartridge cases were fired from the same weapon. This work used the NBTRD dataset to perform experiments, and achieved the best results compared to previous works in literature regarding the firearm pattern classification. Dutta et al. [8] also exploited the power of deep leaning algorithms in this field. Specifically, the authors isolated the striations obtained from the firing process from the base of the bullet through a process of identifying and segmenting these evidences. This task may be necessary for accurate forensic ballistics examination because some striations may appear hidden, poorly highlighted. Authors used CNN architectures based on semantic segmentation called U-Net [21], Inception U-net [31] and Residual U-net [7]. The experiments show that these deep learning architectures can solve this task excellently, achieving validation accuracy of 88% (U-Net), 88.30% (Inception U-net) and 88.79% (Residual U-net).

Tai and Eddy [33] proposed a fully automated method in order to match topographical measurements of cartridge cases. This work exclusively focused on breech face marks and exploited 3D data. In experiments carried out on the NBTRD dataset comparison were performed using the old reflectance microscopy technology.

Finally, Basu et al. [2] compared different feature-extraction methods for calculation of likelihood ratios from 3D digital images of fired cartridge cases fired approximately by 300 firearms of the same class. The comparison was made considering features extracted from only the firing-pin impression, from only the breech-face region, and from the entire cartridge case. The authors showed that the best performance was achieved by using the Zernike-moment based features [13, 34, 38] by analyzing the entire region of interest.

It is worth noting that using Virtual Reality (VR) in forensics is not a first [17]. In state of the art there are works addressing the examination of crime scenes [22] or the participation to autopsy operations [14]. While VR introduces many features to the examiner, the collected data resolution and quality is always a limit. This is different for ballistics: cartridges or bullets are small object easier to be scanned and reconstructed in 3D and this activity can be made many times (differently from a crime scene). Moreover, previously mentioned literature techniques demonstrated that 3D dataset available (and therefore acquisition hardware) are already good enough in terms of quality for automatic evaluation.

4 Proposed framework

In this paper a method and software to exploit the 3D reconstructed cartridge case data and explore them within an immersive visualization setting, such as provided by a VR headset. This way the data can conveniently be observed in their entirety and a comparison can be made more extensively. User’s movements made during observations, such as a change of viewpoint or object positioning can be more natural and intuitive, e.g. based on head rotation and hand/body gestures. The observed cartridge case can also be conveniently magnified and seen by vantage viewpoints and even from inside the shell. The proposed approach provides in-depth comprehension of the observed objects, allowing forensic experts to compare and contrast cartridge cases and bullet traces more effectively and accurately. Below we describe user interface and interaction within the proposed virtual environment, and data analysis characteristics. The design of the entire solution took place starting from forensic rules for ballistics examination [11] with three main features: to develop a software with forensics features and in the meantime being interactive and intuitive. This led to the adoption of a Virtual Reality (VR) solution on 3D data to give the examiner both a laboratory environment similar to what he is used to and with new “immersive” features. Figure 4 shows a brief representation of the design process starting from the above mentioned requirements.

Fig. 4
figure 4

Designing the work environment; 1) Problem conceptualization; 2) Software design; 3) Virtual Forensic Laboratory design

Full size image

4.1 Virtual environment and user interface

The acquired data are visualized within an immersive ballistic theatre requiring a 3D virtual environment where a user can interact with. The contained objects are:

  1. 1.

    Working Table. A large flat desk surface acting as support area for a dashboard panel, various objects including visual displays, and the cartridge case 3D data representation.

  2. 2.

    Dashboard. An operating console panel containing a configurable sets of buttons, knobs and icons, typically located on the working table.

  3. 3.

    Visual Displays. A panel containing a number of screens, which can be located anywhere in the virtual space, e.g. on room walls, table’s free areas and in “mid-air” space.

Observing the ballistic theatre through an immersive VR system such as a VR headset makes the user feel inside the represented environment. Furthermore, user’s actions are designed to replicate natural human gestures such as hands’ and fingers’ movements as well as head rotation. Those movements can be performed while walking, sitting and standing inside a virtual room. The supported gestures enhance user’s perceived sense of presence in the represented environment.

The hands’ controllers are graphically visualized inside the virtual environment. This is a feature modern VR headsets provide, which is supported by our software. Our software is designed to provide graphical 3D representation of the working environment, and interaction with the console buttons, consistent mapping between actual user’s hand-fingers movements and their representation in the virtual reality world. Through providing consistent mapping between actual user’s hand-fingers’ movements, their, and users’ command selection is also supported by the use of ray-casting to make action selection more precise and facilitate reach buttons, keys and knobs.

Any visualized objects including those for interaction, such as the console bottoms, users’ controllers and cartridge case 3D points, are positioned in the room and calibrated based on the actual space users have at their disposals in the real environment. All settings can be adjusted by users on demand. This includes configuring dashboard’s size and shape and number of buttons. Figure 5 shows an example view of the generated virtual environment.

Fig. 5
figure 5

(a) Actions performed in the real environment. (b) Results obtained in the virtual environment

Full size image

4.2 User interaction

4.2.1 Data exploration and 3D comparison

Typical performed actions include data exploration and 3D comparison. To identify any common elements between the shells under analysis (or between different cartridge cases). A main focus of attention are striations, markings and incisions in the cartridge cases, and the possible patterns those may form. Wearing an immersive system such as a VR headset makes the operator feel entirely immersed in the ballistic theatre and free to act within it. Operators can therefore explore the cartridge case in search for relevant elements of comparison through the use of their hands and a number of graphical visual aids, while being able to apply any viewing transformation they wish.

4.2.2 Visual aids

A number of visual aids have been designed to support operators in the environment in order to facilitate comprehension and object comparison. These typically are alignment and discrepancy enhancers, which make use of: colours (relevant to clearly identify elements which would not otherwise be visible); illumination (such as specifically distributed light sources to enhance distinctiveness); and lighting effect (allowing for example the use of virtual torches relevant to highlight specific object areas). Figure 6 shows examples of visual aids.

Fig. 6
figure 6

(a) Change the color of objects under analysis; (b) Changing the lighting of the virtual environment; (c) Light effect using a virtual torch

Full size image

4.2.3 Vantage viewpoints

User’s viewpoint and objects position can freely be adjusted. This includes:

  • Fixed Viewpoint. Fixing specific observation locations in the virtual world may facilitate comparison. Viewpoints can represent e.g. 1st and 3rd person viewing.

  • Fixed Object Position. Fixing the object location may allow users to walk around the cartridge case and gain a clearer overview, i.e. just like one would do in the real world.

  • Magnified Viewing. The observed objects can be enlarged and therefore observed in greater detail.

  • Enhanced 3D Vision. Artificial stereoscopic 3D viewing can specifically be set to increase depth comprehension and spatial awareness e.g. through the use of hypo or hyper stereo settings.

  • Inside Viewing. Users can see the cartridge case from the inside, an operation typically coupled to magnification, therefore benefiting from seeing the object from a view setting that would be impossible to recreate in the real world.

  • Simultaneous Examination. Exploration and comparison can also include objects seen at the same time. As an example, shells, bullets and guns, can be contemporarily observed one inside the other.

Figure 7 shows a illustrated summary of vantage viewpoint.

Fig. 7
figure 7

(a) Magnify viewing and fix object position; (b) Inside Viewing; (c) Examples of multiple objects in the same scene

Full size image

A viewing option can also be the one of a virtual comparative microscope, therefore associating data as it one would do when observing through a real optical microscope. This includes e.g. activating and deactivating lights, rotate shells and scale viewing. With this options, 3D data are placed in a working table dedicated to the virtual comparative microscope and all the operations applied to the data are projected onto virtual panels made available for this task.

4.2.4 Object manipulation

The environment’s objects, such as those representing panels, knobs, displays and data points, can be pulled, pushed, grabbed, released, scaled and rotated. Objects can therefore be re-positioned and differently-viewed through the use of intuitive hands’ and arms’ gestures. They represent relevant user-interaction element provided by our user interface, which improve the observation quality and minimize the risk of having hidden or not visible elements.

Compared to the use of the optical microscope, a forensic operator is no longer limited by the functionalities the instrument provides. The observed objects are digital (e.g. the point cloud of cartridge cases and polygon meshes), and can be transformed in their appearance (scaled, translated, colored, etc.) to improve their visibility and without losing any detail and geometric structure.

4.3 Data flow

4.3.1 Pre-processing

This subsection is devoted to describe the RIS dataset. The dataset points cloud is initially loaded into the system and represented through the use of computer graphics. Data visualization is set and optimized for three-dimensional and immersive viewing. This is done initially with operator assistance who manually provides the inputs to align 3D data and operating environment. The origin of the points cloud and the related depth location are set based on the size and shape of the real operating space (with further changes and adjustment made possible during data analysis). This initial setting is relevant because it allows users to conveniently manage rotations, translations and scaling operations of the visualized data, and set vantage viewpoints to observe the dense points cloud or the extracted 3D polygons. Once all initial settings are processed the virtual environment is created and all data loaded. Any data that is not part of the cartridge case and bullet, is discarded because unnecessary to the forensic analysis.

4.3.2 Examination and report

Once the system is all set the user can start exploring data and perform 3D comparisons. The outcome of this phase is saved in a report, typically consisting of a file and an HTML page. The report includes: logs of performed actions, ballistic analysis, conclusions about whether the examined bullets were fired by the same weapon and the associated uncertainty, as well as the operator’s ability to perform the ballistic analysis.

4.3.3 Repeatability

Compared to current forensic analysis tools, our system allows for revisiting and repeating the performed data analysis at any time. Repeatability represents a key element in the fields of Digital Forensics, which gives law enforcement for ballistic comparison the possibility to understand if mistakes were made in the past. With our system is possible to entirely reproduce a past procedure with full accuracy going again through all steps the forensic operator followed during the performed comparison.

5 Implementation

The proposed 3D forensic ballistics tool has been developed with the Unity softwareFootnote 5 and allows to be used with the VR Oculus Rift S. The overall framework is composed by the following main components, graphically represented in Fig. 8:

  • Initialization: A new project is created or loaded, and a folder path selected, which contains the point clouds of the cartridge cases that need to be compared. Additionally, point clouds (from different cartridge cases) can be loaded (a number greater than or equal to 1). Figure 8(a) shows the initialization interface.

  • Space Calibration: The forensic expert is able to set the center of the virtual environment simply by moving the working table in a point of interest by means of a joystick. All the tools for the 3D forensic ballistics analysis and comparison of shells are then positioned accordingly. This functionality is relevant because it adapts the center of the virtual environment to the size of the room where the user is located. This is of great importance because it avoids possible interference and clashes with room’s objects like tables, chairs, furniture, etc. Figure 8(b) shows the space calibration environment.

  • Workspace Design. The work environment contains different control panels through which the point clouds of the cartridge cases can be manipulated. It allows for a preliminary analysis to be carried out using only the VR headset’s joysticks. A detailed analysis is possible by exploiting the various functionalities available in the virtual environment. This includes simulating the working structure of the comparative optical microscope used by law enforcement agencies. Details of the characteristics of the working environment can be found in the Subsection 5.1. Figure 8(c) shows the workspace design.

Fig. 8
figure 8

3D forensic ballistics analysis interface. (a) Start interface; (b) Setting the center of the environment; (c) Working environment

Full size image

After the analysis it is possible to generate a HTML report containing each operation made by the forensic expert. This is crucial for verification and later assessments on the quality of the analysis having at the same time an action log. The analysis can be paused and continued at will. The above characteristics represent a substantial difference to the comparative optical microscope nowadays used by law enforcement. As an example, in case the forensic expert accidentally hits an object, or an involuntary movement damages the comparative microscope during a forensic analysis, a change of bullet’s position can take place and this can affect analysis integrity. In worst case scenario a new comparison may be required. With the proposed new tool a forensic expert has the possibility to pause, save, repair and restore all the operations previously performed.

5.1 Work environment

The virtual room consists of a working table where all main analysis operations on cartridge cases and bullets can be performed.

Figure 9(a) show the environment. The forensic expert can take the shells from the panel shown in Fig. 9(b-1) using only the joysticks. Operations such as rotation, scaling, and translation can be performed by the user already from the initial stage of analysis, simply through hand motion. These functions can be constrained to find common shapes or for detailed analysis of the input data. This is possible with the 4 panels highlighted in Fig. 9(b-9):

  • Start panel. In the first panel the expert can select the point cloud to which to apply the operations listed in the other 3 panels.

  • Rotation and translation panel. The rotations and translations can be blocked on different axes of the point cloud chosen by the forensic expert. This is done to analyze the point cloud in detail in order to block involuntary movements that can lead to excluding or not seeing elements that characterize the data under examination.

  • Pins panel: The rotation operation is performed with respect to the center of the point cloud. It is possible to change this constraint through the pin objects. Four different pins have been considered: Up, Down, Left and Right. When the operator select one (ore more) of these buttons with the virtual hand, the color of the corresponding pin in Fig. 9 (b-7) changes and will be placed at one of the four edges of the bullet to constrain the rotational movements with respect to the pin coordinates. In other words, all rotations will be made according to the new reference point chosen by the operator.

  • Overlapping cartridge cases: As the name of this panel suggests, it is possible to overlap point clouds to look for elements in common between shells. The forensic expert touches one of the “\(B1 \rightarrow B2\)” or “\(B2 \rightarrow B1\)” buttons with his virtual hands and bullet B1 will overlap with B2 (the dimensions of B1 will be set to the same dimensions as B2) or vice versa.

Fig. 9
figure 9

Worktable (a). Components of the environment (b): 1) Position of incoming shells; 2) Torch light parameters; 3) Light room setting; 4) Turn on/off the comparative virtual microscope (CVM); 5) Reset the position of cartridge cases; 6) Save the project; 7) Pins; 8) Torch; 9) Constraint panel to manipulate shells; 10-11) Dedicated table and panels for the CVM

Full size image

It is possible to capture and save images that represent the current scene as seen by the expert wearing the Oculus VR headset. This is done by pressing the X button on the joystick, in order to save information that may contain important details for the final evaluation of the shells under analysis.

In addition, the expert can save or reset the position of the cartridge cases as shown in Fig. 9(b-5, b-6). When the forensic expert touch on the Save as... button, a new dashboard appears in which the items that were analyzed in the shell can be selected (e.g. breech face impression, firing pin impression, etc.). The HTML report that will be created after this operation will contain all these data. For example, if the expert has saved 5 images of the firing pin impression, the HTML report will show them grouped by type (in this case with respect to the firing pin impression). Finally, it is possible to restore the original position of the bullets when the expert presses the Reset Bullets Position button. In the latter case all cartridge cases in the scene will be positioned in the panel shown in Fig. 9(b-1).

Shell analysis can be improved with the use of a torch object available in Fig. 9(b-8). The torch is turned-on when the expert presses the A button on the joystick. Different light colors and other characteristics can be set through the control panel Setting of the torch parameters in Fig. 9(b-2). Finally, the ambient light can also be changed through the Change room light panel as described in Fig. 9(b-3).

A further relevant feature available is the virtual comparator optical microscope, simulated by means of a working table (Fig. 9(b-10)) and two panels (Fig. 9(b-11)). When the forensic expert touches the green button with the virtual hands, as shown in Fig. 9(b-4), the analysis with the virtual microscope begins. and the shells will be projected into their respective panels. Specifically, the cartridge cases are moved to a dedicated desk as shown in Fig. 9(b-10). A camera and a light (not visible to the forensic expert) are placed over each shells under analysis. The cameras will be used to capture and project the status of the shells in their respective panels shown in Fig. 9(b-11).

The light from each camera was added to better simulate the real functionalities of the comparative optical microscope currently used by law enforcement. Once the bullets are placed, the working table changes as shown in Fig. 10, allowing for the activation of various features, such as zoom, rotation, translation, and light necessary for accurate ballistic comparison.

Fig. 10
figure 10

Working table in the virtual comparative optical microscope mode

Full size image

Furthermore, the above mentioned operations allow the respective cameras to be “manipulated”. In particular, if we want to enlarge a shell, the respective camera will be moved up or down. The “Zoom and Rotation” operation allows the user to scale, magnify and rotate bullets and cartridge cases. The values “0.001” and “0.5” are used to set the scaling and rotation parameters of the camera respectively. These values can be increased or decreased by the user using the “+” and “-” buttons.

Through the “Light” panel, it is possible to turn on, turn off and set the “spot angle” of the lights associated at the respective cameras. All these operations are performed using only the Oculus’ joysticks. To activate and deactivate each of the described operations, the virtual hand of the forensic expert must press the corresponding “On/Off” button (“On/Off” of the Zoom and Rotation panel, “On/Off” of the Translation panel, etc.). All projected images of the cartridge cases in the respective panels can be saved and reported in the HTML report. When the analysis with the virtual comparative microscope is finished, the operator through the red button of Fig. 9(b-4) will restore the whole working environment: the working table will be restored to its original shape (as described in Fig. 9(a)) and the shells will be moved to the starting panel (as shown in Fig. 9(b-1)).

5.2 Preprocessing of RIS dataset

As shown in Fig. 3 the 3D point clouds of the RIS dataset contains a square shape with depth information. This mainly depends on the structure of the laser scanner used in the acquisition phase. In order to proceed with an accurate ballistic comparison, it might be useful to remove this geometric shape. Another problem is the coordinate in the XYZ system of origin of all 3D bullets, because this does not correspond with the center of the point clouds, as it is not fixed at (x = 0,y = 0,z = 0), as shown in Fig. 11(a). It is necessary to change the origin of the point clouds because otherwise the forensic expert will not be able to correctly manipulate the input data in the new ballistic comparison tool. To set it correctly, simply identify the corners P1,P2,P3 in order to calculate the coordinate (x0,y0,z0), as shown in Fig. 11(a) of the input point cloud and apply the following subtractions for each point:

$$ \forall (x,y,z) \in PC \Rightarrow \left\{ \begin{array}{rl} x = x - x_{0}\\ y = y - y_{0} \\ z = z - z_{0} \end{array}\right. $$
(1)

where PC represents all point clouds in the RIS dataset.

Fig. 11
figure 11

Set the origin of the 3D data in the RIS dataset

Full size image

This sets the origin of the point cloud at the point (x = 0,y = 0,z = 0). Figure 11(b) shows an example of this operation.

Finally, as shown in Fig. 12, to remove the square at the base of the cartridge cases, simply identify all those pairs of XYZ points that have the minimum z coordinate and remove them.

Fig. 12
figure 12

Detection and removal the square in 3D point clouds in the RIS dataset

Full size image

6 Experimentation

We conducted two types of experiments as described in [15, 16]: a Pilot test and a test with ordinary people. The Pilot test, based on field experts, aims to help the authors design the usability evaluation, and the other for a wider audience aimed to assess the effectiveness and performance of the proposed approach. They described and defined it as innovative software in this domain, due to the functionality and, in general, the overall structure and organization of the virtual environment. Working with 3D point clouds, (dataset provided by RIS) turned out to be a winning point and an excellent combination with the hardware instrumentation used: the Oculus Rift S. Usability tests carried out by law enforcement and compared with the functionality and performance (including usability) of the comparator optical microscope led to the statement that, in general, the use of the headset does not prove to be limiting since they do not bring discomfort; the headset can also be used with glasses; the ballistic examination can take up to an hour or so without side effects and with minimal fatigue symptoms (consider the fact that the expert can move in the real world, even walk, and all movements are projected into the virtual environment, but can also perform the ballistic comparison from a sitting position or both. So the concept of fatigue has to be calibrated to the forensic expert and in general, it changes from person to person). Usability was rated positively: the virtual environment appears to be intuitive and instinctive; the expert just needs to learn and become familiar with how to use the Oculus joysticks well. Once the headset was removed from a running ballistics comparison, on average, forensic experts waited about 30 minutes to continue or begin a ballistic examination. In general, working in first person without the aid of other instrumentation was rated as a plus point for the proposed approach. Without any constraints (defined as interacting with other objects such as a keyboard and mouse to manipulate a point cloud in order to highlight salient features to solve the ballistic comparison), the forensic examination can turn out to be much more accurate, as evidences characterizing the weapon that fired the casing can be identified more quickly and easily. So in general, the software has been rated very highly by experts in the field. Some algorithms can, in general, be added to the software to further assist the forensic operator in order to obtain the most objective ballistics comparison possible.

Finally, we carried out usability tests involving some non-experienced people in the field, to try to understand how they manage to interact with the virtual environment and the main limitations. Fourteen persons were called in to assess our approach. They were people with various background, not specialist of virtual reality and with medium experience of computer games. They were aged between 23 and 39 years. The Oculus Rift S was the chosen VR headset used in the test. This system can be used by users wearing eye-glasses. This is a relevant features because it has been estimated that 71.4% of the test-users wear glasses. We gave each test-user the choice to run the test with or without glasses.

The assessment was implemented according to the following procedure, which was attended by each test-user

  1. 1.

    Mini tutorial on how forensic analysis of cartridge cases and bullets works and it is performed;

  2. 2.

    First questionnaire;

  3. 3.

    Practice with comparative microscope and proposed software, as well as with the VR headset (including the use of the Oculus joysticks for objects’ grip, resize, translate, zoom, etc.);

  4. 4.

    Forensic analysis of shells with both comparative microscope and proposed software;

  5. 5.

    Second questionnaire.

A small tutorial (Step 1) was made available to test-users and a practice phase was administrated to give acquaintance with the standard procedures used by law enforcement agencies. A typical operation to be executed is to analyze main components of cartridge cases and bullets in great detail. Users were instructed to replicate the same investigation on the comparative microscope. Two forms were given to user to be filled in. In the first one (Step 2), the user had to provide some information such as age, sex, nationality, use of glasses, skills in ballistic comparison and possible previous use of VR devices by answering the questions reported in Table 1. Figure 13 summarizes some of the answers given by users.

Table 1 Set of questions from questionnaire 1
Full size table
Fig. 13
figure 13

Summary of the responses obtained through the questionnaire 1

Full size image

From this first survey we can summarize that 71.4% of the users were male compared to 28.6% female, with ages between 23 and 39 years old. As for nationality, they were all Italian except for 1 person of Indian ethnicity and 1 person of Pakistan ethnicity. Interestingly, 71.4% of the users wore glasses, and no one experienced any difficulty in testing the proposed software. 85.7% of enrollees possessed a bachelor’s degree, and only 14.3% specified “other” as their education category. Very few users had domain experience in the forensic firearms ballistics comparison field (a general knowledge, e.g., what are the main components of a firearm, etc.): 57.1% voted 0; 14.3% voted 1; 7.1% voted 2; 7.1% voted 3; 7.1% voted 4; 7.1% voted 5; and, finally, no user gave score 6. The survey went into more detail in assessing a scale of 0 to 6 the domain knowledge about the field of ballistic comparison of cartridge cases and/or bullets. In this case, the objective was to understand whether the user had knowledge of what elements the forensic expert goes to analyze during ballistics comparison (e.g., firing pin impressions, generic striations, etc.). In this statistic, the same percentages listed above were obtained. From the point of view of the use of the comparator optical microscope, 50% of the users had never seen and studied the operation and usability of this instrument, only 35.7% gave as score 2, i.e., these users theoretically knew the operations of the comparator optical microscope (the remaining 14.3% voted 1). The percentages increase with 78.6% of users in having experience in the use of video games and tools similar to the Oculus Rift S. In general, from these statistics, it can be seen that most of the users do not have experience this domain, but they do have a lot of experience in the use of video games and headsets similar to the Oculus Rift S. Therefore, it was possible to proceed in evaluating mainly the usability of the software proposed in this paper.

With Step 3 completed, users proceeded to Step 4. In this phase it should be noted that some users started the comparison with the comparator microscope and others, on the other hand, started the comparison with the architecture described in this paper (the choice was made randomly)

Finally, the last questionnaire (step 5) was filled out at the end of the use of all instruments. We collected information on the usability of the proposed approach and the comfort of the headset by answering the questions reported in Table 2. Figure 14 show an overview of the responses of the second questionnaire.

Table 2 Set of questions from questionnaire 2
Full size table
Fig. 14
figure 14

Summary of the responses obtained through the questionnaire 2

Full size image

In general, one of the objectives was to try to understand how comfortable the configuration with the Oculus were and did not bring long-term discomfort of use. When it comes to the comfort of the Oculus, 64.3% of users rated 6; 28.6% rated 5 and 7.1% rated 4. Similar statistics were also obtained regarding the eyes. This information turns out to be very useful, as it was sought to find out whether disorders of any kind may arise. In case the latter statement turned out to be true, then we could not have addressed and designed a solution for forensic ballistics analysis of firearms using these tools. In general, it is also necessary to analyze and estimate how long users can proceed to (a) perform a ballistic comparison and (b) how long it is necessary to stop to perform a new forensic analysis with the proposed tool (concluded a forensic examination). These statistics are also important, for if the time estimated by users would have been too low for item (a) and too high for item (b), then the proposed tool would most likely have been too time-consuming to initiate and resolve other ballistic comparisons. In general, the results were as expected, i.e., for item (a) most users could perform a ballistic comparison for 30 to 60 minutes; for item (b), 42.9% times of 10 minutes, 28.6% times of 5 minutes, 14.3% times of 1 minute, and 7.1% times of 30 minutes were obtained. Finally, users evaluated whether the method tested and described in this article could be of added value for forensic ballistics comparison of firearms compared with the use of the traditional comparative light microscope. All users positively evaluated the virtual tool and the power of immersiveness in this domain. Some users recommended adding audio, sounds that indicate, for example, that a tool has been activated, and including soothing music that could be activated as desired.

All users evaluated the tool positively and most of them stated this type of instrumentation could revolutionize the way 3D ballistic comparison is performed. In addition, we asked 3 Italian law enforcement officers to run the same assessment procedure and encouragingly, they confirmed most test-users comments in terms of advantages brought by the proposed approach. This mainly concerned with accuracy and extended analysis possibilities, which is most helpful to cope with uncertainty.

A video demonstration of the proposed approach is available at the following link https://iplab.dmi.unict.it/mfs/Forensic-Firearms-Ballistics-VR/.

7 Discussion

The possibility given to the forensic expert to manipulate in first person the point clouds representing shell casings and projectiles with any degree of freedom and without the use of any other tools such as mouse and keyboard (which one needs to use with the common software available in this field and could make a comparison more “constraining and complicated”) characterizes the proposed method. This represents the subtle but substantial difference with respect to the 3D data visualization systems of casings and bullets used for forensic ballistics comparison: the forensic expert will necessarily have to interact with various tools in order to manipulate a point cloud; he/she will have to take into account that the screen used may not display the 3D data correctly or that the rendering of the 2D or 3D data may hide characterizing elements of the same. This additional limitation is overcome by the proposed system.

As described in Section 3, literature approaches do not work for both data representations simultaneously even Deep Learning ones: either they are methods that work only by analyzing 2D gray-scale images, or they are methods that work by analyzing only 3D data. Also, in the case where both the shell casing and the projectile are found at a crime scene, then it turns out to be useful to perform forensic ballistics analysis by analyzing both of these. This is another major limitation of the various state-of-the-art methods: either literature approaches are able to analyze only the firing pin impression and the breech face impression, or they are based only and exclusively on the analysis of the striations left by the gun barrel on the bullet. Thus the current state-of-the-art approaches turn out to be almost completely unscalable (under these considerations). The main constraints and limitations just described are almost entirely overcome with the proposed method: we can analyze both 2D gray-scale images (via the virtual comparator optical microscope) and 3D point clouds; the input data can be different in size (this is one of the constraints of methods based on deep learning techniques); it is possible to analyze 2D or 3D representations from totally different acquisition systems; and we can also analyze both shell casings and projectiles simultaneously. It is clear that the quality of the 2D or 3D digital data depends only on the scanning system used (the problems listed above persist). Another key element that differentiates the proposed framework from other state-of-the-art solutions is the ability to examine more than 2 point clouds simultaneously.

Finally, we would like to highlight the fact that the “immersive” modality, achieved through Virtual Reality, is the core of the proposed method that can heavily revolutionize the field of firearms forensic investigation, similarly to other fields. Moreover, so many limitations and problems found over the years in this field could be overcome with immersive comparison. For example, it could prove to be a great achievement to be able to simultaneously perform a ballistic comparison between different law enforcement agencies from different countries and/or cities using the same samples under examination. This principle can be widely exploited to try to create an international and standardized procedure of the ballistic comparison process of firearms in order to define the firing weapon at a crime scene.

8 Conclusion and future works

In this paper a new 3D graphic tool for forensic ballistics analysis and comparison was presented, which operates through the use of Virtual Reality technology. The main feature being a much greater flexibility in setting up working space, immersive visualization and natural interaction. This allows for greater comprehension, accurate and in-depth analysis of details, while benefiting of using a digital manipulation tool both in terms of vantage viewpoint observation (e.g. one an even see the cartridge case from inside-out) and data preservation and restoration.

For the first time, the combination of Virtual Reality and the specific Oculus Rift S headset has given the opportunity to create an innovative immersive working environment dedicated to forensic firearms identification. Forensic experts are projected into a virtual world and, using simple hand gestures, compared to common approaches available in the literature and tools used by law enforcement agencies, are able to manipulate point clouds as never before, in a completely natural way. The absence of devices with which the operator had to interface, such as a mouse and keyboard, facilitates forensic firearms examination in making a more accurate comparison analysis, thanks to the possibility of being able to manipulate the 3D input data under any conditions: with constraints, without constraints, magnifying them, going “inside” the casing without losing quality in the detail to better identify the elements characterizing the weapon that fired in the crime scenes, and much more. It is well understood, that the proposed software has so many new features, even from the forensic point of view so many properties are respected. For example, the property of “repeatability” of the analysis performed: all operations performed by the expert are stored in order to be able to perform/check the work of the forensic expert by other law enforcement agencies to validate the procedure applied for ballistics analysis.

Experiments conducted provided encouraging results. They demonstrated that our software has the potential to be successfully used for forensic ballistics comparison, and has great advantages over the traditional instrumentation. Future works will focus on the improvement of software automations with the use of Machine and Deep Learning approaches to better support users through procedures.