1 Introduction

TrackAER (Rusch and Roesgen 2021) is an event-based measurement system for large scale flow diagnostics that provides real-time visual feedback on the flow field under investigation. On the hardware side it builds upon multiple event cameras (ECs) (Lichtsteiner et al 2008; Posch et al 2010) that are synchronized and calibrated to enable three-dimensional (3D) tracer particle tracking using photogrammetric reconstruction. Due to the functioning principle of ECs, motion information in a scenery is captured in the form of an asynchronous stream of events, effectively focusing on moving objects while suppressing static background data. Each event represents a threshold-crossing change in light intensity on the individual pixel level and is defined by its time of occurrence, the location of the “firing” pixel and a binary polarity indicating relative increase or decrease in light intensity. When compared to conventional particle image velocimetry (PIV) or particle tracking velocimetry (PTV) techniques, which capture image frames at high speed, the amount of data generated by ECs is vastly reduced such that online processing of the cameras’ data streams becomes viable. With a high temporal registration resolution equivalent to more than 10 000 frames per second, ECs allow to capture even high-speed flow phenomena. However, the advantages of ECs come at a significant cost: The asynchronous data stream is not readily processed with conventional algorithms that typically require images with absolute, frame-like (i.e., quasi-static) intensity information as input. As such, on the software side, the focus lies on the development of efficient and fast algorithmic pipelines that enable processing of event data under soft real-time constraints. Such algorithms need to be specifically tailored to the asynchronous and sparse nature of the event data stream to fully exploit the potential of ECs for flow diagnostics. While early ECs were severely limited in resolution (Lichtsteiner et al 2008), recent developments have pushed the number of pixels towards the 1 megapixel mark (Finateu et al 2020). As a result, it is now possible to resolve large measurement volumes on the order of several cubic meters with a spatial accuracy in the lower millimeter/upper sub-millimeter range, comparable to the resolution limit defined by the flow tracer size.

2 Related work

Event cameras have emerged in the neuromorphic engineering community already in the 1990s with the objective to mimic the function of biological retinae (Mahowald and Mead 1991). However, only since the 2000s have they evolved from lab prototypes to commercially available cameras (Lichtsteiner et al 2008). Currently, ECs are heavily researched in the areas of computer vision and robotics in an attempt to accelerate classical image-based computer vision tasks such as feature extraction, simultaneous localization and mapping, segmentation, and pattern recognition, among others. An extensive review of active research areas and applications of event cameras is provided in Gallego et al (2020). ECs have also been applied to fluid dynamics problems in a few studies already. Drazen et al (2011) tracked tracers in a pipe flow with a single EC and compared the results to reference measurements with a high-speed frame camera and a commercial particle tracking software. The main objective of their work was to identify and track the tracer particles directly from the event stream. No particle motion or flow field reconstruction was reported, and the two-dimensional (2D) test section spanned only a few square centimeters. The works of Ni et al (2012), Berthelon et al (2017) and Howell et al (2020) concentrated on the application of ECs to microscopic hydrodynamic flows seeded with micro beads. These scenarios all featured relatively small 2D observation regions on the order of a few square millimeters or less, captured by a single EC. Previous work that led to our current research was carried out by Borer (2014) and Borer et al (2017), who used multiple ECs to track and reconstruct the motion of helium-filled soap bubbles (HFSBs) in 3D in a wind tunnel environment. The approach relied heavily on 2D and 3D Kalman filters for tracking, which proved difficult to initialize when new tracer particles entered the field of view of the cameras. Limited by the comparatively low sensor resolution available at the time, measurement volumes up to about one cubic meter were processed. As the algorithms were implemented in MATLAB, data had to be recorded first and subsequently processed offline. Wang et al (2020) used a pair of ECs to capture the 3D motion of tracers in a centimeter-sized water tank. To extend the sparse measurement data to a volumetric domain and to correct for measurement errors they performed a constrained optimization on the measurement data enforcing criteria such as incompressibility and temporal consistency in the reconstructed flow field. The extensive optimization runs came at the cost of offline processing durations exceeding several minutes per frame. More recently, Willert and Klinner (2022) assessed the potential of ECs for offline PIV-like measurements (two-dimensional, two-components) based on reconstructed, frame-like data.

In our previous work (Rusch and Roesgen 2021), we introduced the TrackAER measurement system approach to facilitate true real-time, online analysis of flow fields across a wide range of flow speeds and in observation volumes exceeding several cubic meters. At the time, TrackAER was limited to measurements in 2D. Recent developments extend the system to 3D measurements by means of multiple ECs and add the extraction and visualization of velocity information. The measurement system now provides real-time visual feedback in the form of labeled Lagrangian particle tracks of the tracers, visualized both in the individual 2D, single camera views and in a joint interactive 3D viewer.

3 Measurement system

The TrackAER measurement system is a combination of tightly-coupled hardware and software components. Figure 1 provides a schematic overview of the system and its operating principle, considering here an aerodynamic test in a wind tunnel: Upstream of the test section the flow is seeded with tracers such as HFSBs that are advected by the flow. Multiple ECs are flexibly positioned at the measurement site. After an integrated camera calibration step, they are used to capture the motion of the tracers. It is noteworthy that no laser illumination is needed. Volumetric illumination by, e.g., light-emitting diodes (LEDs) suffices to generate detectable tracer signatures on the ECs due to the specular reflection on the surface of the HFSBs. This greatly enhances the practical applicability of the measurement system in various experimental environments (especially in wind tunnels) as laser safety concerns are eliminated. All cameras are connected to a host computer, which runs the front- and back-end software of the TrackAER system. The latter detects and tracks tracers in the individual cameras’ data streams, fuses the independent streams and reconstructs the positions of the tracer particles asynchronously. The back-end is complemented by a graphical user interface (GUI) providing means to adjust all algorithmic parameters as well as camera settings while the measurement system is running. At the same time, the front-end visualizes the reconstructed tracer tracks as 3D path lines that are color-coded with their respective velocity histories or other track (Lagrangian) properties. Thereby, an instant and continuous feedback is provided to the observer enabling immediate insights even into unsteady or transient flow fields.

Fig. 1
figure 1

TrackAER measurement system overview

Full size image

Algorithmically, TrackAER builds up an asynchronous “per-event” processing chain, starting from the raw event stream of each EC and ending with the rendering of detected tracer tracks in the 3D viewer. A schematic of the processing pipeline is depicted in Fig. 2.

Fig. 2
figure 2

Event-driven data processing pipeline

Full size image

First, the individual event streams of multiple ECs are filtered using criteria such as the event polarity, consistency checks or regions of interest. Then, at the individual camera level, real-time particle detection and tracking is performed utilizing the spatio-temporal coherence of events as they are registered by the sensor. Noise events are effectively suppressed while particles with a coherent visual signature are reliably detected (Rusch and Roesgen 2021).

Prior to each measurement, all cameras are calibrated to determine their internal and external imaging parameters. We utilize an active, blinking LED target for this task and conveniently employ the same 2D detection and tracking algorithm for target detection that TrackAER also uses for tracer tracking.

Once the photogrammetric calibration of the cameras is established, tracer particles are tracked in each camera’s view and temporal coherence as well as epipolar conditions constrain the 3D correspondence problem. Unambiguous matches are triangulated from multiple camera views, resulting in an asynchronous stream of 3D incremental tracer position changes. Spatio-temporal coherence is again the key to assign the 3D reconstructions to distinct tracer path lines. These discretized path lines are fitted with 3D curves so as to generate a description of continuous tracks of the tracer particles in closed analytical form. Whenever a new 3D reconstruction is assigned to an existing particle track, the 3D curve fit is updated recursively. Having access to the analytical representation of a spatio-temporally resolved tracer path line, the corresponding velocity (or higher order derivative) information is readily obtained by differentiation. As a final step, the reconstructed 3D path lines of the tracers are rendered in the TrackAER GUI. Compared to a simple 3D track reconstruction, the availability of both, tracer positions and arrival times, allows for an enhanced labeling of the path lines. In the Lagrangian sense, particle velocities and accelerations are readily available for color coding, but also certain topological features of the tracks may be rendered such as their curvature and torsion, which can be related to vorticity (Braun et al 2006).

The processing paradigm follows an event-based approach throughout the computational chain. The sequential and asynchronous processing of tracer events facilitates the independent operation of a variable number of cameras. An eventual upper limit is given by the number of cores available on the host computer. Potential buffer over- or underflows are effectively prevented by throttling the complete data pipeline, not only individual processing steps. Along the computational pipeline, the number of data packages forwarded from one processing stage to the next is reduced, while the information contained in the individual data packages is condensed and enriched. This ensures that the computationally more demanding processing steps at the multi camera level can afford to spend more execution time on a single data package.

A specific challenge arises during the 3D reconstruction step, where event candidates from the different cameras have to be matched. While spatial proximity can be assumed for proper matches via the epipolarity constraint, this does not strictly hold true for the temporal coincidence. The timestamping provided by the EC includes a random latency component (jitter) that requires the introduction of a temporal acceptance window and, thus, a form of interval search/recursion in the streamed data. To maintain the overall real-time capability, the size of this window has to remain limited, leading to a potential loss of matched events. Note, however, that this does not imply an increasing probability of incorrect matches.

On the analog side of the EC sensor processing, the detection of HFSBs poses some special challenges as well. The small optical signature of the tracers leads to a speed-dependent decrease in detection efficiency as fast moving tracers create reduced contrast changes in the pixels.

The TrackAER velocity measurement precision is presented in Fig. 3. Here, the velocity estimates derived from the track reconstructions in a simple, parallel and undisturbed wind tunnel flow are compared with readings of an independently calibrated pitot-static probe equipped with an industry-grade 30mbar differential pressure sensor. The dispersion of the TrackAER velocity estimates in terms of the interquartile range (IQR) settles in the 1.5–3.5% range of the corresponding median flow speed estimate, resulting in a probable error due to data dispersion of ±(0.75–1.75%) across the measurement range of approximately 5–60m/s. Note that due to the velocity computation being based on the ratio of two noisy measurement quantities – position & time – the error statistics are better captured by median and IQR rather than, say, mean and variance. Typical velocity distributions show a distinctly non-Gaussian characteristic, better described by a Cauchy distribution for which the statistical moments are not defined. The data dispersion is dominated by the imprecision in the temporal domain. The event generation and timestamp assignment processes of the ECs feature a significant jitter, which disperses the velocity estimates much more significantly than the spatial reconstruction errors. Similar verification experiments on the dynamic spatial reconstruction precision yield values consistently below 1mm across the complete cubic-meter-sized measurement volume. These values are smaller than the typical size of the HFSBs used.

Fig. 3
figure 3

Comparison of TrackAER velocity estimates with reference data from a pitot-static probe

Full size image

4 Results

To present different aspects of TrackAER’s performance and capabilities we look in Sect. 4.1 at the flow field of a large (meter-scale) turbulent free jet in air. This is followed by measurement results for a jet in cross-flow (Sect. 4.2) that highlight the real-time responsiveness of TrackAER. Finally, we process and visualize the flow in the wake of a delta wing aircraft model in a wind tunnel to demonstrate the inherent detectability of complex vortical structures (Sect. 4.3).

4.1 Free jet

An air purifier (Dyson Pure Cool Me) is modified with a cylindrical nozzle to produce a round turbulent free jet. The test setup is shown in Fig. 4.

Fig. 4
figure 4

TrackAER setup for the investigation of a large turbulent round free jet

Full size image

Based on the nozzle exit diameter of \(D =\) 7.14cm and nozzle exit velocites of up to 8m/s, Reynolds numbers of more than 36 000 are achieved. The jet flow is directed upward in a large room with approximately 5m ceiling height. To seed the flow, HFSBs are introduced in the exit nozzle of the device. Subsequently, these tracers are advected with the jet flow. Figure 5 depicts the measurement scenario with the resulting color-coded Lagrangian particle tracks. Note that an accumulation of more than 255 000 tracks is shown here since the online, real-time behavior of the measurement system cannot be adequately captured in a single still picture.

The actual measurement system does, indeed, continuously update the 3D view of the scenery as new tracer reconstructions become available without any limit on the duration of the measurement. Every 3D curve corresponds to a single time-resolved particle track – no spatial binning or temporal averaging is applied. The opacity of the rendered tracer tracks is reduced in Fig. 5 to reveal the inner flow structures of the jet in the densely reconstructed flow field. Close to the air purifier exit there are almost no tracks due to the air purifier casting a shadow. The chaotic turbulent structure of the jet is clearly visible (even more so, of course, in the real-time visualizations), as is the decay of the velocity profile with increasing radial distance away from the cylindrical axis. Furthermore, the jet speed decays along the cylinder axis with increasing distance from the nozzle exit. The accompanying radial spread of the jet conforms with the observed flow deceleration. The measurement volume covers about 3\(\times\)1\(\times\)1 m\(^{3}\).

Fig. 5
figure 5

Particle track reconstruction of an air purifier ejecting into quiescent ambient air. Shown are 255 000 tracks of a continuous five-minute measurement

Full size image

As a demonstration of the TrackAER capability to simultaneously resolve flows at very different spatial and temporal scales, we also present in Fig. 6 a detailed view of the averaged velocity field in a plane perpendicular to the jet axis at a height of \(x/D=15.4\). A residual swirl component is apparent due to the fan rotation in the purifier. The magnitude of the velocities amounts to only 2.5% of the simultaneously present axial flow velocity. The velocity field is accumulated and displayed here on a regular grid for better visibility.

Fig. 6
figure 6

Velocity field in a plane perpendicular to the jet axis at a distance of \(x/D =15.4\) from the nozzle exit

Full size image

4.2 Jet in crossflow

The same jet generator is placed inside the large subsonic wind tunnel at ETH Zurich (2m\(\times\)3m cross section) to create a jet-in-crossflow scenario. HFSB seeding is introduced upstream of the jet generator in the bulk wind tunnel flow. As the jet hits the much slower crossflow, the tracer particles are strongly accelerated and dispersed as is readily captured by the measurement system and visualized in Fig. 7.

Fig. 7
figure 7

Lagrangian flow field reconstruction of a jet in crossflow with \(u_{\text {jet}}/u_{\infty }=6.42\). 128 000 tracks gathered in a continuous measurement of four minutes duration are visualized

Full size image

While the flow upstream of the jet generator is parallel, laminar and of uniform flow speed, the downstream region is highly turbulent and chaotic. Due to the injected momentum, the overall flow speed downstream of the air purifier is increased significantly. Furthermore, a system of two counter-rotating vortices is developed by the jet-in-crossflow configuration. This complex flow topology is also captured by TrackAER as demonstrated in Fig. 8.

Fig. 8
figure 8

Streamline reconstruction of the jet in crossflow as seen from downstream of the jet generator. Two counter-rotating vortices are generated. The color-coding indicates the rotational sense and strength along the streamlines

Full size image

The equivalent rapid response to a temporal change in flow speed is difficult to visualize in printed form. For this, we refer to the supplementary information of this paper, where a video with the live visualization of turn-off/turn-on cycles of the jet is presented (Online Resource 1).

4.3 Wake of a delta wing fighter jet model

The final example showcases the flow behind an aircraft model in the large subsonic wind tunnel at ETH Zurich. The specific aircraft design is representative of a fifth generation fighter jet and features a delta wing optimized for maneuverability at high angles of attack. The wing geometry leads to the evolution of a complex vortex system above the wing that maintains controllability in the post-stall regime. At an angle of attack of 15\(^{\circ }\), the model is tested at varying bulk flow speeds to investigate the real-time capabilities of the TrackAER measurement system as well as its transient behavior at different flow speeds. The test setup is shown in Fig. 9.

Fig. 9
figure 9

TrackAER setup in ETH Zurich’s large subsonic wind tunnel to investigate the flow in the wake of a delta wing fighter jet model

Full size image

An HFSB seeder injects tracers into the free stream flow upstream of the wind tunnel contraction. The seeder outlet’s lateral position is chosen such that the tracers target one half of the model’s delta wing. TrackAER is set up around the aircraft model to observe the wake region. At a bulk flow speed of 5m/s, Fig. 10 depicts TrackAER’s virtual 3D scene including a computer-aided design (CAD) model of the aircraft for easier spatial referencing of the flow features. A sliding time window containing 20s of the dynamically updated, real-time rendered tracer paths and accompanying color-coded velocity information are shown. Note that despite the low flow velocities, the model’s sharp leading edges lead to the desired flow separation and generate a realistic vortex system. Again, the real-time behavior of the measurement system cannot be delivered in the form of static images. The actual TrackAER measurement was performed continuously for 20 min in this case, demonstrating the unlimited processing (and recording) time of the system.

Figure 10 displays the scenario from three different points of view: Figure 10a shows the entire model from upstream along with the visualization of the vortex system. Figure 10b also depicts the situation from upstream, but with the view zoomed in to capture more details of the vortex system. Figure 10c depicts the flow field as seen from the side. Looking downstream, on the left-hand side of the large primary delta wing vortex, Fig. 10a, b additionally reveal a less prominent, secondary vortex detaching from the wing tip that interacts with the primary delta wing vortex downstream of the aircraft. This observation conforms with the two-stage compound delta wing geometry of the airplane model.

Fig. 10
figure 10

Visualization of the vortex system in the wake of a fighter jet wind tunnel model at \(u_{\infty } =\)5m/s. The accumulated particle tracks correspond to a sliding time window of 20s. Local flow speeds are color-coded

Full size image

As indicated, some of the test cases shown in this section were specifically performed at relatively low flow speeds to generate accompanying video material, in which the tracer movement and online generation of particle tracks is still human-perceivable. We have successfully performed measurements with TrackAER at flow speeds up to 60m/s in the subsonic wind tunnel at ETH Zurich. Beyond that speed, fading tracer signals massively reduce the quality of the data and cause reconstruction drop outs.

Further statistical processing of the data, such as the online accumulation of track information in certain measurement planes is possible as well. Since this type of co-processing can occur independently of the real-time data stream, however, it is not an essential part of the event-based processing pipeline and is not further elaborated in the present context.

5 Conclusion

We have presented a novel flow measurement system, TrackAER, which uses multiple ECs to reconstruct 3D, time-resolved flow fields in large measurement volumes with real-time visual feedback and analytical capabilities. By efficiently processing the asynchronous event data streams of multiple cameras and subsequently fusing them into 3D information, we are able to visualize the resulting Lagrangian particle tracks with velocity information online in a virtual 3D scene. Three flow scenarios are presented that demonstrate TrackAER’s capability to accurately capture topological characteristics of the flow fields and to provide insights on the turbulent, unsteady flow structures present. A large range of flow speeds from well below 1m/s to approximately 60m/s is covered by the measurement system. It should be stressed that we cannot fully reproduce in printed form the online behavior of the measurement system through still pictures and by providing an accumulation of time-resolved tracks as the final visual result.

Our future work will aim at reconstructing even larger flow fields with higher precision and expanded Lagrangian diagnostics. Furthermore, we will extend the application of the system to specific flow field scenarios where other experimental techniques are inapplicable due to storage, bandwidth or speed limitations, such as extremely slow flow phenomena or high-speed transient flows.

6 Supplementary information

A video of the TrackAER measurement system in operation is included in the supplementary material (Online Resource 1). It shows the jet-in-crossflow scenario described in Sect. 4.2, where the jet generator is turned on and off dynamically demonstrating TrackAER’s ability to resolve even transient flow phenomena in real-time. Visualized are the dynamically updated tracer tracks corresponding to a sliding time window of 1s. Next to a screen capture of TrackAER’s virtual 3D viewer, a conventional video is presented as picture-in-picture overlay.