1 Introduction

Event-based vision (EBV), also termed dynamic vision sensing (DVS) or neuromorphic imaging, is a new upcoming field within the field of computer vision. Contrary to conventional frame-based imaging, EBV only records changes of image intensity (i.e., contrast changes) on the pixel level, triggering a positive event (\(+1\)) for increasing intensity and a negative event (\(-1\)) for a decreasing intensity change. The typical threshold of the intensity-change trigger is on the order of 20% but can be fine-tuned. As the pixels on the detector respond individually, the events appear asynchronously throughout the detector area resulting in a continuous stream of asynchronous data, with each event datum \(E_i = E_i({\mathbf {x}},t,p)\) consisting of pixel coordinates \({\mathbf {x_i}} = (x_i,y_i)\), a time stamp \(t_i\) and a polarity \(p_i \in \{+1,-1\}\) indicating the sign of the intensity change.

Original prototype development of the technology dates back to work of Mahowald and colleagues at the California Institute of Technology in the 1990’s and was initially referred to as silicon retina (Mahowald 1992) as the intention of the imaging approach was to mimic the function of the eye’s retina. First practical implementations of EBV resulted from work at University of Zurich as well as ETH Zurich around 2008 (Lichtsteiner et al. 2008; Posch et al. 2014). In recent years, several ready-to-use cameras and sensor evaluation kits based on the EBV technology have become commercially available. This has broadened the range of applications as testified in a steadily increasing number of publications (see, e.g., Robotics and Perception Group 2022) and also has made the present feasibility study possible. For recent reviews of event-based sensing technology and underlying concepts, the reader is referred to Gallego et al. (2022) and Tayarani-Najaran and Schmuker (2021).

The application of EBV for the visualization and measurement of fluid flows is by no means new. Initial work was performed by Drazen et al. (2011) on particle tracking velocimetry (PTV) of dense particles in a solid–liquid two-phase pipe flow using an EBV sensor of 256 \(\times\) 256 pixels and continuous-wave (CW) laser illumination at 5W. Ni et al. (2012) used an EBV array of 128 \(\times\) 128 elements to demonstrate micro-particle tracking (\(\upmu\)PTV) with 12 \(\upmu\)m microspheres and were able to detect Brownian motion. Measurements in microchannels based on event-based imaging of 1–10 \(\upmu\)m fluorescent beads were performed by Howell et al. (2020) at exposure times down to 100 \(\upmu\)s and velocities exceeding 1 m/s. First PTV measurements in an air flow were performed by Borer et al. (2017) using three synchronized EBV cameras (128 \(\times\) 128 pixels) to track helium-filled soap bubbles (HFSB) in volumes up to about 1 m side length using white-light LED arrays for illumination. The flow was only sparsely seeded allowing individual particles to be tracked throughout the volume with final data sets containing up to O(1000–10,000) tracks. More recently, Wang et al. (2020) implemented a stereoscopic EBV system using two event cameras each with 480 \(\times\) 360 pixels. They reconstructed three-dimensional tracks using 2d tracking results from each of the cameras. Their flow experiment consisted of a small hexagonal cell with stirrer inducing a swirling flow containing O(100 \(\upmu\)m) polystyrene spheres.

The present work aims at assessing the potential of EBV for more “traditional” planar 2d–2c flow measurement, ideally being able to extend the findings to full 3d–3c measurements. In particular, much higher seeding densities than in previous studies are investigated and are made possible through recent advances in EBV hardware development. Two commercially available event-based imaging cameras are used for the investigations with their specifications provided in Table 1. While the data presented in the remainder of the article is acquired with the high-resolution version, both cameras produce event data streams of similar quality with a detailed comparison beyond the scope of the material presented herein. Measurements are demonstrated in both water and air using the same seeding particles and densities as used for conventional PIV. To the best of the authors’ knowledge, EBV-based fluid flow measurements so far have not been demonstrated at considerably higher particle image densities.

The paper is structured as follows: after briefly explaining the event generation process, various recently introduced motion detection schemes for EBV are described with regard to their suitability for fluid velocity estimation of flows seeded with small particles. Two approaches are chosen to be adapted to the present application. Section 3 introduces measurements on a simple water flow and provides illustrative, real-world examples of both the acquired event image data and the application of the motion detection schemes. In Sect. 4, the performance of the algorithms is assessed using a known motion (spinning disk) and scatter plots of a turbulent flow to highlight possible systematic sources of error. The current limits of event-based imaging velocimetry (EBIV) are then tested using an air flow with a wide range of velocities and scales (Sect. 5). The discussion section provides an overview of both the advantages and disadvantages of the EBIV technique, followed by concluding remarks concerning possible improvements and further applications.

Table 1 Specifications of cameras used for the present study
Full size table

2 Event-based vision and optical flow

2.1 Event generation and particle imaging

The event generation for a single pixel of the event image sensor is schematically outlined in Fig. 1 using a Gaussian profile to represent the intensity variation over time. The example considers the time-varying light I(t) scattered by a single particle as it passes the sensitive area (photodiode) of the pixel (Fig. 1a). The resulting photocurrent is fed into a contrast detection unit within the pixel that evaluates the intensity contrast C(t) with respect to a reference intensity \(I_{\mathrm {ref}}\):

$$\begin{aligned} C(t) = \frac{\log I(t) - \log I_{\mathrm {ref}}}{\log I_{\mathrm {ref}}} \end{aligned}$$
(1)

where the reference intensity generally is the intensity measured at the previously generated event, \(t_i\) in Fig. 1. An event is triggered when the contrast change C(t) exceeds a certain threshold level:

$$\begin{aligned} \Vert C(t) \Vert > C_{\mathrm {thr}}. \end{aligned}$$
(2)

Typically, this threshold is in the range (10–20%) and is part of the camera’s parameter adjustment. Also the threshold may differ for positive and negative event generation, resulting in asymmetric response. Due to the logarithmic differentiation, the contrast measurement in low light conditions becomes less reliable and is more susceptible to dark current noise, that is, spontaneously generated photoelectrons may trigger “false” events. This can be minimized through a sensitivity adjustment of the pixel.

Figure 1 illustrates a few important aspects in the context of particle imaging: for one, a particle can trigger several events as it comes into the pixel’s view. This may introduce uncertainties in the motion estimation because the actual time of a particle’s appearance is not clearly defined—preconditioning of the event data such as using only the first event of such a burst of events may be necessary. Then, the recorded events by themselves give no indication with regard to the location of the particle image center (maximum intensity). Rather, the events are associated with the edges of the particle images, that is, the regions of significant intensity gradients. This may be of relevance when tracking individual particles, especially so if their imaged size increases beyond a few pixels.

Fig. 1
figure 1

Illustration of the event generation process for a single pixel of the event sensor: a time-varying intensity signal seen by the photodiode of the pixel, b log intensity signal processed by the contrast detection unit of the pixel, c asynchronous event stream produced by the pixel after reaching a certain contrast threshold C

Full size image

To illustrate the concepts of motion estimation from asynchronously recorded event data, Fig. 2 shows the motion of a single particle P both through planar space (\({\mathbf {x}} = (x,y)\)) as well as in the space-time domain (Fig. 2b) and the trail of discrete events \(E_i({\mathbf {x}},y,t)\) it may produce in space-time (Fig. 2c). Assuming a locally constant velocity, a first-order estimate of the particle velocity \({\mathbf {V}}\) is given by the slope of the particle path (i.e., \({\mathbf {V}} = \Delta {\mathbf {x}} / \Delta t\)) in the space-time domain, which is the sought-after quantity for which a suitable estimation algorithm has to be found. Figure 3 provides a 3d rendition of actually recorded events of particles images in a water flow (see Sect. 3).

Fig. 2
figure 2

Illustration a single particle P moving at a fixed velocity \(V = (v_x,v_y)\) in 2d space (a), in \(x-t\) space (b) and trail of events produced the particle in \(x-t\) space (c)

Full size image
Fig. 3
figure 3

3d rendition of a sub-volume of events of size 40 \(\times\) 40 pixels and 20 ms duration. Blue color coding marks past events while red indicates later events. a isometric view; b view aligned with x-axis; c view direction chosen to align events into clusters

Full size image

2.2 Suitable processing algorithms for EBIV

Within the computer vision community, considerable research has been performed in recovering the optical flow from conventional frame-based image sequences and, more recently, event-based image data. In part, these efforts are aimed at transferring the concepts from frame-based imaging to event-based imaging, but have only partially been successful, which is mostly due to the fact that the well-established optical flow approaches rely on brightness constancy (Horn and Schunck 1981; Lucas and Kanade 1981). In event-based imaging, image brightness per se is not available, such that some of the proposed optical flow algorithms rely on reconstructed image intensity as an intermediate quantity (Benosman et al. 2012) or recover the brightness field along with the optical flow (Bardow et al. 2016).

The output of the optical flow algorithm can be classified as “dense” or “sparse,” respectively, providing motion estimates for every pixel or on selected pixels only. Event image data produced by discrete, small particles initially only provide sparse motion fields where the sparseness directly corresponds to the source density, namely, the imaged particles. To obtain a dense field (for every pixel) requires a smoothness regularization that results in a continuous variation of the velocity estimate within the sampled volume. Ideally, the estimated velocity field predicts the motion of the individual particles taking into account the spatiotemporal filtering introduced by the discretely sampled particle image data.

The motion estimation approach chosen here follows the concepts used in the computer vision community for optical flow estimation. Preferably, the optical flow estimation algorithm should not require to explicitly reconstruct image intensity, that is, it operates only by using the spatiotemporal resolved events \(E_i = E_i({\mathbf {x}},t,p)\). Event data generated from imaged objects (i.e., cars, people, trees, buildings, etc.) typically consist of broad “swaths” of events that, when aligned, result in sharp edges. By fitting a plane in x–y–t space to these events, Benosman et al. (2014) show that optical flow data can be retrieved but the approach has its limitations on more complex scenes (Rueckauer and Delbruck 2016). Imaged particles, on the other hand, typically only cover a few pixels when properly focused. In fact, for the PIV technique an optimal particle image has a size of 2–3 pixels [\(e^{-2}\) diameter (Adrian and Westerweel 2011; Raffel et al. 2018)]. This makes a plane fitting approach challenging. The line-like nature of the events created by the particles suggests the use of algorithms suited for the detection of line segments such as the Hough transform (Bagchi and Chin 2020), but may be computationally expensive. Another approach relies on the convolution of the event data with separable space-time filter kernels to estimate the motion from the maximum response (Brosch et al. 2015). Some optical flow techniques for frame-based image data, such as the Lucas–Kanade algorithm (Lucas and Kanade 1981), rely on structure tensors (based on image intensity gradients) while others employ principal component analysis to extract motion baring elements from the image data. In principle, these approaches could also be extended to the space-time event-based data but have not been investigated in the scope of this work.

At present, one of the most suitable approaches for EBIV seems to be the so-called motion compensation introduced by Gallego et al. (2018). The idea behind motion compensation is to align the events such that they line up using a linear skewing (warping) of the sampled \(x-y-t\) sub-volume, assuming constant (linear) motion within the sampled volume. Conceptually this can be visualized by choosing a viewing direction onto a \(x-y-t\) sub-volume that makes the events in this volume appear to be lined up, as illustrated in Fig. 3c. Applied to imaged scenes the motion compensation in effect removes the motion blur, hence its name.

In practice, the motion compensation is performed iteratively on sub-volumes of dimension \(N_X \times N_Y \times T\) using velocity candidates \(\mathbf{V }'\) that map (“warp”) the original events within the sub-volume according to

$$\begin{aligned} \mathbf{x }'_i = \mathbf{x }_i + \mathbf{V }' (t_i - t_0). \end{aligned}$$
(3)

The resulting set of \(N_e\) mapped events is then used to create an image of warped events (IWE) by summing all events that coincide with a given pixel \(\mathbf{x }\):

$$\begin{aligned} {\mathbf {I}}(\mathbf{x },\mathbf{V }') = \sum _{k=0}^{N_e} \delta (\mathbf{x } - \mathbf{x }'_k) . \end{aligned}$$
(4)

Here, the function \(\delta\) can take the form of a nearest neighbor operation or higher interpolation scheme. For an optimal choice of \(\mathbf{V }'\) the number of coinciding events will concentrate into a reduced number of pixels \(\mathbf{x }\) within the IWE—the imaged particles will appear concentrated into a few small clusters, as illustrated in the third column of Fig. 4. Here the intensity variance \(\sigma ^2_\mathbf{I }\) serves as a measure of image contrast:

$$\begin{aligned} \sigma ^2_{\mathbf {I}} = {\mathrm {Var}}(\mathbf{I }(\mathbf{x },\mathbf{V }')) = \frac{1}{N_{\mathrm{p}}} \sum _{i,j}^{N_{\mathrm{p}}} (I_{i,j} - \mu _{\mathbf {I}})^2 \end{aligned}$$
(5)

where \(N_{\mathrm{p}} = N_X \times N_Y\) is the number of pixels comprising the IWE and \(\mu _{\mathbf {I}}\) its mean intensity. As investigated in detail by Gallego et al. (2019) a wide variety of so-called loss functions (objective functions) can be used to maximize local contrast; image variance was found to be among the most reliable, is simple to implement and is used in the present implementation.

An alternative method of retrieving the optical flow from event data is proposed by Nagata et al. (2021) who claim that their method is more robust in comparison to the previously described iterative variance-based motion compensation approach. In their case the local optical flow is obtained from two sub-volumes at the same location but slightly offset in time \(\tau\) and measures the consistency of the time-surfaces formed by the events in \(x-y-t\) space between the two samples. The minimization-based algorithm relies on spatial derivatives of the time surface and requires a smoothness regularization term to stabilize the minimization.

For the present study, the algorithm by Nagata et al. (2021) has not been implemented. Instead, we propose to use their concept of sampling the \(x-y-t\) volume with two sub-volumes separated by time \(\tau < T\) and apply a motion estimation scheme that is more suited for particle image data. The basic principle is outlined in Fig. 5: the two sub-volumes are subdivided into \(N_T\) equal time-slices of duration \(T/N_T\). Events within each time-slice are then combined and result in \(N_T\) planes of events within \(x-y\) space for each of the two samples. Next, the cross-correlation is computed for each pair of event planes. Finally, the \(N_T\) cross-correlation planes are summed to form a single combined correlation plane. The position of the maximum correlation value with respect to the origin represents the most likely particle displacement within the sampled volume. Division of the correlation peak displacement by the sample time-offset \(\tau\) yields a first-order estimate of the “pixel velocity” within the sampled sub-volume.

For both approaches the velocity estimation is performed symmetrically around time \(t_0\) making them central difference schemes which are inherently second-order accurate (Wereley and Meinhart 2001). To improve sub-pixel performance, 3-point parabolic fits or 3-point Gauss fits are, respectively, applied in X and Y direction using the neighboring values of the maximum variance or maximum correlation.

In order to retrieve velocity data across the field of view, the simplest algorithm involves a patch-wise processing very similar to conventional PIV. The sample size is fixed throughout the field of view and the interrogation is performed on a regular grid regardless of the local particle image (e.g., events) concentration. For the motion compensation approach a sufficient range of “pixel velocities” needs to be defined a priori to ensure the capture of the variance peak. Similarly, an adequate time-offset \(\tau\) has to be chosen for the sum-of-correlation scheme to capture events of the same particle(s) in both \(x-y-t\) sub-volumes.

The logical extension of these single-pass algorithms is a grid-refining scheme that starts coarse and iteratively moves into finer grid resolutions, both spatially and with increasingly finer velocity resolution. Here deformation schemes that allow a continuous variation of the velocity estimate within the sampled volume can be directly imported from state-of-the-art PIV processing algorithms.

When converged, the iterative, patch-wise particle motion detection scheme effectively results in clusters of pixels that are associated with the motion of single particles (see, e.g., Fig. 9c, g). Therefore, it is a (nearly) trivial step to single out the events for an individual particle image from the original event data set. These events can then be used to recover the track of the given particle in both space and time using an adequate fitting scheme [see, e.g., methods used for 3d Shake-The-Box Lagrangian tracking, STB (Gesemann et al. 2016; Schanz et al. 2016)].

In the following section implementations of both the motion compensation algorithm and the sum-of-correlation scheme are used to recover the flow field of a simple water flow experiment.

Fig. 4
figure 4

Principle of patch-wise motion compensation; a events are sampled in volume of spatial dimension \(N_X\) and time interval of T, b motion compensation is performed iteratively using velocity candidates \(v_k\); c optimal velocity \(\hat{v}\) is found when events line up at a constant position \({\mathbf {x}}_{i,{\hat{\mathbf {v}}}}\)

Full size image
Fig. 5
figure 5

Principle of patch-wise velocity estimation using a temporal offset; a events are sampled in two volumes of equal spatial dimension \(N_X\) for a time interval T, but separated by a time delay of \(\tau\), b mean displacement \(\Delta x\) is determined by summing \(N_T\) separate cross-correlations of events for different time slots \(t_k\). Mean velocity is given by ratio \(\Delta {\mathbf {x}}/\tau\)

Full size image

3 EBIV applied to a water flow experiment

A water basin of about \(400\times 250 \times 60\,\text {mm}^3\), is seeded with nearly neutrally buoyant silver-coated, hollow glass spheres of about \(10\,\upmu\)m diameter. A small pump on the side introduces a globally recirculating flow with underlying turbulent structures (see Fig. 6). The particles are illuminated by a \(\approx\)100 mm wide laser light sheet with about 1 mm waist thickness created with a laser (Kvant Laser, 4W max., \(\lambda\) = 520 nm) operating in CW mode. The light sheet is induced through the vertical side of the tank and the camera is rotated by 90\(^\circ\) such that the glass wall appears at the bottom of the recorded images.

Figure 7 shows an “image” of events compiled from a time interval (10 ms) of recorded events sampled on a field of view (FOV) of 54 \(\times\) 30 mm\(^2\) at a magnification of 42.3 \(\upmu\)m/pixel. The data was acquired at a data rate of 2 \(\times\) 10\(^7\) events per second which is two orders of magnitude below the maximum supported by the camera. The reconstructed image of Fig. 7 contains a total of about 130,000 events that are color-coded using their respective time of occurrence. Events marked blue occurred in the “past” while red events occur in the “future” in reference to time \(t_0 = 0\), for which events are coded in white. Note, that the wall through which the light sheet is introduced is not at all visible, which is a direct result of the lack of temporal intensity variations at the glass–water interface. A dashed line in Fig. 7 indicates the approximate position of the wall. Animations of the raw event image data and recovered velocity fields are provided in the supplementary material.

The two small squares in Fig. 7 indicate samples of 40 \(\times\) 40 pixels. The areas surrounding them are highlighted in Fig. 8. Event tracks produced by the particles have a width of 2–5 pixels. The procedure of the motion compensation algorithm is further demonstrated in Fig. 9. Here, Fig. 9a, e show the sampled events color-coded with the time of arrival. For better visibility, Fig. 9b, f show only the currently “active” pixels in the sampled sub-volumes. By applying motion compensation, these pixels are aligned such that a maximum of events coincide, as shown in Fig. 9c, g. In order to achieve this alignment, a measure based on the variance with respect to the space-time warped event data is used. Figure 9d, h show the distribution of the intensity variance spanned by a range of possible “pixel velocity” candidates. These variance intensity maps exhibit clearly detectable peaks that coincide with the most probable velocity estimate. In effect, this is very similar to the 2d cross-correlation map used in conventional PIV processing.

Figure 10 illustrates the sum-of-correlation approach on the same sample as in Fig. 9. Here, the summed correlation plane is akin to the correlation planes obtained with conventional 2d–2c PIV processing, exhibiting, for instance, multiple correlation peaks that appear when events of a given particle correlate with those from a different particle. As the number of imaged particles within the sampled volume decreases, the likelihood of particle mismatch increases and the “false” correlation peaks become more prevalent. Figure 10f also shows one of the main advantages of the sum-of-correlation method: any time-slices without events produce no correlation signal and have no contribution to the final correlation sum. Also noise in the individual correlation slices is averaged out.

Fig. 6
figure 6

a Photograph of a simple water flow EBV imaging setup in the “home-lab” of the first author involving a small water tank, laser light sheet and event-based imaging camera. b Image of a ruler moving sideways recorded by the event camera illustrating the event generation within a period of 10 ms (white pixels are positive (“on”) events, black pixels are negative (“off”) events)

Full size image
Fig. 7
figure 7

Image constructed from 10 ms worth of positive events triggered by particles moving in a tank of water and illuminated by a CW laser light sheet. The mean flow is from left to right. The black rectangles indicate sample areas of 40 \(\times\) 40 pixels used in the following figures. The thin line near the bottom indicates the position of the glass tank wall which is essentially invisible due to the absence of intensity events

Full size image
Fig. 8
figure 8

Zoomed-in portions of Fig. 7. The black rectangles indicate sample areas of 40 \(\times\) 40 pixels

Full size image
Fig. 9
figure 9

Sampled events from Fig. 7 and corresponding images of warped events (IWE) after applying motion compensation. Black arrows in leftmost sub-figures indicate the velocity estimate determined by finding the maximum in the 2d variance distribution

Full size image
Fig. 10
figure 10

a, b Sampled events from Fig. 7 obtained with time separation of \(\tau = 3\) ms (color map as in Fig. 8). d, e \(x-t\) projection of both samples. f \(x-t\) view of \(N_T = 40\) individual cross-correlations over 10 ms sample interval. c Cross-correlation map obtained by summing correlation planes below. Black arrow in upper left sub-figure indicates the velocity estimate determined by finding the maximum in the 2d cross-correlation map

Full size image

In PIV processing, the displacement should be limited to about one-fourth the sample dimension to warrant an optimal displacement recovery by restricting the in-plane loss of particle image pairs (so-called one-quarter rule, see, e.g., Adrian and Westerweel 2011; Raffel et al. 2018). At first sight, this sort of restriction is not present for the described variance-based motion compensation algorithm. This can be observed in Fig. 9e which shows a displacement of about 25 pixels in a sample of 40 \(\times\) 40 pixel. The event data shown in Fig. 7 contains particle image tracks whose length exceeds 30 pixel. Nonetheless, fast moving particles will produce a reduced number of events in the sampled space-time sub-volume which in turn reduces the reliability of the velocity estimate. The sum-of-correlation approach, on the other hand, should adhere to the “one-quarter rule” in order to reliably line up the event tracks from both volumes. Faster moving particle images are captured by reducing the separation \(\tau\) or by increasing the size of the sample.

As an example, the result of a single-pass processing of the water flow is provided in Fig. 11 using a fixed sample size of 40 \(\times\) 40 pixels on a time-slab of 10 ms. For the motion compensation algorithm the velocity search domain is set to \(v_x = \left[ -1,+5\right]\) pixel/ms and \(v_y = \left[ -2,+2\right]\) pixel/ms at an initial sampling resolution of 0.250 pixel/ms which is iteratively refined on a 5 \(\times\) 5 neighborhood around the maximum variance until the resolution is below a cutoff value of 0.05 pixel/ms. The sum-of-correlation approach uses a temporal offset between the samples of \(\tau\) = 2.5 ms (1/4th of the sample interval). Each pair of \(x-y-t\) sub-volumes is resampled to \(N_T\) = 20 planes from which 20 individual correlation planes are calculated and subsequently summed. Figure 11e, f present the velocity field obtained using a conventional PIV processing algorithm. This is realized by first generating conventional PIV recordings from the event data simply by combining events from a given time interval (1.5 ms) into a single image frame using the time stamps as intensity values. The temporal spacing between the image frames is chosen at \(\Delta t\) = 2.5 ms which equates to a frame rate of 400 Hz. No further image processing, such as filtering or intensity thresholding, is performed prior to applying an iterative, coarse-to-fine cross-correlation scheme using 4 frames per time-step (Lynch and Scarano 2013).

Overall, the agreement between the results of the three processing approaches is very good; the differences are on the order of 0.1 pixel/ms. The data obtained with the motion compensation scheme appears noisier in comparison to the other two approaches but exhibits the least amount of spurious (outlier) vectors. The sum-of-correlation scheme has difficulty in the quiescent regions whereas the conventional PIV processing exhibits most of the outliers near the wall (\(y \approx\) 40 pixel). Due to their reduced motion, particles in the quiescent regions produce a reduced number of events such that the algorithms fail to produce reliable results in these areas.

The processing speed of the algorithms depends on numerous parameters and varies linearly with the number of sampling points and sub-sample size in space and time. For the present data, the sum-of-correlation approach requires about 1 s per time-step (single thread processing on a 4 GHz Intel CPU) and is about five times faster than the motion compensation approach.

Fig. 11
figure 11

Velocity fields obtained with the proposed processing algorithms using event data shown in Fig. 7. Top row: a, b motion compensation approach, middle row: c, d sum-of-correlation approach, bottom row: e, f 4-frame PIV analysis using images created from events of duration 1.5 ms. Physical velocity is obtained by multiplying with the scaling factor, \(m = 0.021\) mm/pixel. Animations of the data obtained with the sum-of-correlations approach are provided in the supplementary material

Full size image

4 Performance assessment

While the previously shown measurements of the water flow demonstrate the feasibility of EBIV, little can be learned regarding measurement accuracy and uncertainty. One approach to at least assess the influencing factors is to use an experiment or simulation with known characteristics. This can be either achieved using synthetic data, which is a challenge in itself (see, e.g., Hu et al. 2021), or by recording event data from a clearly defined flow. In the present case, the “flow” is a solid body rotation of constant angular speed achieved by placing a plate of fixed particles on a turn table (i.e., record player). Sufficiently small particles are provided by reflective spray containing small glass spheres of about 50 \(\upmu\)m diameter. The lens of the event camera is fitted with a ring of white-light LEDs such that the illumination is essentially on-axis with the lens, thereby achieving optimal intensity of the retro-reflective glass beads. Event data is acquired for the inner radius extending out to \(R \approx\) 60 mm and the outer radius in the range \(85< R < 150\) mm with the turn table operating at 45 rpm.

Acquired event data is processed along the radius using the previously described motion compensation algorithm with a fixed sample size of 20(W) \(\times\) 40(H) pixels for the smaller radii and increased to 20(W) \(\times\) 80(H) pixels for \(R > 85\) mm to capture a sufficient number of events. Velocity information is determined by sampling the 2 s long records with sampling intervals T of 5, 10 and 20 ms. Figure 12 presents both the raw velocity estimates in the form of scatter plots along with a linear fit to the data. In agreement with the underlying solid body rotation, the velocity increases proportional with increasing radius. The deviation of the data from the linear fit is plotted with red dots and exhibits a normal distribution. The scatter increases with increasing radius and decreases with increasing sampling time T. The measurement of the solid body rotation clearly shows that measurements up to (and beyond) 12 pixel/ms (or 12,000 pixel/s) are feasible and show no systematic deviation.

In effect, the sampling time defines how many events of a particle are captured and used for the estimation of the velocity. Unlike PIV, in-plane loss of particle images is not really relevant here because other particles come into view and also contribute to the velocity estimate. For the 10 ms sampling condition the dynamic velocity range (DVR, Adrian 1997) can be estimated at (3.5/0.020) = (175\(\div\)1) which is comparable to that of 2d–2c PIV with sampling at 32 \(\times\) 32 pixel. While the rms deviation reduces by 50% when doubling the sampling time from 5 to 10 ms, the reduction is not as significant when doubling the sampling time from 10 to 20 ms. The presented test case of “particles” rotating with constant angular speed does not mimic actual fluid flow. Variations in particle path curvature or particle acceleration/deceleration along with other effects will undoubtedly contribute to uncertainty in velocity estimation of fluid flows. It is clear, that additional Monte Carlo simulations and modeling of the EBV imaging process (which can be described as a Poisson distribution) are required to further quantify the factors influencing measurement uncertainty. While not presented here, it should be noted that the sum-of-correlation approach is not able to capture the entire range of radii without adjusting the temporal sample offset \(\tau\) to warrant that the correlation peak is captured at higher velocities.

Fig. 12
figure 12

Solid-body rotation data obtained by imaging small particles on a turn table rotating at 45 rpm. The same data set is processed with the motion compensation algorithm using different event samples times: 5, 10 and 20 ms. Left column (a, c, e) is processed with samples of 20 \(\times\) 40 pixel, the right (b, d, f) with 20 \(\times\) 80 pixel. Data in the grayed out regions are excluded

Full size image

Another common practice of assessing PIV data quality is to plot the histograms of the displacement data in order to uncover artifacts such as pixel locking. Using 1s worth of processed event data from one of the experiments (turbulent water flow, see Sect. 3) both 2d scatter plots and projected (1d) histograms of the velocity data are provided in Fig. 13. The first four sub-figures show velocity scatter plots obtained with the motion compensation approach (left) and sum-of-correlation approach (right) for time samples of T = 10 ms (top row) and T = 20 ms (second row). The motion compensation data results from an iterative scheme that reevaluates the area around the most probable velocity value is the variance map using a grid of 5 \(\times\) 5 samples. The cutoff is chosen at 0.025 pixel/ms. All sub-figures in Fig. 13 clearly show effects similar to pixel locking found in PIV. In the present case, the imaged turbulent flow should show a uniform distribution; the gridded nature of velocity clustering is non-physical. In particular, the motion compensation algorithm produces pronounced star-shaped features that can also be observed in the variance maps of Fig. 9, right. The artifacts may be rooted in an inadequate peak finding scheme of the variance maximization that is unsuited for the rather broad peak of the variance map. Gradient-based optimization methods are likely to improve the performance but have not been implemented in the present study. Unlike pixel locking in PIV, these clustering artifacts are believed to be systematic errors that are purely associated with the respective processing algorithms. This, however, does not imply that event-based imaging does not suffer other shortcomings that propagate into the velocity estimates. In order to further characterize the sources of uncertainty, the event-based imaging of small particles has to be modeled using, for instance, approaches such as put forth by Hu et al. (2021).

Noteworthy in Fig. 13 is the absence of velocities near zero. This is due to the fact, that particles at rest do not produce sufficient events to be registered. This is clearly a shortcoming of the measurement technique for which no obvious solution is presently available. In principle by increasing the sampling time T more events can be collected allowing velocities closer to zero to be captured.

Fig. 13
figure 13

Histograms obtained from 35 reconstructed flow fields of the turbulent water flow using three different algorithms for time intervals of T = 10 ms (a, b) and T = 20 ms (c, d). Subfigure (e) shows result obtained using multi-frame PIV applied to images separated by 2.5 ms, each containing 1.5 ms of events

Full size image

5 EBIV measurement in air

Figure 14 shows a photograph of measurements performed in a small wind tunnel with a square cross section of 76 \(\times\) 76 mm\(^2\) and 1.5 m length. A steel cylinder of 14 mm diameter spanning the height of the channel is placed on the centerline. At a bulk flow velocity of \(U_{\mathrm{b}} \approx 1.5\) m/s the diameter-based Reynolds number is \({\mathrm {Re}}_{\mathrm{D}} = 1400\), which is well beyond the transition to turbulence resulting in a turbulent wake. At the upstream end of the wind tunnel the flow is seeded with 1 \(\upmu\)m paraffin droplets produced by an atomizing aerosol seeder (Laskin type). The droplets are illuminated by a CW laser with 5.6 W maximum power (Kvant Laser, \(\lambda\) = 520 nm) using a light sheet with a waist thickness of about 1 mm.

Measurements are performed at different magnifications and fields of view. Here, we will focus on close-up measurements of the shear layer formed downstream of the flow separation on the cylinder surface. This area is characterized by high dynamics within the flow and includes quiescent regions on the downstream side of the cylinder and strong velocity gradients within the shear region. Flows such as these are challenging to capture using planar flow measurement techniques. At a magnification \(m = 41.2\) pixel/mm the imaged area covers 31.0 \(\times\) 17.5 mm\(^2\). The light sheet is aligned with the centerline of the test section and passes from bottom to top in the photograph of Fig. 14a, while the flow moves from left to right.

Figure 14b shows an image of the cumulative intensity-change events and represents a total of 1.3\(\cdot\)10\(^8\) events collected in 6 s. The event rate, measured in events per pixel and time, varies by about a factor of 3 between the quiescent wake and the faster outer flow in spite of the fact that the flow throughout the field of view in uniformly seeded. In the photograph of Fig. 14a, the wake region also does not appear brighter than the outer flow. An explanation for this effect may be the limited latency [typ. \(\approx\)100 \(\upmu\)s, see Table 1 in Gallego et al. (2022)] of the event camera sensor: particles in the outer flow either move too fast or are too faint to trigger the contrast detection circuit on the pixel level that result in intensity-change events. With increased speed of the intensity gradient moving across the pixel (e.g., illuminated particle), the pixel will have a reduced likelihood of responding to the change. The event generation is a probabilistic process that has a stochastic nature depending a number of parameters (Gallego et al. 2022; Hu et al. 2021). Among the faster moving particles in the outer flow, it is only the brighter ones that trigger events.

Noteworthy is the absence of laser flare in the event recordings such as on the cylinder itself. In the particle streak image of Fig. 14, bottom, the outline of the cylinder can be identified by the mirrored particle image event tracks on “inside” the cylinder. Pixels on the cylinder surface produce no events which is either due to time-constant intensity levels or due to saturation of the photodiodes. Image-blooming, which is common for overexposed CCD sensors, cannot be observed. The fact that no stationary “particles” are present on the surface allows the motion of actual particles to be tracked within a few pixel distance of the surface which in itself has been a challenging task for the PIV technique.

Compared to the water flow, the air flow has a much higher “pixel velocity” exceeding 50,000 pixel/s. This is about four times higher than the measurements performed with the rotating plate of particles (c.f. Fig. 12). In their work on adaptive time-slice block-matching optical flow algorithms (ABMOF) (Liu and Delbrück 2018) quote values of >30,000 pixel/s as “extremely fast.” The presently available processing algorithms based on motion compensation and sum-of-correlation have been found not suitable to reliably recover velocity information simultaneously in the slow wake region and the fast outer flow. In order to capture the outer flow, the single pass motion compensation algorithm requires sampling areas with dimensions exceeding 100 pixel. On the other hand, the wake region can be reliably processed with samples of 40 \(\times\) 40 pixel (1.0 \(\times\) 1.0 mm\(^2\)). Similarly, the temporal offset \(\tau\) of the sum-of-correlation approach needs by sufficiently short in order to capture the event tracks of fast moving particles in both sampling volumes. To properly capture the high dynamic range of velocities, an adequate processing strategy first needs to be developed.

5.1 PIV processing of the air flow event data

Through temporal sampling of the event data, image sequences can be generated that can then be processed with conventional, cross-correlation-based PIV algorithms. The present data of the near-cylinder shear layer is sampled for a duration of T = 250 \(\upmu\)s at intervals of \(\Delta T\) = 250 \(\upmu\)s. With this sampling, the outer flow exhibits a particle image displacement magnitude of about 20 pixels for a velocity of 1.7 m/s. The resulting image sequence is processed with the previously mentioned multi-frame, grid-refining algorithm using final sampling windows of 64 \(\times\) 64 pixel to accommodate the rather low event density and correspondingly low particle image density of the outer flow.

Selected velocity maps from a sequence of 1000 “reconstructed” PIV recordings are provided in Fig. 15 alongside with corresponding particle event tracks. The high flow velocity and reduced event density of the outer flow result in considerable data dropout, whereas very high validation rates are achieved within the cylinder wake. The recovered data set has a temporal resolution of 4 kHz albeit low-pass filtered by the multi-frame PIV processing scheme. An adaptive processing scheme is required to further bring out the fine-scale structure of the immediate cylinder wake without compromising the velocity estimates from the outer flow obtained at coarser resolutions.

Fig. 14
figure 14

Top: Photograph of imaging setup for flow around a cylinder of 14 mm diameter within a square duct of 76 mm side length. Droplet-based aerosol seeding of 1 \(\upmu\)m is present; the shadow of the cylinder within the light sheet is visible. Middle: Event recording rate within the imaged domain. The cylinder surface is outlined with the red-dashed circle. Bottom: Recorded particle event tracks for a duration of 50 ms highlighting the surface of the cylinder. The outer flow has a velocity of 1.7 m/s; magnification is 0.0237 mm/pixel

Full size image
Fig. 15
figure 15

Left column: Event-images of the near-cylinder shear layer for different times. Right column: velocity maps obtained with PIV processing of the event data sampled for time intervals of T = 250 \(\upmu\)s. Bottom row: shows both the event data and mean flow field for a duration of 0.25 s

Full size image

6 Discussion

From the previous experiments, image velocities on the order of ±10,000 pixel/s were found feasible with the present event camera hardware. Given a field of view 100 mm, that is imaged by a sensor of 1000 pixels side length, flow velocities in range of (\(\pm 10\) pixel/ms)  \(\times\)  (100 mm / 1000 pixel) = ±1 m/s are possible. By increasing the field of view to 1 m, the measurements range is linearly increased to ±10 m/s. As these limitations seem to be rooted in the sensors’ latency, it will require advancements on the detector end to extent the velocity range to higher values.

The measurement of quiescent regions in the flow was found to be a limitation, in particular, in making a reliable distinction between the actual absence of particles and the absence of events due to resting particles. Here, a combination between a framing camera and an event camera, such as the DAVIS camera introduced by Tedaldi et al. (2016), could be used to localize stationary particles at regular time intervals.

While it is true that an event camera actually produces a reduced number of data while viewing a scene, in particular one with limited dynamic content, the imaging of many small and constantly moving particles quickly leads to a considerable amount of incoming events. The investigated water and air flows produced on the order of 20 \(\times\) 10\(^6\) events/s resulting in data rates of up to 60 MB/s. With increased seeding density, the event data rate increases proportionally, resulting in data dropout in the limited bandwidth data acquisition channel. During the previously described experiments, the seeding density and directly coupled event data rate had to be carefully adjusted in order not to cause data dropout, which was accompanied with loss of time-stamp information or even a complete loss of signal (blank areas). With the current hardware, reliable measurements could be obtained with data rates of up to 50 \(\times\) 10\(^6\) events/s at an associated compressed data stream of about 130 MB/s. This data rate corresponds to the 8-bit data stream of a high-speed camera with the same sensor resolution (1280 \(\times\) 720) recording at 150 frames/s. With a specified contrast change detection latency of 100 \(\upmu\)s, the event cameras are capable of providing equivalent frame rates of 10,000 frames/s as demonstrated, for instance by Howell et al. (2020).

Some of the particle event tracks of the recorded air flow in Fig. 14 have a width of only 1 pixel which is known to result in so-called pixel-locking artifacts in conventional PIV processing. Possible influences of this and other effects on the measurement uncertainty of EBIV will have to be investigated further. This includes possible nonlinearities arising through the generation of repeated events at the same pixel location as a bright particle image passes the pixel.

The following summarizes a number of advantages and disadvantages of flow velocimetry using event-based cameras based on practical experiences gained in the process of developing the EBIV technique.


Advantages

  • Event-based sensing is suitable for real-time flow visualization. The event tracks produced by the movement of the particle field directly visualize the flow field without the need of further processing.

  • Compared to conventional imaging, event-based imaging is associated with a considerably reduced data rate.

  • Event-based cameras are tolerant against spatial intensity variations within the light sheet because its sensors only react on intensity changes.

  • Event-based cameras are tolerant against laser flare (light scatter on surfaces) and non-uniform background intensity.

  • Event-based cameras are tolerant against high background intensity. This allows particles to be tracked in bright environments, assuming this intensity is essentially constant in time (or only varies slowly in time).

  • Particle event tracks are clearly distinguishable from the background and can be extracted either by feature detection or by using the clusters formed in the motion compensation analysis.

  • Both in-plane and out-of-plane loss of particle images is not as critical as for PIV. For particles leaving the sample volume the number of events contributing to the velocity estimate is reduced which results in an increased uncertainty.

  • The time-resolved nature of the event data can be used to extract additional information of the particle motion such as acceleration or path curvature. Contrary to pulse-illuminated PIV (or PTV) image frames, the event-trails generated by moving particles provide continuous information on their location with microsecond resolution.

  • Conventional PIV processing is possible by generating sub-frames from the time-resolved event data.

  • Hardware requirements are less demanding in comparison to conventional PIV systems relying pulsed light illumination (see below).

Disadvantages

  • Slow moving particles will produce less events per given time. In areas of slow or nearly quiescent flow, an increase in the sampling time will result in more events from which to reconstruct the velocity. On the other hand, faster moving particles have a reduced likelihood of producing events which is due to the inherent latency of the EBV sensor (typ. O(100 \(\upmu\)s)).

  • Temporal intensity variations within the image scene, such as 50 Hz lamp flicker, will trigger many simultaneous events that are not related to the motion of objects. This can be addressed through the design of sensor using, e.g., the Global Hold, Global Reset approach (see Ryu 2019). The Gen.4 sensor of the evaluation kit by Prophesee has an anti-flicker filter option available. This spontaneous generation of events could be harnessed to briefly make stationary or slowly moving particles visible thereby overcoming the previously mentioned shortcoming.

  • The velocity range is restricted which limits the applicability of EBIV to slow flows. While for PIV the velocity range can be adapted by simply changing the separation of the laser pulses (or frame rate of the high-speed camera), no such option currently exists for EBIV. The upper bound on reliable velocity detection is on the order of 50,000 pixel/s and seems due to sensor’s latency with present day event camera hardware.

  • The bandwidth of EBV hardware can be a limiting factor as the number of events scales linearly with seeding density (i.e., illuminated particles).

7 Conclusion and outlook

The material presented in the preceding sections constitutes a feasibility study on the use of event-based vision (EBV) for the estimation of fluid flow velocity maps. The work is focused on obtaining dense 2d–2c data using particle image densities previously not reported with event-based imaging. While there is no intention in rivaling the well-established PIV techniques, which follow frame-based approaches, event-based imaging velocimetry (EBIV) may nonetheless provide some attractive advantages. The flow measurements clearly demonstrate that EBIV enables measurements in close proximity of surfaces where PIV and other frame-based imaging methods generally have problems with excessive light scatter. As surfaces generally do not move, they do not trigger any intensity-change events and hence are not visible by the event camera (this also makes calibration using stationary targets a little more challenging). Another advantage is that individual particles can be identified efficiently once the underlying flow field has been determined. Although not implemented in the present work, the continuous event tracks produced by the imaged particles allow the reconstruction of the particle paths in time and 2d space. As demonstrated in previous works Borer et al. (2017) and Wang et al. (2020), this can be readily extended to time-resolved 3d particle tracking.

From an implementation point of view, the investment cost is reasonable, given the fact, that instead of a pulsed laser, a low-cost CW laser or other constant-intensity light source (e.g., LED) can be used. The sensitivity of event cameras is considerably higher such that moderate laser powers in the 1-5 Watt range are already sufficient to adequately image micrometer-sized particles in water and air flows with viewing domains of 10 cm side length. Beyond this, EBIV does not require elaborate synchronization between camera and laser. The currently available event camera hardware can be synchronized between different units for multi-view imaging and can additionally record external synchronization events in the data stream. Finally, unlike specialized cameras used for PIV (e.g., actively cooled sensor, double-shutter capability), event cameras are available at a significantly reduced unit cost as their use is intended for a much wider range of applications such as autonomous navigation, 3d sensing, object counting, to name a few.

Processing of the captured event data can be optimized considerably through GPU implementations such that real-time processing becomes feasible. Latency between data acquisition (event data input) and result can be on the order of milliseconds, as already demonstrated in a multitude of applications using EBV [e.g., real-time autonomous navigation (Rebecq et al. 2018; Kim and Kim 2021; Liu et al. 2021) or collision avoidance (Falanga et al. 2020)].

EBV sensor development is ongoing with some event cameras already employing sensors with megapixel (HD) resolution along with small pixels in the 5\(\,\upmu\)m range. The use of back-side illuminated pixels, such as in the device used in the current study, further increases sensitivity and dynamic range. Currently, the systems can image at an equivalent frame rate of up to 5–10 kHz. The latency of future EBV sensors could be reduced through the implementation of alternative read-out architectures which would increase the temporal precision of fast occurring events. The previously shown velocity data of the cylinder wake flow already has an effective temporal resolution of 250 \(\upmu\)s with a corresponding frame rate of 4 kHz.

Beyond the application described herein, event-based vision will certainly find other applications in the field of flow diagnostics and related areas such as time-resolved background-oriented schlieren (BOS) and flow visualization.