MEMS-based lidar for autonomous driving

Abstract

Lidar, the acronym of light detection and ranging, has received much attention for the automotive industry as a key component for high level automated driving systems due to their high resolution and highly accurate 3D imaging of the surroundings under various weather conditions. However, the price and resolution of lidar sensors still do not meet the target values for the automotive market to be accepted as a basic sensor for ensuring safe autonomous driving. Recent work has focused on MEMS scanning mirrors as a potential solution for affordable long range lidar systems. This paper discusses current developments and research on MEMS-based lidars. The LiDcAR project is introduced for bringing precise and reliable MEMS-based lidars to enable safe and reliable autonomous driving. As a part of development in this project, a test bench for the characterization and performance evaluation of MEMS mirror is introduced. A recently developed MEMS-based lidar will be evaluated by various levels of tests including field tests based on realistic scenarios, aiming for safe and reliable autonomous driving in future automotive industry.

Zusammenfassung

Lidar, ein Akronym für Light Detection And Ranging, erhielt als Schlüsselkomponente für autonome Fahrsysteme in der Automobilindustrie viel Aufmerksamkeit, da es hochauflösende und hochgenaue 3D-Bilder der Umgebung bei verschiedensten Wetterbedingungen liefert. Derzeit entsprechen die Preise und die Auflösung der kommerziell verfügbaren Lidar-Sensoren jedoch noch nicht den Zielanforderungen, um als Basissensor zur Gewährleistung der Sicherheit während des autonomen Fahrens akzeptiert zu werden. MEMS-basierte Scan-Spiegel waren als potentielle Lösung für ein leistbares weitreichendes Lidar-System Fokus einer kürzlich durchgeführten Arbeit. Diese Arbeit erörtert die derzeitige Entwicklung und Forschung von MEMS-basierten Lidar-Systemen und stellt das LiDcAR-Projekt vor, dessen Ziel es ist, ein präzises und verlässliches MEMS-basiertes Lidar-System für autonomes Fahren zu entwickeln. Als Teil-Entwicklung dieses Projekts wird ein Prüfstand zur Charakterisierung und Evaluierung von MEMS Scan-Spiegeln vorgestellt. Das kürzlich entwickelte MEMS-basierte Lidar-System wird auf verschiedenen Testebenen evaluiert, inklusive Feldtests, basierend auf realistischen Szenarien, mit dem Ziel, in Zukunft sicheres und verlässliches autonomes Fahren zu gewährleisten.

Introduction

Lidar is the acronym of light detection and ranging, in an analogy to radar. The basic structure of any pulsed time of flight (TOF) lidar can be described as follows (see Fig. 1): A laser module generates a short laser pulse and the light hits the target object. Then the receiver detects the reflected pulse from the target. From the elapsed time from the transmitted pulse to the received pulse, a data acquisition unit calculates the distance.

Fig. 1.

Structure of lidar system: flash lidar (left) and scanning lidar (right)

Since its first use for highly accurate lunar ranging, many lidar systems have been developed for various applications as a high resolution and highly accurate measurement technique, e.g. for earth surface measurements [1], wind speed measurements [2], building construction operations [3], mining industry [4], forestry [5], and robotics [6].

Recently lidar sensors have received much attention for the automotive industry as a key component for high level automated driving systems [7–9]. Compared to other 3D sensing techniques, such as stereo cameras and radar, lidar sensors can provide high resolution and highly accurate 3D measurements of the surroundings under various weather conditions [10]. The expected function from automotive lidar sensors is to ensure reliable and safe automated driving such as collision detection, blind spot monitoring, object and pedestrian recognition [11, 12], terrain mapping [13]. Currently Google [14], BMW [15], Volvo [16] and other autonomous car developers include lidar sensors in their development to ensure safe driving [17].

The commercially available lidar sensors can be categorized into two types according to the maximum range and the resolution of the system: short range and long range lidar sensors. The short range lidar sensors usually measure less than 50 m distance with a narrow angle for forward collision warning and blind spot detection [18]. They usually implement a flash lidar structure, shown in Fig. 1. In a flash lidar, the transmitter illuminates the whole scene and an array of detectors measures the distance of each pixel of the image. Continental (Hanover, Germany) and LeddarTech (Quebec City, Canada) provide affordable flash lidar sensors to the market [19, 20]. Short range lidar sensors are low-priced due to the simplicity of the setup, but the measurement distance is fairly short since the light intensity from the transmitter is dispersed with a relatively large angle, and is also limited by eye-safety considerations [21]. In contrast, scanning lidar sensors, also shown in Fig. 1, focus their light on a very small area, making them suitable for long distance and high resolution imaging. A scanning lidar sensor diverts a collimated beam to the target and illuminates a small fraction of the target object at one shot. Then the reflected light from the small area of the target object is focused on the detector, recording a pixel. By moving the beam position, the distance of the next pixel is measured and a 3D image of the entire scene can be generated. Currently Velodyne (Morgan Hill, CA, US) and Quanergy (Sunnyvale, CA, US) provide entire view (360^∘) lidars in the market and Ibeo (Hamburg, Germany) provides lidars with a full front range (145^∘ ) [22–24]. Scanning lidars can provide rich 3D maps of the surroundings, which allows further analysis such as identification and avoidance of objects and pedestrians.

The currently popular scanning solution in the automotive lidar market is the rotating scanner [7, 25]. The rotating scanner is currently most popular for many commercial lidar sensors because it provides straight and parallel scan lines with a uniform scanning speed over a wide field of view. Furthermore, it provides less stress and low vibrations on the mirror surface [26]. As a drawback, it is generally bulky and less scalable due to the limiting duty cycle by the edges between facets for polygon mirrors [27]. In addition, a large inertia of the rotating module limits the frame rate of lidar sensors and requires rather high power consumption.

As an alternative solution, optical phase arrays and microelectromechanical system (MEMS) mirrors receive much attention as promising solutions for affordable lidars [18, 28] and compete with each other for the future automotive lidar market. An optical phase array (OPA) is a solid-state device that steers the beam by micro-structured waveguides. Since there is no mechanical moving part, OPA allows extremely high scanning speed over 100 kHz over large angles. However the insertion loss of the laser power is a drawback [29, 30]. As a developing technology with high potential, the interests on OPA for automotive lidar is growing in academia and industry even though OPAs are still not proven for long range lidar yet. Microelectromechanical systems (MEMS) mirrors carry a high potential as a future automotive lidar solution since both the form factor and the cost in a large volume can be drastically reduced. The lidar manufactures are keen to test these two promising technologies. Quanergy announced a solid state lidar system (S3) based on optical phase array (OPA) scanner at the Consumer Electronics Show (CES) 2016, which has 120^∘ optical scanning angle for horizontal and vertical direction and with 0.05^∘ resolution [31]. Innoviz (Kefar Sava, Israel) also released a MEMS-based solid state lidar (InnovizPro), which allows ranging up to 150 m distance with a $73^{\circ} \times 20^{\circ}$ field of view (FOV) with 0.15^∘ horizontal resolution [32].

This work discusses the current research on MEMS-based lidar sensors and introduces a MEMS-based lidar sensor. In Sect. 2, the current state of the art about MEMS-based lidar sensors is reviewed. A MEMS-based lidar approach in LiDcAR project is introduced and its challenges are elaborated in Sect. 3. A MEMS test bench to evaluate MEMS mirror for lidar application is presented in Sect. 4. Verification and validation of automotive functions using lidar sensors are explained in Sect. 5. Then Sect. 6 concludes the paper.

Review on current development of MEMS-based lidar

MEMS-based scanning mirrors have recently received much interest for their use in automotive lidar applications as a light-weight, compact and low power consumption scanning solution. Several MEMS-based lidar sensors have already been developed in various application such as space applications [33, 34], robotics [35, 36] and wind velocimetry [37], showing the feasibility of the technology. For automotive applications, Hoffman et al. developed a 360^∘ scanning 2D MEMS scanner lidar with 60^∘ of azimuth angle [38] in the EU project of Minifaros [39]. Stann et al. also developed a short range lidar sensor for the small unmanned ground vehicle (UGV) and air vehicle (UAV), whose range is up to 160 m with a $42^{\circ}\times 21^{\circ}$ FOV and 6 fps [40]. Ito et al. developed MEMS-based lidar with single photon CMOS focal plane detectors, which can measure distances up to 25 m with a $45 ^{\circ }\times 11^{\circ}$ FOV [41]. MEMS lidar sensors are also used for testing of the lidar technique itself, e.g. digital modulation [42] and optical code division multiple access (OCDMA) [43].

MEMS scanning mirrors can be mainly categorized into resonant MEMS mirrors and non-resonant MEMS mirror according to the operating frequency with respect to their mechanical mode [44]. The resonant MEMS mirrors provide a large scan angle at a high frequency and a relatively simple control design while the scan trajectory is sine-like, i.e. non-uniform scan speed. In addition, MEMS scanner as a nonlinear oscillator can cause softening and or stiffening, limiting the operation frequencies [45]. Non-resonant MEMS mirrors, also called quasi-static MEMS mirrors, provide a large degree of freedom in the trajectory design. Although rather complex controller is required to keep the scan quality, desirable scanning trajectories such as triangular or saw tooth scanning with constant scan speed at large scan ranges can be generated by an appropriate controller design [46, 47]. Quasi-static MEMS mirrors, however, usually have a smaller scanning angle compared to the resonant MEMS mirrors. To enlarge the scan angle, an additional optics and lens are used [40, 46, 48, 49]. An immersion of the MEMS mirror package also increases the scan angle by Snell’s window effect with additional damping as a tradeoff [50].

Actuation and scanning principles get diversified, looking for the best technique in the future lidar market. The most common actuation principle is currently the electrostatic comb drive while electromagnetic actuation [51] and piezoelectric actuation [52] have also been reported for lidar applications. Not only flat mirror based scanning, a 1D refractive MEMS scanner has been developed for a ±5^∘ optical scan angle based on the rotation of a convex lens with comb drive. A commercially available digital micro mirror device (DMD) has been applied for a scanning lidar sensor as a programmable grating [53]. Wang et al. developed a new concept of optical phase array based on MEMS approach using vertically actuating comb drives, which can steer the laser beam ±11^∘ [54].

The figure of merit (FoM) for the MEMS mirror for automotive lidar is different from a typical MEMS mirror for other applications. A FoM for MEMS mirrors is proposed for pico-projector application, which is the product of the scan angle, mirror size, and resonant frequency (i.e. scanning frequency), which defines the resolution of the projector [44]. For long range lidar, scan angle and mirror size are accepted as important parameters. Especially for the bistatic lidar structure, a larger aperture at receiver is important to ensure enough signal to noise ratio (SNR) for long distance detection. Sander et al. propose a dedicated MEMS mirror structure, which consists of a resonant MEMS mirror for the transmitter and 14 identical MEMS mirror for the receiver, operating in a synchronized manner [55]. Besides the mirror size, robustness to the harsh environment conditions such as extremely high or low temperature and large vibrations should be considered [56]. A trade-off between the mirror size and the vibration rejection performance is discussed while an immersion medium in the MEMS mirror package is reported to provide enhanced robustness against external vibrations and shocks [57].

LiDcAR project: evaluation of MEMS-based lidar

The LiDcAR project, which is a collaborative (Infineon Technologies Austria AG, TU Wien – Automation and Control Institute (ACIN), and the Virtual Vehicle Research Center) FFG funded research project, aims to explore and to assess the two most promising lidar technologies in order to pave the way for the long sought automotive-qualified, long-range, robust, and low-cost Lidar solution. This will be accomplished in a three-step process:

• 1. LiDcAR’s first step is to perform a comprehensive laboratory-based evaluation and comparison of the most promising technologies (such as MEMS mirror, rotating polygon mirror, optical phased array, flash Lidar, and fixed multi-beam). Based on the identified strengths and weaknesses, the two most promising technologies are selected and two experimental prototypes are realized.

• 2. The second step of LiDcAR is to research and develop methodologies that will enable essential lidar-metrology and verification methodologies. This effort will pave the way towards a proper lidar evaluation system and will thus enable the path towards an automotive-qualified lidar system.

• 3. In a third step, in tight collaboration with ALP.Lab, the two experimental lidar prototypes will be integrated into an autonomous test vehicles. Extensive in-field tests at Styria’s autonomous driving test-region will provide valuable data that will be analyzed and assessed according to the developed lidar-metrology and verification methodologies. The gained results and insights will finally pave the way towards the future automotive-qualified and low-cost key lidar technology made in Austria.

The LiDcAR project and its consortium identified the micro-scanning 1D MEMS-mirror lidar as one of the currently most promising lidar technologies. Therefore, the consortium is currently intensively working on a very first MEMS-based lidar prototype. Its fundamental scanning concept is depicted in Fig. 2. A 1D MEMS mirror oscillates and deflects a laser pulse into the scenery. It is controlled and actuated by a dedicated driving circuitry. Laser pulses reflected off objects in the scenery are focused by optics and detected by an array of photo diodes. The received analog signals are amplified via transimpedance amplifiers and high-speed analog-to-digital and time-to-digital conversions are carried out. Finally, a 3D point cloud can be computed and forwarded to a central sensor fusion ECU. In this particular approach to lidar sensors the MEMS mirror oscillates about only one axis, horizontally scanning the scenery left-right. At the same time, the laser beam fans out and paints a vertical line in the distance, resulting in a full scan of the scenery. This unique approach results in a number of key features: The horizontal resolution of the system is entirely determined by the timing accuracy of the MEMS driving circuit and can reach 0.1^∘ or better. Since all angles in the vertical direction are scanned in parallel with the same laser pulse, the scan speed is very high. With the laser, light focused into one thin line the pulse energy is concentrated in a small area enabling long range distance measurements. And finally, a MEMS mirror with only one axis is far less complex to manufacture and to operate than a two axis mirror and it is also far more robust to vibration and shock.

Fig. 2.

Working principle of the micro-scanning 1D MEMS-mirror lidar approach

A major LiDcAR result achieved so far is the realization of a first feasibility study. This feasibility study includes the MEMS mirror and a very first version of a future MEMS driver circuitry, as depicted in Fig. 3. Together both components form the central and most crucial part in the micro-scanning 1D MEMS-mirror lidar concept. Preliminary results already proved successfully the feasibility of this lidar concept. Furthermore, this platform enables rapid technology explorations and rapid prototyping, which is crucial in the light of system complexity and the fast-paced lidar research arena.

Fig. 3.

MEMS mirror and driver feasibility study

Test bench development for MEMS scanning mirror

The characterization of MEMS scanning mirrors is crucial in the development of the MEMS mirror, in order to measure and evaluate the performance and reliability of the developed mirror prototypes in order to ensure to meet all requirements needed to be use in the automotive environment as essential part of automotive lidar systems. In our 1D scanning lidar approach the MEMS mirror is used to steer the laser beam in a deterministic and reproducible manner. The characterization of such MEMS mirror needs the real-time measurement of the mirror deflection together with actuation voltage signal and the implemented current sensing approach, but with guaranteed accuracy and resolution in a calibrated measurement setup. The characterization environment thus should provide the scanning trajectories in degree within a required measurement uncertainty, resolution and possibly appearing distortion. Repeatability and reproducibility of the developed setups should considered with included geometric calibration procedures, to correct for all relevant systematic error contributors to the scan trajectory measurement. The signal quality from all sensor readouts should be high to meet the required challenging resolution and measurement uncertainty requirements, regardless of noise and external disturbances such as electromagnetic interference (EMI) from high voltage operation of the MEMS scanning mirror. For such a purpose, a test bench for MEMS scanning mirrors has been developed and implemented.

Figure 4 describes schematics of the mirror test bench. The test bench mainly consists of a laser unit, a position sensitive device unit, and adjustable mounts for the direct measurement of the mirror deflection, including calibration and alignment. The laser unit consists of an adjustable fiber collimator from an external fiber laser sources. The laser is focused via the central region of the MEMS mirror onto a position sensitive device (PSD) (S5991-01, 2D PSD, Hamamatsu). This PSD element allows for real-time high bandwidth tracking of the reflected laser beam position on the active PSD area. For the accuracy of geometrical alignment, the laser collimator is attached on a 5 DOF mount, correcting position and direction of the laser beam. The MEMS scanning mirror is placed on a 6 DOF mount to correct all possible device orientation and tilt errors of the MEMS mirror in the given package prototype. Proper positioning of the MEMS mirror after replacement and inclusion in the setup. The position error during the mirror exchange procedure is measured by a high resolution digital camera with attached telecentric objective. An image processing software provides the position and direction error of the mirror, allowing an intuitive support for proper manual adjustment of the MEMS mirror mount. The PSD unit is attached on a fixture mounted on a 1D motorized linear translation stage, which enables the measurement and of the geometrical correction factors as needed by the geometrical calibration process. The enable maximum range for mirror deflection sensing the PSD is tilted by 45^∘ on the mount, having the scan trajectory aligned with the diagonal of the quadratic-shaped active PSD area. On the electrical side the PSD unit is connected to a dedicated preamplifier module with transimpedance amplifiers (TIA) and a following analog processing module. All analog processing circuits have optimized for robustness against the EMI generated by the high voltage operation of the closely located MEMS mirror.

Fig. 4.

Schematics and a picture of the test bench. The laser is equipped with a 5 DOF optics mount, the mirror is attached with a 6 DOF mirror mount, and the PSD is mounted on a 1 DOF motorized stage for the alignment

Systematic errors from the alignment of the individual test bench components are the major contributors to the achievable accuracy of the mirror deflection measurement. One of the major errors source is the distance between the PSD and the MEMS mirror, which needs to be determined with an uncertainty of less than tens of micrometers to ensure the required accuracy in the measurement. A dedicated calibration process has been developed to estimate the needed geometrical correction factors for enabling a highly accurate mirror deflection measurement.

Figure 5 provides an example of a measurement result from the developed test bench, showing a typical response curve as one of the main characteristics of a MEMS mirror prototype operated in open loop. The diagram shows the evolution of MEMS mirror deflection angle amplitude over frequency when starting the mirror and operating up to the high angle operation point – in this case at 15^∘ mechanical angle amplitude.

Fig. 5.

Frequency response of the MEMS mirror, measured by the developed test bench. The response curve of the MEMS mirror consists of top response curve (red solid line) and bottom response curve (blue dashed line). When sweeping down the excitation frequency on the mirror along the bottom response curve, a jump (labeled 1, with an up arrow) to the top response curve at higher angle amplitudes is observed. The scan angle increases further by an following up sweep after the jump, following the top response curve. At a certain driving frequency, the angle amplitude falls back (labeled 2, with a down arrow) to the bottom response curve (Color figure online)

By providing accurate mirror angle data the developed setup – including the geometrical calibration procedure – allows for reliable high accuracy characterization of MEMS scanning mirrors with high repeatability and reproducibility.

Verification and validation of MEMS-based lidar for autonomous driving

Before a new vehicle function can go into serial production a thorough verification of its functional correctness is necessary not only to ensure customer satisfaction, but also to avoid safety threats to the vehicle inhabitants or other traffic participants in case of safety critical functions. This is achieved by performing tests at different test levels, where the developed function parts are integrated and tested using real or simulated stimulations. With increasing test level the test environments become closer to reality, where in the lowest level the function is tested using purely simulated inputs and in the highest level only real sensor data is used obtained by during test drives on public roads.

An example of the main test levels considered in the automotive industry is shown in Fig. 6, where also a commonly used test environment is depicted for each test level. The shown test levels reflect the integration steps of the developed function into the vehicle, where in level one and two its hardware and software parts are tested and in level three to five the function is tested in combination with other vehicle functions.

Fig. 6.

Test levels ranging from 1 to 5 and their typical test environments used for testing automotive functions

Since testing at test levels three to five is very costly and time consuming, the LiDcAR consortium cooperates tightly with ALP.Lab (Austrian Light Vehicle Proving Region for Automated Driving) in order to get access to wide range of test hardware, proving grounds and permissions for public road tests. In particular, the access to large number of test tracks from AVL, Magna Steyr, Zentrum am Berg, Testregion Lungau, or the Redbull race track in Spielberg allows the simulation of a wide range of environment conditions and driving situations. On these test tracks automated robots will be used to move objects like pedestrians, cars and cyclists into the field of view of the lidar sensors. This approach allows the execution of tests under a wide range of weather conditions with known objects and their position, which ensures the reproducibility of the test results and ensures that the sensor measurement data can be automatically verified for correctness. The simulation environments provided by ALP.Lab can also be used to specify the tests performed on the proving ground, increasing efficiency during test development.

In addition, ALP.Lab provides highly accurate sensor systems, which can be mounted on the test vehicle, allowing a performance assessment of the used lidar sensors. Moreover, ALP.Lab provides a cloud-based data management solution for seamless data acquisition, storage, and automated analysis allowing global access to the obtained measurement data and test results. By using this cloud based storage and analysis framework it is possible to process a large number of test drives within a short time by leveraging the massive parallel computation power provided by such systems. The cloud based storage approach, where each ALP.Lab customer has its own tenant ensuring protection against unauthorized access, is capable of handling many terabyte of data. This large amount of storage is necessary because the different sensors, especially lidar systems, produce measurement data at a very high rate, making the use of latest hard disks infeasible.

Conclusion

This paper discusses the current developments and research on MEMS-based lidar systems. To ensure safe autonomous driving solutions in future mobility, besides 2D cameras and radar sensors an automotive qualified lidar system is seen essential to enable proper perception of the car environment during all driving situations. Stringent requirements on such lidar systems are long range, high resolution and accuracy in distance but also high angular resolution in lateral and vertical direction w.r.t. sensing direction, secured long lifetime in the automotive environment conditions (temperature range, temperature changes, vibration and shock robustness), and, altogether at an acceptable low price. MEMS-based lidar is regarded as one of the most promising solutions for affordable automotive systems, and many rising but also well-established companies as well as research institutions are currently working with high intensity to bring it into the future automotive lidar market.

To cope with the challenging tasks during lidar system development, the LiDcAR project aims in investigation and validation of automotive lidar solutions. The aims and underlying main work packages of the LiDcAR project have been described throughout the paper, targeting in development on characterization environments for lidar components but also lidar systems, and the validation of lidar system prototypes in the automotive environment with the ALP.Lab automated driving test roads. Currently the concept of the feasibility study is successfully finished, resulting in a working MEMS mirror based scan system where a 1D MEMS mirror is interfaced by an FPGA-based evaluation platform. This evaluation platform is based on the current version of a MEMS driver ASIC sensing and control concept – implemented on the FPGA board as a part of the evaluation system. First results have proven suitability for the chosen mirror and driver concept for use in automotive scanning lidar applications. As another part of the project work, a test bench for detailed characterization of scanning MEMS mirrors has been developed. The test bench includes a real-time 2D PSD based optical readout of the mechanical deflection of the MEMS mirror for all operation modes, enabling reproducible high accuracy electromechanical characterization of MEMS mirror devices.

As next steps, the developed test benches will be further optimized and extended towards the robustness evaluation of scanning lidar components and systems. Lidar system prototypes will be realized and these MEMS-based lidar solutions will be evaluated for use in automotive applications, including field tests based on realistic driving scenarios.