An overview of developments and challenges for unmanned surface vehicle autonomous berthing

Abstract

With the continuous progress of contemporary science and technology and the increasing requirements for marine vehicles in various fields, the intelligence and automation of ships have become a general trend. The autonomous control of surface Unmanned Surface Vessel (USV) generally covers the USV path planning, path tracking control, and autonomous collision avoidance control. But in the whole navigation process of USV, autonomous berthing is also a crucial part. And the research on the algorithm of the automatic berthing process of the USV is less. Mature USV autonomous berthing technology can effectively reduce the cost of human and material resources and financial resources while reducing the accident rate reasonably and safely. Therefore, it is of great importance to comprehensively promote the development of USV autonomous berthing technology.

Introduction

The USV, with their low cost, small size, fast action, intelligence, and other advantages, play a very important role in daily life, emergency response, and scientific research. USV generally consist of a platform system and a mission payload system [1]. It can be used in civil fields such as environmental monitoring, search and rescue and salvage, and hydrological exploration, as well as intelligence collection, regional inspection, Mine clearance, anti-submarine, and other military fields [2]. With the development of intelligence, higher requirements are put forward for the "autonomy" of USV, and the autonomous berthing technology of USV is an important link to realize the real "unmanned" of USV.

The International Maritime Organization (IMO) and intelligent shipping companies have conducted some research on the autonomous berthing function of USVs [3]. In 2018 IMO divided the level of ship autonomy into 4 classes as shown in Table 1; The fourth class is a fully autonomous ship, which can realize autonomous navigation from shore to shore under completely unmanned conditions, including autonomous berthing control. In the same year, the Norwegian operator Norled's ferry "Folgefonn" has become the first ferry in the world to be equipped with an automatic berthing system, which was tested during several stops during a round trip to the port. During the test, the captain did not perform any manual control operations. Nippon Yusen Kabushiki Kaisha (NYK), NYK Group companies MTI Co. Ltd. And Japan Marine Science Inc (JMS) have jointly developed a system to assist ships in berthing in 2019; In May of the same year, China's first unmanned autonomous navigation system experimental vessel—"Zhiteng" was launched in Qingdao for sea trial; the vessel tested remote control and automatic berthing and other functions [4]; In 2020, researchers at the Computer Science and Artificial Intelligence Laboratory(CSAIL) at the Massachusetts Institute of Technology (MIT) said they had created a self-driving boat that could move autonomously, carrying passengers across a river and guiding berthing by detecting QR codes on the shore; in the same year, the South Korean KASS project conducted testing and evaluation of autonomous ships. In 2021, Ouka Smart Hublot has also carried out research to upgrade its equipment, making it possible for USV to berth automatically by means of automatic magnetic suction. In January 2022, the U. S. Navy tested an autonomous berthing and recovery test of the Explorer USV in the Persian Gulf. In the same month, MEGURI2040, an USV project supported by the Nippon Foundation, completed its first demonstration test, successfully conducting the world's first autonomous navigation demonstration of a small sightseeing boat in Saru Island, Yokosuka. In June 2022, China's first fully domestic 100-ton USV successfully completed its first autonomous voyage at sea in the waters off Zhejiang.

Table 1 IMO ship autonomy class and berthing method

On the theoretical side, in 2011, Van Phuoc Bui et al. used autonomous tugs for berthing navigation [5]; In 2015, Mizuno et al. designed experiments based on a variety of beating algorithms combined with Model Predictive Control (MPC) to complete automatic berthing [6]; In June 2016, Joohyun Woo et al. designed a vector field-based berthing planning method [7]; In 2017, Park, Jong-Yong, et al. proposed a berthing system with berthing rope based on Proportion Integral Differential (PID) control design [8]; In the same year, Zhang et al. designed an artificial potential field-based solution to the autonomous berthing problem [9]; In July 2020, Jia et al. conducted experiments on data training under a berthing coordinate system [10]; In October 2020, Huang et al. designed a fuzz Line Of Sigh(LOS)-based berthing path tracking control system [11]; In 2022, Wang et al. proposed an unmanned berthing water shoreline detection-based boat autonomous berthing method [12]. From the above-mentioned current popular research projects (As shown in Fig. 1), the research on autonomous berthing of USV has important scientific and technological value and economic value.

Fig. 1

Development of Autonomous Ships and USV

Aiming at the review of the research of USV autonomous berthing methods, the subsequent chapters of this paper are arranged as follows: the second part introduces the now popular USV berthing architecture; the third part introduces the existing methods of USV berthing key technologies; the fourth part introduces the opportunities and challenges of USV autonomous berthing; the fifth part summarizes the key technologies of USV autonomous berthing and looks forward to the extensive use of USV in the future.

The architecture of USV berthing

At present, most ships need manual assistance to leave the dock, such as the captain driving the yacht to complete the dock berthing (Fig. 2), and the USV needs manual assistance to leave the shore (Fig. 3 [13]); the USV needs manual assistance to put away the boat on large ships [14] (Fig. 4). With the development of artificial intelligence, if there is a control system that can copy or transplant the rich driving experience of the "old captain", the maneuvering control of USV will become easy; if all USV can leave the dock independently, then a lot of manpower and financial resources for operating USV will be saved; if all USV If all the USV can leave the dock or mother ship independently, then the "unmanned" management of USV will be truly realized.

Fig. 2

Schematic diagram of yacht berthing

Fig. 3

USV rely on human-assisted berthing [13]

Fig. 4

Large ship-assisted stowage [14]

Most existing USV are intelligent platforms that rely on multiple shipboard sensors and navigate on the water surface in an autonomous or semi-autonomous manner [15]. They are only equipped with paddle, rudder, or double propeller devices, which are typical under-driven control systems, especially when berthing and unberthing maneuvers are performed in ports, where the speed is low; the interference of external forces such as wind, waves, and currents are relatively increased, and the crowded and narrow port environment significantly reduce the hull maneuvering performance, and the USV control system shows strong nonlinear characteristics [16]. For these problems that cannot be avoided when the USV is berthing, a set of architecture with deliberate thinking is needed to organize the various functional modules in an orderly and strict manner to realize the autonomous berthing function of the USV.

For the autonomous berthing problem of unmanned craft under complex environmental conditions, to improve the robustness and practicability of the entire berthing process of unmanned craft, the classic OODA control cycle [17] is generally used as the system architecture, which is composed of observation, judgment, decision, and action. The USV berthing system [18] includes USV situational awareness, USV berthing modeling, USV berthing path planning, and USV berthing control. Connecting their functions in tandem, the parts always react to unknown environments, dangerous situations, and various emergencies, perform uncertainty event monitoring, and subsequently autonomously perform hierarchy and decision-making, prompting the computer into manual warning and human intervention.

Then autonomous machine learning will be carried out through manual modeling, manual decision-making, and remote control throughout the systems. The USV will also always be for the input instructions to output and feedback to the information acquisition system for cyclic processing to get the optimal operation process, that is, the completion of the USV's autonomous berthing. Among them, berthing equipment includes shore-based equipment and shipboard equipment. The shore-based equipment includes intelligent USV shore-based monitoring computers and remote communication equipment. On-board equipment includes an autonomous driving control subsystem, navigation subsystem, intelligent planning subsystem, audio processing subsystem, and optical visual processing subsystem. The berthing system starts with human–computer interaction, opens the whole berthing process, and sets up the autopilot program; the planning system carries out path planning and obstacle avoidance for the area, an emergency treatment for the unexpected environment, and updates the environmental information in real-time; the navigation and intelligent control system:carries out real-time berthing control for the ship, adjusts its heading and position. The audio and image processing system converts the detected sound and image information and then gets the required information. After converting and then getting the required important information, the USV carries out real-time navigation and completes the berthing task. In this process, the on-board equipment always dynamically collects audio and images of the environment where the USV is located and fuses the obtained information to continuously make multi-loop action determination in real-time, after which the previously obtained information is transmitted in a unified manner to present the whole USV's movement process on the computer. The general process of autonomous berthing of the USV is shown in Fig. 5. And the semi-physical berthing system is illustrated in Fig. 6.

Fig. 5

The general process of autonomous berthing of the USV

Fig. 6

The semi-physical berthing system

USV situational awareness

Situational awareness is the beginning of the USV berthing process. When a USV berths at a terminal, it first needs to sense the port environment [19, 20] and locate the target berth. There are many USV sensing devices such as Millimeter Wave Radar (MMW-radar), sonar, Global Positioning System (GPS), ultrasound, camera [21], etc., which work in concert [22] to realize the modeling of the berthing environment.

The situational awareness system is an integrated hardware and software system that fuses sensor information with different sensing capabilities and evaluates the berthing posture. The system is generally equipped with navigation radar for collision avoidance and path planning [23], automatic positioning of ships, target detection, and signal processing during navigation [24], etc.; Automatic Identification System (AIS) [25] is used to improve dynamic navigation track planning of ships [26], to assess the risk of collision of ships, to provide early warning and guarantee for ships' navigation [27], and to combine with other equipment for ship speed judgment [28], etc.; GPS is used for positioning of ships [29]; The electronic nautical charts are used for navigation of ships [30]; The forward-looking sonar is used to identify underwater targets [31], which is combined with other systems to improve the stability and accuracy of target tracking [32]; the electronic chart information system is used for route processing and radar processing; the accurate and timely acquisition of specified position tidal information is used to draw up navigation routes [33], etc.; The information from the Beidou Navigation System (BDS) and GPS will be integrated to improve the accuracy and reliability of ship positioning; the picture information from the visible camera and the infrared camera [34] will be integrated to enhance the detection of obstacles in the surrounding environment of USV; LIDAR and MMW-radar are used to compensate for the blind spots of nautical radar, which is the obstacle information at close range. The block diagram of USV berthing situational awareness and evaluation is shown in Fig. 7.

Fig. 7

Block Diagram of USV Berthing Situation Awareness and Assessment

Through these systems, the equipment can complete berthing operations, using microwave, laser, MMW-Radar carried by USV to obtain the distance between the ship and the shoreline, then combined with the ship's GPS, BDS terminal data to determine the ship's position relative to the berth, and get the ship's position and attitude relative to the shore; MMW-radar, attitude sensor, and front sonar are used for real-time feedback of the ship's position information and heading; Due to the limited accuracy of GPS, BDS and other devices, when the ship is close to the berth, the judgment and processing of environmental information are carried out by the ship carrying devices with higher accuracy; real-time dynamic local planning [35] is carried out to come up with feasible paths; finally, relying on the linkage between devices and the implementation of commands by the ship, the computer carries out the fusion of each data to realize the automatic berthing of the ship.

In the field of situational awareness of USV, Sarang Thombre et al. [36] proposed that in order to ensure the perception function of the unmanned ship itself and the accurate cooperation between individual sensors, by combining sensors and AI technologies with each other to achieve adaptability and compatibility with the environment; in the process of perception for the environment and targets, Chen et al. [37] conducted an integrated framework for automated ship identification and behavioral analysis to improve the functionality of video detection infrastructure, thus increasing the safety and accuracy of perception of the surrounding environment; Won-Jae Lee et al. [38] record images to understand other ships around them through cameras installed on the ship and improve the performance of detection, localization, and tracking execution throughout the voyage through deep learning models, which play an important role in the perception of the surrounding environment also plays an important role. Sharma et al. [39] and Wang et al. [40] investigated ship navigation situational awareness, Rowen et al. [41] investigated enhanced navigation environment awareness, and Zhang [42] investigated target detection and obstacle autonomous avoidance of unmanned ships by monitoring the visible light possessed by the tracked targets.

Multisensor information fusion

The sensors in the unmanned sensing system are the top priority, in which there is a clear division of labor among the various layers, and the integration of complex and diverse environmental information is the most essential requirement for information fusion. Information fusion is the complementary combination of information from various sensors at multiple levels and spaces, and processing optimization to ultimately produce a consistent interpretation of the berthing environment [43]. Information and data from multiple sensors or multiple sources are automatically analyzed and synthesized under certain guidelines using computer technology to accomplish the required information processing and estimation. In this process, multiple sources of data are to be fully utilized for rational domination and use, while the ultimate goal of information fusion is to derive more useful information by combining multiple levels and aspects of the information based on the separated observations obtained from each sensor. This not only takes advantage of multiple sensors operating in concert with each other but also synthesizes data from other information sources to improve the intelligence of the overall situational awareness system. In nearly 50 years of scientific research, many effective approaches have emerged.

Weighted Average Methodis one of the most used scenarios in the fusion algorithm and has been improved through continuous improvement in the process to obtain better results [44].

Kalman Filter Method is suitable for the real-time fusion of redundant sensor information in dynamic environments and can effectively eliminate external noise interference when applying Kalman filter method to wireless sensors [45, 46].

Structural Layer Fusion Processing is a multi-sensor data fusion algorithm model, which can fuse different information levels of complex situations at three levels [47].

Neural Network a neural network is a nonlinear, adaptive information processing system composed of a large number of interconnected processing units, proposed based on modern neuroscience research results. Neural networks are a new jinn approach to information processing by simulating the way the brain's neural networks process and remember information [48].

Fuzzy Set Theory Fuzzy logic is a multi-valued logic that specifies a real number from 0 to 1 to represent the degree of affiliation and will perform well for mathematically model agnostic systems [49]; Bayesian theory: Bayesian estimation is a common method for fusing multi-sensor underlying data in static environments. Its information is described as a probability distribution and is suitable for uncertainties with addable Gaussian noise [50].

D–S evidence Theory Dempster–Shafer (D–S) evidence inference is an extension of Bayesian methods, and its basic principle is to describe sensor information using trust intervals [51].

Genetic Template Method The genetic algorithm is a population optimization process that consists of a set of initial values. The optimization process is the process of continuous reproduction, competition and genetic and mutation of this population [47].

Cluster Analysis Method The so-called clustering is the aggregation of a large number of d-dimensional data samples into n classes so that the similarity of samples within the same class is maximized and the similarity of samples within different classes is minimized. The cluster analysis method is a heuristic algorithm that is effective in landmark applications where the number of pattern classes is not precisely known, and it involves grouping (clustering) the data according to some clustering criterion and interpreting each data group as the corresponding target class [52, 53].

Logical Template Method The logical template method is essentially a matching identification method that matches a predetermined pattern (template) of the system with observation data to match and determine whether the conditions are satisfied to perform inference [47, 54].

KohonenFeatureMap Kohonen proposes a self-organizing system based on a network composed of adaptively laterally interconnected neurons that form a dimensionally simplified projection of a multidimensional input space [55]. The aforementioned multi-sensor information fusion methods [56], are shown in Table 2.

Table 2 Information fusion methods for multiple sensors

The information that needs to be sensed during the berthing of the USV mainly includes information about environmental factors, dynamic obstacle information and static obstacle information [57], and all other factors that affect the safety of ship navigation. Environmental factors mainly include meteorological information, hydrological information, and navigational information. Dynamic obstacles mainly include the navigation of large ships, small fishing vessels, and large floating objects and the speed, heading, and real-time position information of these obstacles. Static obstacles mainly include moored ships, islands, piers, and water structures. There are many shortcomings in the single sensing device of ships, therefore, the research of integrated sensing system that fuses multiple sensing devices and realizes multiple sensing devices to compensate for each other's shortcomings is one of the main research directions at present (Fig. 8). The 3D-aware autonomous berthing system mentioned later is also an important derivative research direction of this approach.

Fig. 8

Block diagram of information fusion of multiple sensors

The fused sensor information is characterized by redundant, complementary, timely, and low-cost information, which helps to improve the shortcomings of single sensor observation with insufficient information, as well as the resulting target monitoring, identification, and tracking defects [58].

With the current social development, various sensors are updated, multi-sensor information fusion has a better solution, various sensors are more perfectly matched together, and better and faster access to the data of USV in the course of travel is the key to USV berthing technology.

Study of berthing posture assessment method based on variable autonomy level

Due to the variability and complexity of the marine environment, coupled with the limited sensing capability of the existing sensors, from time to time there will be raw data that cannot be processed, and if left unprocessed, adverse consequences such as USV collisions and berthing failures will occur.

Autonomy is one of the many capabilities possessed by USV [59]: sensing, observing, analyzing, interacting, planning, deciding, and executing to accomplish the tasks delivered by humans. For variable autonomy levels as evident from Table 1 above, dividing the evaluation of unmanned systems for their autonomous capabilities, the autonomy of USV under different levels corresponds to different manning and berthing methods. USV generally have the following four components: observation, judgment, decision, and action. The paper [60] proposes a berthing posture evaluation method based on variable autonomy levels. The problematic sensor data, including anomalies such as unidentifiable dynamic obstacles, the uncertainty of berthing targets, and missing information about its positioning, are uploaded to the shore-based server to request manual decisions from the monitoring personnel. Most of the anomalies can be solved based on the human–machine interaction information. However, if no manual feedback is received, the USV will increase its autonomy level (as shown in the flow of the autonomy level link in Fig. 9 [60]) to ensure its safety by assessing the current environment accurately in real-time through the berthing posture assessment method with variable autonomy level. Selecting the autonomy level of the system. The robustness of the USV berthing can be improved by the berthing posture assessment method based on variable autonomy levels. The link of the whole variable autonomy level berthing posture assessment method (Fig. 10).

Fig. 9

Autonomous grade link process

Fig. 10

Architecture of multiple example learning and variable autonomy levels for berthing potential awareness

Improving the efficiency of USV in performing their tasks, optimizing the structure process of berthing posture assessment method with variable autonomy level, satisfying the information requirements of USV system in working, improving the accuracy and versatility of the system, enhancing its ability to handle risky environment and communication, making the USV berthing with better effect and improving the efficiency of the system.

The situational awareness system of USV is similar to human eyes, ears, nose, and other senses, which is the information basis for USV to make autonomous berthing decisions. With the rapid development of dynamic sensing fusion, high-performance computing decisions, high broadband information interaction, and high-speed USV platform technology, the future development trend of intelligent and unmanned maritime USV has become the industry consensus.

Judgment of USV berthing

The modeling of the berthing process, namely includes global modeling of the map, target point modeling, environment modeling, modeling of external natural conditions (wind, waves, natural objects.), and modeling of the ship itself. For the modeling of the ship itself, Wang [61] proposed a 4-degree-of-freedom ship maneuvering motion system identification modeling based on support vector machines; Zhang [62] used virtual reality technology to simulate the state of the ship 3D model in different scenarios and proposed a study on the 3D modeling design of the ship based on virtual reality technology; Ye [63] analyzed and improved the basic genetic algorithm and used the algorithm to control the parameters in the ship operation model to obtain a better ship model; Ruiz et al. [64] studied the maneuvering test in two different water depths, in calm water and regular waves; Li et al. [65] elaborated on the study of the effect of wind, waves and tidal currents through Norrbin model on the influence of ship turning ability by means of Norrbin model. Shi et al. [66] proposed a regional multi-ship collision risk assessment modeling method based on fuzzy logic considering the influence of ship crossing angle as well as navigation environment, which provides an important basis for the monitoring of maritime regulatory authorities, crew's perception of ship risk and safe berthing of ships.

Multiscale raster modeling method based on electronic nautical charts

Berthing path planning for an USV is how to find a collision-free path from a given starting point to a berthing point in an environment with obstacles so that the USV can safely and collision-free bypass all obstacles during its motion. The USV has to build a representation of the obstacles present in the environment it is in, in its internal world model, before planning. The first thing to do is to transform the description of the environment from its original external form through a series of processes into an internal world model suitable for planning. A reasonable representation of the environment is what facilitates the reduction of the search volume in planning and the reduction of the Spatio-temporal overhead. It is on this different environment modeling that different planning approaches are based. The global path planning of an unmanned surface boat rasterized by electronic charts [67] is proposed based on a multi-scale raster modeling approach (Fig. 11).

Fig. 11

Comparison of ordinary raster and multi-scale raster modeling methods

Due to a large amount of raw data on the marine environment (including layer data of continents, islands, reefs, port facilities in electronic charts, data from AIS, navigation Radar, LIDAR, MMW-Radar, photoelectric equipment, sonar equipment of obstacles. and also need to consider wind and wave currents and other data), only reasonable environmental data representation can simplify the map model, reduce the search volume of berthing paths, reduce data storage volume and search time. By analyzing the description characteristics of electronic nautical charts, a multi-scale marine environment modeling method is proposed. For dynamic obstacle data, when modeling the unnavigable area, the attitude prediction of the obstacle is applied to consider the trend of its speed and steering, together with the speed modeling and steering modeling of the dynamic obstacle, to provide a more reasonable map model for the path search.

In summary, the traditional raster method has the advantages of easy maintenance, high adaptability, and direct image; however, it cannot solve the direct contradiction between accuracy, real-time, and time consuming; through the improved multi-scale and variable-scale raster method incorporating human physiological perceptual characteristics imitated, the cognitive effect on the surrounding environment can be accomplished more finely, in real-time and swiftly [68].

Research on multi-scale modeling methods that consider complex environmental information

Before planning, the USV needs to build a map model considering the marine environment and the surrounding obstacles. The description of the environment is first transformed from the external raw data through a series of processing to build an internal world model suitable for optimal path search. Regarding environment modeling, there is no lack of research in this area in the last 20 years. The paper [69] discusses that ship collisions pose a serious threat to navigation safety. An ordered Probit model is used to fit the state equation showing the degree of risk. And conclusions were drawn by interpolating the collision risk degree judged by humans and the ordered probability model; the paper [70] proposed a ship collision risk assessment method using AIS and historical accident data. A collision candidate detection model based on fuzzy quaternion ship domain theory is established; paper [71] establishes a fuzzy quaternion ship collision field to better describe the risk condition near the bridge area; paper [72] describes the ship risk assessment and identification field. The method investigates the collision risk of ships from the aspect of a single ship. It is concluded that the model of the shipping field is closely related to the dynamic and static characteristics of the ship.

Therefore, by analyzing the descriptive characteristics of electronic nautical charts, a multi-scale-based marine environment modeling method is proposed. For dynamic obstacle data, when modeling the unnavigable area, the posture prediction of the obstacle is applied to consider the trend of its speed and steering, together with the speed modeling and steering modeling of the dynamic obstacle, to provide a more reasonable map model for path search and realize the planning of berthing path with high efficiency.

USV berthing decision

The behavior decision system of the USV is similar to the human brain, which can comprehensively judge the berthing situation and make a reasonable berthing behavior decision. Decision-making is the premise of USV berthing control, and only with deliberate behavioral decision-making can the intelligence of USV be reflected. And the agility of USV berthing action requires the design of a high-precision berthing decision algorithm.

Collision hazard warning and intelligent collision avoidance for USV in complex marine environments [73] research

When an USV performs a berthing task autonomously, it is necessary to model the marine environment around the USV and plan an optimal route to the berthing target. In the process of autonomous piloting control, there exist both negotiated collision avoidance (which requires course adjustment according to the "international maritime navigation rules" [74], reactive collision avoidance (which will avoid obstacles and try to approach the current waypoint or the target point at the best speed and manner, which will not consider the "international maritime It will take the fastest speed and direction to avoid the obstacle regardless of the original course or target point constraints) are not described in detail here.

Comprehensive research on the impact of marine environmental loads such as strong winds, waves, high currents, and obstacles (dynamic obstacles such as islands, reefs, and ships) on the navigation safety of USV, and propose collision hazard warning and intelligent collision avoidance algorithms for USV; propose to divide collision hazard situations into no need to avoid obstacles, negotiated collision avoidance situations, dangerous situations, and emergencies, with emergencies having the highest priority, and through a three-tier collision avoidance algorithm Improve the berthing safety of USV in high sea state. Multi-level warning of potential navigational hazards to achieve USV avoiding all obstacles during navigation. The paper [75] proposes an uninterrupted collision-free path planning system that facilitates sampling tasks in complex marine environments and improves the operational performance of multipleUSV.

Research on autonomous decision-making methods for USV under dynamic and multivariable scenarios

However, there can be a hundred and one cases, which require the study of decision-making methods based on variable autonomy levels. USV face dynamic and changing water environments, how to ensure the safety of the boat and can accurately berth to the designated berth, need to study the autonomous behavior of the USV decision-making methods. Based on the research of global route planning and intelligent collision avoidance of USV, the decision method based on variable autonomy level is studied. When the berthing path planning fails, berthing path tracking fails, and the output heading speed is abnormal, the USV is required to make a reasonable next-action decision autonomously and prompt the shore crew to make an auxiliary decision through the system. Through manual auxiliary decision-making, the practicality of the USV in the actual marine environment can be improved.

To achieve autonomous and safe berthing of the USV in the dynamic and changing external environment, it is necessary to be able to plan the optimal berthing path in real-time according to the changing mission scenarios; and to avoid dynamic obstacles in the environment and reach the target point of berthing. The block diagram to realize the autonomous berthing decision of the USV under dynamic and variable task scenarios (Fig. 12).

Fig. 12

Block diagram of autonomous berthing decision for USV

Research on autonomous decision-making methods for USV based on humanoid intelligent schema [58]

The structure diagram of the USV autonomous berthing control system (Fig. 13) is roughly divided into three rows and three columns, three rows: motion planning control, information perception and understanding, and knowledge learning and memory; three columns: central commander level, organization and coordination level, and basic control level. The "motion planning and control" row is the functional module of the USV to generate various actions; the "information perception and understanding" row is the progressive feedback perception process of the USV, which compares the expected value with the actual effect achieved at different levels, and thus continuously generates It uses various feedback information to re-present the external world, thus forming the "internal model" of the USV, through which the execution process of the program can be controlled; "knowledge learning and memory" is the knowledge base of the unmanned The "knowledge learning memory" is the knowledge base of the USV at berth, which saves the maneuvering performance parameters, various feature memories, feature models, control modes, thrust rules, various navigation rules, and various optimized control parameters.

Fig. 13

System structure of USV autonomous berthing system

The idea based on hierarchical progressive control [76] is mainly reflected in the following three columns: central commander level, organization and coordination level, and basic control level. The central commander level is mainly responsible for the coordination and decomposition of motion control tasks, and it is the "brain" of the unmanned ship berthing, which is the embodiment of intelligence; the organization and coordination level can organize different functional modules according to the motion tasks of the higher level, and coordinate the role of each functional module to achieve the purpose of self-organization, self-adaptation, and self-decision. The basic control level is the lowest level of ship motion control, which has the highest control accuracy.

Using an autonomous decision-making method for USV based on humanoid intelligence schema, we realize the autonomous decision-making for route planning and collision avoidance maneuvering of USV in dynamic and variable ocean scenarios. It improves the ability of USV to cope with dynamic and changing scenarios and then enhances the robustness and reliability of USV for autonomous berthing in the actual marine environment.

USV berthing path planning

In the whole berthing process of the USV [77], the path can be divided into far-end berthing trajectory planning and end trajectory planning at the near-shore terminal. Good path planning is the basis for the success of the whole berthing process. The so-called distal berthing trajectory planning refers to the process that the USV wants to arrive at the target berth from the far shore. In the whole process of approaching the berth, the influence of the external environment during the trajectory needs to be fully considered; the first task in the trajectory process is how to effectively carry out the path planning, complete the navigation, and avoidance of collisions and various obstacles (Fig. 14).

Fig. 14

USV berthing planning schematic

Regarding berthing path planning, the dynamic path planning algorithm for unmanned ships based on deep reinforcement learning [78] can continuously improve the learning ability and the ability to combine information so that a high-precision and the most intelligent path planning approach can be obtained; the new algorithm obtained by improving the A* algorithm [79] can achieve obstacle avoidance more precisely and quickly to complete the collision avoidance process; the artificial potential field is one of the most common paths planning methods, but its experimental process also has more or fewer problems appear, fuzzy improvement of the artificial potential field method [80], to obtain a more effective planning approach; Ou et al. [81] studied the current path planning algorithm and sampling-based autonomous navigation algorithm for autonomous mobile robots. The paper [82] proposed a combination of water shoreline detection and visual odometry to, achieve accurate positioning of berths and berthing control. The paper [83] improves the ship berthing motion control based on improved simulation positioning and mapping algorithm, which improves the problems of fast speed and high perception accuracy for ship berthing requirements. Pape [84] proposes an improved network of YOLOv5s-CBAM to solve the real-time sea obstacle detection problem of autonomous berthing in practical application, which makes accurate berthing at the end of berthing. The positioning accuracy of the Global Navigation Satellite System (GNSS) commonly used by ships is susceptible to environmental impacts, which can lead to threats to the precise positioning of the last 10 m of the ship. To improve the existing problems, Wei Liu et al. [85] improved the GNSS vector tracking loop to enhance the USV navigation performance, resulting in an error rate reduction of 85. 4%; Sakakibara et al. [86] monitored the ship-to-shore distance and berthing speed by installing a docking sonar system on the quay to interact with the ship's forward-looking sonar to provide an accurate final berthing Perkovic et al. [87] proposed a laser ranging docking system using more accurate and responsive LIDAR and MMW- Radar, and applied it to measure the ship's distance and lateral velocity relative to the quay; Wang et al. [88] associated the application of 3D printing into the berthing system, where a 3D laser scanner accurately captures and constructs the target object's 3D surface geometry, which in turn allows the 3D laser scanner to measure the distance from the bow and stern to the quay, the angle of approach [89] and the speed; Martinsen et al. [90] used Radar and ultrasonic distance sensors to measure the distance to the ship's berth for the next decision; P. Leite et al. [91] used 3D Radar scanning to extract the geometric features of the berth and estimate the position and orientation of the ship relative to the berth to perform accurate berthing actions.

The paper [92] provides a way to divide the automatic berthing of a ship into two stages by using a path planning algorithm according to the berthing rules of the ship: the first stage is to calculate the rudder angle by calculating and combining with the data processed by Radar, sonar and other equipment to drive the ship to the turning point; the second stage is to adjust the ship's attitude after the ship reaches the turning point and then drive the ship into the berth along the wire. The specific process of automatic berthing is introduced in the paper when the ship is berthing on the left and right side of the berth centerline respectively, after reaching the turning point. The relationship between the distance of the ship and the berth, the relationship of the heading angle, and the steering operation required when the ship reaches the steering point are given in Fig. 15. (Six microwave Radars are installed on the experimental ship in the paper, and the distance of each Radar from the shoreline is Si, and Φ is the bow angle).

Fig. 15

Schematic diagram of radar installation, ship berthing and ship-related quantity relationship

USV berthing path planning research based on meteorological data and marine geographic environment information

The difficulty of the path planning problem [93], the planning process should consider the influence of the USV's motion characteristics and the natural environment in which it is located (e. g., wind, waves, and currents). Nowadays, there are studies such as the design of meteorological routes for ships by A* algorithm [94]. To deal with the more complex marine environment, such as continents, islands, reefs, and other unnavigable areas, as well as strong currents, large waves, and other harsh sea conditions (which can have an impact on the berthing safety of USV), the study of USV berthing path planning methods based on meteorological data and marine geographic environment information can make USV automatically avoid all unnavigable areas in the planning process and minimize the impact of time-varying dynamic wind and wave currents on the USV The impact of time-varying dynamic wind and wave currents on the navigation of the USV can be minimized. The berthing decision system will start "collision avoidance hazard judgment" when dynamic obstacles appear on the ferrying path.

USV berthing path planning based on heuristic A* algorithm [95]

The A* (A-Star) algorithm is an optimal path planning algorithm proposed by Hart et al. [96] and others, which is based on Dijkstra's [97] algorithm, a heuristic, a classical graphical algorithm that is widely used to solve route planning problems [98].

Berthing path planning for USV refers to how to find a path from a given starting point to a berthing point in an environment with obstacles so that the USV can safely and touch-free bypass all obstacles during its movement. The A* algorithm is applied as the theoretical basis to optimize the route, and 8 search directions are used as an example to realize the basic path planning. As shown in Fig. 16 [95] below, it is a two-dimensional raster model with 8 search directions, and each hollow circle represents a node. Each grid point as a node, green dots represent the starting point, pink dots represent the target point, blue dots represent the 8 search directions, black boxes represent obstacles, and the three numbers in each square from top to bottom represent the F value, G value, and H value of this node. The red line is the final planning path.

Fig. 16

Path planning method considering wind and waves

Considering the complex and changeable sea conditions, a path planning method considering wind and waves is proposed based on the A* algorithm, which can optimize the berthing route of the USV in real-time, so that it can avoid the area where large wind and waves may be expected to occur, and derive a navigation path with the shortest time consumption while guaranteeing the safety of the USV. After the weather information is imported, the wind and wave realities at each grid point can be known, so the weather influence factor is added to the value function of the A* algorithm, and according to the different weather realities at each grid point, its influence on the ship's speed is different, and the ship can lead to serious stall conditions in a windy and rough environment, so the path is planned using the A* algorithm to select a sailing path with the shortest The route is planned by using A* algorithm to select a route with the shortest sailing time, and at the same time effectively avoiding the area with high wind and waves.

This algorithm, through multiple data calculations, obtains the best heading from the departure point to the target point, so that the ship can obtain an effective and reasonable route when traveling. Furthermore, the algorithm is combined with the electronic chart and the influence of the changes of wind and wave currents, so that the route can be further and dynamically judged, to make the docking of the ship more intelligent, and the combination of this method with more external data may also be a research direction in the future.

Berthing path planning considering USV maneuvering constraints and environmental disturbances

Based on the nonlinear mathematical model of USV maneuvering motion, the effects of wind pressure and wind pressure moment, tidal current, and other disturbing forces on the maneuvering motion of USV are studied. By parameterizing the maneuvering data such as the minimum slew diameter, tactical diameter, longitudinal distance and inverse transverse distance of the USV, the maneuvering performance of the USV can be quantified so that the planned path is consistent with the maneuvering motion characteristics of the USV. When berthing, the general speed is low, and the USV heading is easily disturbed by wind and wave currents, and the factors of environmental disturbance need to be fully considered when planning the berthing path (Fig. 17).

Fig. 17

Berthing path planning considering multi-objective constraints

Due to the interference of the marine environment, when berthing, the USV needs to choose the direction with the least influence of wind and wave currents for berthing, generally, the top wind mode is chosen for berthing, which is beneficial for the USV to control the heading, as shown in Fig. 18

Fig. 18

Schematic diagram of berthing path planning under different environmental influences

Under the north wind conditions, the USV needs to turn to the north direction for berthing operations, to allow the USV to autonomously choose the best direction for berthing, the method of adding virtual berthing obstacles in the map is proposed, using the previous raster modeling and A* search algorithm, which is to realize the autonomous choice of berthing direction.

Smooth path planning for USV based on an ant colony algorithm

In the path planning before berthing, how to accurately and precisely navigate the USV from the complex environment, overcome the problems of large steering angle, many paths inflection points, the high energy consumption of navigation, and finally navigate to the target berth. A comprehensive study of the improvement of the ant colony algorithm and the problems in the algorithm. In the path planning before berthing, how to accurately and precisely navigate the USV from the complex environment, overcome the problems of large steering angle, many path turning points, and high energy consumption of navigation, and finally navigate to the target berth. A comprehensive study of the improvement of the ant colony algorithm and the problems in the algorithm.

Paper [99] on fusing particle swarm and ant colony algorithms to obtain a new path planning method that can substantially improve the initial path finding efficiency as well as the global search capability, thus significantly and effectively reducing the preliminary time for path planning; paper [100] by fusing A* algorithm and wolf swarm mechanism, thus improving the ant colony algorithm, which makes excellent reliability and convergence; papers [101, 102] improve the algorithm with better global search capability and convergence by extensively collecting important factors such as distance traveled, road smoothness, elevation gradient, and steepness during the travel path and a newly designed information update mechanism. Paper [103] by combining the ant colony algorithm with the Bayesian network and introducing the maximum association length parameter, the method has better effectiveness and more stable convergence after simulation tests.

The paper [104] is an USV smooth path planning method based on the ant colony algorithm. The method is exemplified by its use of the raster method for environment modeling; setting the initial point information; through calculation, deriving the transfer probability to proceed to the next step; updating the local pheromone; judging the correctness or incorrectness of reaching the target raster; deriving the current optimal path and performing global information update; judging whether the maximum number of iterations is reached, and finally obtaining the optimal path through post-processing, and performing path smoothing The optimal path is finally obtained through post-processing, and path smoothing is performed. The changed ant colony algorithm can effectively improve the problems of the traditional ant colony algorithm, improve the efficiency of path planning; the ability of path search; and the effect of path smoothing processing (see Fig. 19).

Fig. 19

Smooth path planning method of USV based on ant colony algorithm

The key technology of USV autonomous berthing control

For the autonomous berthing problem of this USV, there are five main research aspects: USV berthing situation sensing and evaluation under complex sea conditions, USV autonomous berthing decision, intelligent berthing control algorithm, variable autonomous level decision, and USV autonomous berthing test validation (Fig. 20).

Fig. 20

Technology roadmap to be adopted for the study of autonomous berthing control methods for USV

As an intelligent mobile body on the water's surface, an USV can be regarded as a kind of robotic system that can sense the external environment through various sensor systems to realize autonomous movement and accomplish certain tasks in the complex and unknown marine environment. Since the actual marine environment is dynamic, uncertain, and complex, the traditional control methods and the architecture based on the cognitive model have been difficult to meet theoperational requirements of the USV in the complex marine environment. The internal structure of the USV berthing control system (Fig. 21).

Fig. 21

Internal structure of USV berthing control system

The control of the hull is also an aspect worth considering during the berthing trajectory of the ship, and the instability of the hull itself can cause the whole trajectory to face problems. This leads to more or fewer difficulties. The constrained nature of the ship can lead to the navigation of the ship and may cause significant safety problems. By using a nonlinear genetic algorithm [105, 106], the small local problems are optimized and improved; facing the collision avoidance path problem of the ship in the course of navigation [107], the changes in the ship load and navigation environment are added to be considered thus improving the maneuverability of the ship; the influence of the natural environment on the ship navigation is also obvious, and the paper [95] considers the complex meteorological changes to improve the intelligent ship dynamic route planning to improve the maneuverability of the ship; using a single ship group and multiple ship groups, loading the conditions constrained with the ship navigation to reduce the limitations arising from the maneuvering, thus obtaining excellent route planning functions [108].

The slewing performance of the hull of the small USV is good and easily affected by wind and wave currents; in terms of the power system, its slewing index will change with the change in engine speed. In other words, the USV system is a complex system with time-varying, non-linear and vulnerable to disturbance. It is difficult to ensure the global stability of the USV bow adaptive system designed by using the local parameter optimization method, plus in the actual marine environment, the hull slew index may change rapidly and the change magnitude is relatively large, which may lead to system instability, so better control design methods are needed to ensure the global stability of the system.

Design of berthing controller for USV using model-referenced adaptive method

Model-referenced adaptive control techniques have been widely used in aircraft autopilots, ship autopilot systems, photoelectric tracking telescope follow-on systems, silicon-controlled speed control systems, and robotic control systems. The basic principle of using the model-referenced approach to design an adaptive controller is that the control parameters are modified according to some function criterion of the difference between the actual output of the controlled object and the output of the reference model, trying to make the generalized error between the actual output of the controlled object and the output of the reference model tend to zero so that the controlled system achieves or approaches the desired dynamic behavior. The principle structure of a typical model-referenced adaptive control system (Fig. 22).

Fig. 22

Schematic structure of model reference adaptive control

In the model-referenced adaptive control system, the ideal reference model represents the designer's desired dynamic characteristics of the actual system. When external conditions change or disturbances occur, the dynamic response of the controlled object also changes accordingly, deviating from the designer's desired dynamic response characteristics. To compensate for the impact of the external environment or other disturbances on the actual system, an adaptive mechanism needs to be designed to adjust the parameters of the adjustable system to reduce the error between the response of the reference model and the response of the actual system and the response of the ideal model.

Model-referenced adaptive berthing control algorithm based on Lyapunov stability theory [109]

According to the designed adaptive control law of USV heading and speed, an auxiliary signal generator is introduced, which constitutes a feedback gain adaptive mechanism, and together with the feedforward gain adaptive mechanism, it constitutes the whole adaptive mechanism and then forms the model reference-based adaptive USV heading controller and speed controller, through the continuous learning of the controller and optimization of the control parameters, and finally achieves that regardless of the actual USV Hongde Qin et al. [110] combined Liapunov stability theory, back-stepping method, and tan-type BLF to obtain a control algorithm with smaller error.

Automatic berthing control method based on the artificial neural network [111]

Artificial Neural Networks (ANN) [112], is a research hotspot that has emerged in the field of artificial intelligence since the 1980s. The use of ANN in intelligent combination with other theories has received a great deal of attention in the marine world. FL (Fuzzy logic) and artificial neural networks have been proposed, discussed, and evaluated in conjunction with each other intelligently as "neuro-fuzzy autopilots" [113,114,115].

Artificial neural networks abstract the network of human brain neurons from the perspective of information processing and build some kind of simple model to form different networks according to different connections. An artificial neural network is a nonlinear, adaptive information processing system composed of a large number of interconnected processing units. It is proposed based on modern neuroscience research results and tries to process information by simulating the way the brain's neural network processes and remembers information. The experimental results of artificial neural networks [116] have four basic characteristics: nonlinear, nonlimited, nonconstant, and nonconvex. Thus, the neural network is an obvious breakthrough for solving the problems faced by USV control and travel path: nonlinearity, unknown model design, and weak anti-interference ability. The ship first collects the real-time dynamic environment outside the path of travel, and after data processing by the neural network, passes and calculates the network weights and deviations through the BP neural network algorithm [117], followed by the mathematical model of the ship berthing process established. Of course, if the initial real-time variables are input, different output algorithms for automatic berthing control will be derived. This design process (Fig. 23) [118]. In the paper [119], by improving the problem of large deviation of the ship's heading from the shoreline at the end of berthing of the previous ANN controller, some data from the berthing process is used as controller input instead of extracting all information from the berthing process, as a way to improve the training of the controller and thus obtain excellent berthing results. However, the neural network control algorithm, which needs to pass a large number of training and the difficulty ensuring real-time in the control process, makes the neural network control method needs more practice [120].

Fig. 23

Neural network berthing control process

Deep learning-based 3D perception algorithm for autonomous berthing applications [121]

The 3D perception algorithm is a deep fusion based on combining multi-sensor point cloud detection technology [122], image data detection technology [102], and sensor data detection technology [123] to improve the accuracy of target detection and perception, which is applied to find a suitable berthing location. Accurate environment perception and scene construction are performed in deep learning by cameras, Radar, sonar, and GPS, which are applied to the ship to enable fast detection and discrimination of the surrounding environment and improve the rate and accuracy of berthing. The construction of the perception algorithm is shown in Fig. 24, and the expected accuracy is obtained through its architecture with the ship control scenario and the situational awareness network, and finally based on the autonomous berthing control system, it is successfully applied in the actual berthing test, where the data and time cannot be accurately matched due to the low efficiency of the algorithm operation, which will make the error of the unmanned ship slewing. However, the test verified the reliability and practicality of the method, and the 3D perception algorithm based on deep learning will become a new research direction and idea for autonomous berthing.

Fig. 24

Construction of the perception algorithm

Vision servo-based method for autonomous berthing of USV [124]

Vision servo, in general, refers to the behavior of automatically receiving and processing images of real objects through optical devices and non-contact sensors, and through image feedback information, the machine system can perform further control or corresponding adaptive adjustment behavior of the machine. The simulated scenes of the berths docked are obtained on the unmanned surface boat by the vision acquisition system on board, from which the markers of the target berth are extracted as tracking objects. The distance and orientation between the USV and the target berth are judged by the size of the marker image, the different positions of the geometric center point of the image in the vision system, and the change of the marker image shape during the motion of the USV, and a route is simulated and planned to guide the USV to complete the autonomous berthing task. When the markers of the target berth are easily disturbed by the outside world, the berthing process will be affected and the berthing and path planning will not be accurate. However, this will become an important direction for future research.

Model reference-based adaptive berthing control

To achieve high-precision berthing control, the control input required to bring the tracking error to zero is calculated by comparing the error that exists between the berthing path point and the current USV motion information and the trend of the error change using a model-referenced adaptive control algorithm. Through a large number of berthing control practices, the controller will continuously optimize the control parameters to form a high-precision adaptive berthing control system suitable for this USV. To cope with the sudden control failure problem, the USV berthing control architecture with variable autonomy level is proposed to improve the reliability of the USV berthing navigation. The block diagram of the model reference-based adaptive berthing control (Fig. 25).

Fig. 25

Block diagram of berthing control based on model reference adaption

The basic principle of using the model reference method to design an adaptive controller is that the control parameters are modified according to some functional criterion of the difference between the actual output of the controlled object and the output of the reference model, trying to make the generalized error between the actual output of the controlled object and the output of the reference model converge to zero so that the controlled system achieves or approaches the desired dynamic behavior.

Combining all these functions in one package gives a perfect berthing system. Therefore, future research still needs to explore the ability to improve the anti-disturbance capability and deal with nonlinear problems, to improve stability and swiftness when responding to problems, to be resilient in the face of different and changing dynamic environments, to completely disengage from human or assistance vessel participation.

Opportunities and challenges for USV berthing technology

There is still a big difference between theory and practice, the actual navigation environment is complex and changeable, and there are various problems; there is a big difference between artificial intelligence, computer programming, and human brain intelligence. Therefore, USV berthing technology also faces many challenges and opportunities. The past research work is compared with future work as shown in Table 3.

Table 3 Past research work comparison with future work

• (1) Control algorithms, as an important development direction for unmanned navigators, to improve operational effectiveness and achieve tasks that cannot be accomplished by a single navigator, will have great potential for application in military and civilian fields [125].

• (2) For environmental perception in complex scenarios, the complex sea state environment is divided into static and dynamic environments. The static environment includes channel environment; berth; signal position; navigation facilities. the dynamic environment includes incoming and outgoing docked ships; sudden celestial events, and meteorological environment. At present, these are only carried out under simple experiments, for this complex environment navigation or have limitations. Therefore, there is still a long way to go for direct input into life, production, and the military.

• (3) The processing of big data, it is also a major difficulty for the USV to berth. Need to enhance the ability to handle the four key aspects of the USV observation, judgment, decision-making, and action. Therefore, the interactivity and synergy of hardware and software on the USV is also an important link, which also needs a long time to explore.

• (4) The uncertainty of the ship model [111], has a potential impact on the berthing of the ship due to its nonlinearity during the travel path. The current common modeling methods are computational fluid dynamics simulation method, ship empirical formula method. All of the above methods have their advantages and disadvantages. For the accurate modeling of the ship and the model processing of the real-time dynamics of the ship, it will also become an important research direction for the research in the future.

• (5) Algorithm research should be carried out on the cooperative work of multiple unmanned surface boats [126], so that the boats can have the advantages of multi-ship cooperative work.

• (6) Make full use of the existing theoretical, technical basis, and simulation test results to avoid unnecessary duplication of work, resulting in the waste of human, material, and financial resources. Achieve high efficiency and high results [127].