1 Introduction

Because of the growing population and increased per capita consumption of aquatic products, global fishery production faces a serious challenge. Although technological advancements have been applied in the fishing industry, production has been not increased significantly. In fact, these advancements have even caused undesirable cycles of overfishing worldwide [1]. Therefore, many countries have turned to aquaculture to satisfy their demand for aquatic products [2]. In Taiwan, aquaculture mainly employs manual methods. Over the last 30 years, many high-tech companies have been established in Taiwan, and some of these companies are dedicated to aquaculture development. However, whereas IT (Information Technology) companies are turning into multinational corporations, aquaculture companies have experienced limited development. Many aquaculture owners maintain family-based aquaculture farms and do not use IT advancements to increase the efficiency of their operations. Moreover, conventional aquaculture owners in Taiwan are extremely reliant on inland water and soil resources. The aquaculture industry in Taiwan is entering a phase where transformation is required. Industry, government, and academia in Taiwan have recognized the necessity of changing inland aquaculture to offshore aquaculture.

In response to high global demand for aquatic products, offshore aquaculture, particularly cage culture [3], is a preferred solution. However, cage culture earnings are affected by the complicated, harsh offshore environments and extreme climates; thus, cage culture has a high operation barrier to entry. Unless operated by experts in the field, cage culture often results in difficulty or failure. Most fishers are hesitant to adopt this culture method, and its promotion is also challenging. Only countries with favorable sea conditions, such as Norway, can promote this culture method on a large scale. Cage cultures in Norway and Chile account for nearly half of total omni salmon production [4]. To promote cage culture, the high barrier to entry of operations must be addressed. Therefore, the current research team researched and developed a smart cage culture management system based on Artificial Intelligence and Internet of Things (AIoT) to solve relevant problems and promote large-scale cage culture. This system may increase omni fish production to meet the needs of a growing population.

The goal of the current research team was to apply artificial intelligence innovation technology to establish a useful smart cage culture management system, which includes a omni-IoT (Internet of Things) system, underwater aquatic creature analysis, and artificial intelligence feeding system modules [5,6,7,8,9,10,11] to minimize production costs and maximize fish production. Thus, an aquaculture industry with world-class competitiveness can be established. The economic benefits can prompt fishery transformation. Fishers can use the management system to obtain real-time information [12, 13], achieve optimal aquaculture performance, and reinvigorate the developmental energy for limited fisheries.

2 System Architecture

To establish the AIoT system for smart culture management, we designed a system model. Figure 1 illustrates the system architecture. We integrated sensors, underwater camera and a communication system into a platform and placed it in a cage. Data from an autogiro and a remotely operated vehicle (ROV) were integrated into an Omni IoT system, which can integrate the monitoring and sensing system of feed delivery. All the obtained data is transmitted to an onshore base through communications systems and transmitted through an Internet connection to a cloud system.

Fig. 1
figure 1

System architecture

Full size image

Currently, our research team uses the Ocean Cloud of National Taiwan Ocean University (NTOU) as the data collection center. Figure 2 depicts how sensor and video camera data collected at the NTOU cage culture data platform are presented on the Ocean Cloud platform and monitoring center. The data collected through the Omni IoT system can be used for AI calculation, such as analysis of aquatic creature images and AI feeding systems. Figure 3 shows the flow chart of data collection of the Omni IoT system, transmission of the data to the cloud system for AI computation, and the feedback to the system. All big data collected are stored in the Ocean Cloud to facilitate the AI computations of the feeding system, fish behavior monitoring and analysis system, ROV system, and autogiro AI computation, forming an efficient AIoT smart cage culture management system.

Fig. 2
figure 2

Presentation of collected onsite data in Ocean Cloud

Full size image
Fig. 3
figure 3

System operation flow chart

Full size image

3 Results and Discussions

The AI smart cage culture management system includes Omni IoT, underwater aquatic creature behavior monitoring, and AI feeding system modules. In the Omni IoT system, water quality sensors (including dissolved oxygen, temperature, salinity, and pH values), cameras, an ROV, and an autogiro sensing system were integrated to collect environmental parameters and image data from the offshore cages and transmit the data to a waterproof platform. Subsequently, a directional antenna was used as the backbone of long-distance wireless transmission. All data were transmitted to the data center before data storage, analysis, and AI computation. The resulting information could serve as a reference for aquacultural operators.

The data collected using the Omni IoT system are available on the Ocean Cloud webpage (Fig. 2). On the webpage, all data are organized. In addition, the unmanned ROV can identify the distance and swimming direction of the fish in the image in relation to the ROV. This tool is expected to facilitate underwater monitoring, obviating the need for scuba divers and thus avoiding risks and lowering the cost of hiring underwater personnel. Figure 4 displays the swimming directions and distance from the ROV, as identified by the ROV.

Fig. 4
figure 4

Identification results of swimming directions and distance of tilapia

Full size image

The research team established an autogiro sensing system, on which sensors and cameras were installed in combination with a processor to perform multiple tasks and functions, such as automatic water quality measurement, and lower costs. The collected data were uploaded to the data center to establish a database. The current system can use Raspberry Pi processors and webcams to achieve simple color and size identification. The advantage of this computer system lies in its low weight, compact size, and convenience for task performance. During the task execution process, the sensor readings can be observed. Figure 5 reveals the measurement operations of an autogiro, and the bottom of the Fig. 5 presents the variation of the measured temperature. The data revealed different temperatures in the air and water.

Fig. 5
figure 5

Temperature measurement by the autogiro

Full size image

By employing sensors, cameras, an ROV, and an autogiro, the omni IoT system can overcome the harsh offshore environmental limitations. Because of the combination of communication and waterproof technologies as well as system installation in the offshore area, the system can not only monitor water quality but also help monitor underwater images and provide data for AI training.

We also provide functions of fish mobility analysis and fish body length and weight estimation using AI techniques, namely combining AI and image processing to achieve real-time estimation of the body's length and weight and the current mobility status of the aquaculture creatures under the premise of non-invasive and non-intrusive to reduce the problem of sudden death of creatures. These functions allow the aquaculture operator to quickly determine whether the farmed creatures are suffering from diseases, and carry out spreading control with necessary countermeasures. Our research results can be applied to various creatures in the future, such as crabs, cuttlefish, shrimps, etc. To achieve our purpose, the first thing to overcome is the detection of serious interference due to underwater visibility. First, information provided by the deep learning-based object detectors is utilized to detect the body of fish. Well-known object detectors include: Faster-RCNN and YOLOv3. The advantage of Faster-RCNN is that the accuracy of the bounding box is more accurate than YOLOv3, so it is suitable for tasks that require high accuracy (e.g. length and weight estimation). On the other hand, the advantage of YOLOv3 is that the processing speed (fps) is faster than Faster-RCNN, and it is suitable for fast processing tasks such as mobility analysis. To train Faster-RCNN and YOLOv3, we prepared 1204 labeled images and augmented 2986 images to achieve the satisfactory object detection. After detected the fish, the length and weigh are calculated using ordinary stereo matching algorithm and linear regression method, respectively. Figure 6 shows the results. Fish body length and weight estimation have reached 90% accuracy for tilapia.

Fig. 6
figure 6

a Object detector result of Tilapia, b the estimation result

Full size image

In this work, fish mobility is divided into three levels, of which the mobility is calculated based on tracking the fish speed over a period of time. Figure 7a shows the tracking results, and Fig. 7b illustrates the distribution of three different swimming speed values, with blue, green, and red indicating high, normal, and low levels, respectively.

Fig. 7
figure 7

a Tilapia mobility analysis and tracking results, b the swimming speed of fish (high in blue, normal in green, low in red, and mean in black)

Full size image

To assess AI feeding, we developed a set of core software to analyze the intensity of splash caused by fish competing for feed. The analysis results were used as the basis to determine the feeding time. Figure 8 depicts the identification of fish appetite through the splash intensity. The system adjusts the feeding time according to the level of competition of fish competing for feed. The experimental results verified that the loss of feed was reduced.

Fig. 8
figure 8

Identification of feeding using splashes. From left to right, the images respectively show no competition for feed, weak competition for feed, and strong competition for feed

Full size image

The system was also applied in offshore cage culture in TAIWAN Pingtung and experimental field in Gongliao. To obtain the estimation data, cobia cultured in 16 cages of 16 m in diameter and 8 m in depth was used as the example. Although introduction of said system increased capital costs, it was estimated to conserve human costs by 30% and feed costs by 10%. In conventional aquaculture, the conservative estimation of the survival rate is approximately 50%; the introduction of the current system may increase the survival rate to 55%. The internal rate of return was increased from 6.47% without the system to 15.4% after introduction of the system. In 16 cages of 30 m in diameter and 15 m in depth, the internal rate of return increased from 29.86% (without the system) to 41.92% following the introduction of the system. The assessment results highlight the advantages of the introduction of this system. Crucially, the system can also lower the barrier to entry into the aquaculture industry.

4 Conclusions

The goal of this research was to establish an AIoT smart cage culture management system, including Omni IoT, underwater image analysis, and AI feeding system modules. The Omni IoT system comprises sensos, autogiro, ROV, underwater and waterproof platform, communications system and cloud system to rapidly collect big data regarding fish and feeding in the cage environment. Big data can help operators monitor and control parameters such as the environment and fish feeding. Moreover, the big data collected can aid in the analysis of underwater aquatic creature images and AI feeding system modules in AI computation. Underwater image processing technology provides non-invasive and non-intrusive approaches of real-time image analysis and current creature status to aquaculture operators. AI feeding time is determined by the intensity of splashes caused by fish competing for feed. The experimental results verified that the incorporation of AI into the automatic feeding system reduced the amount of residual feed. The proposed AIoT culture technology can help aquaculture operators perform professional aquaculture, reduce the operation barrier to entry of the industry, and promote cage culture.