Background & Summary

Coal will remain the dominant energy source worldwide for decades to come1. Autonomous coal mining machines in longwall mining face can assist or replace human to complete the dangerous mining work, achieve safe and efficient production in coal mine. But it still needs human participation to complete some complex tasks. However, the underground coal excavation of fully mechanized longwall mining face exists many problems, such as poor operating environment, high disaster risk, high accident rate and so on. The intelligence mining has become one of the important ways to address the high-risk underground work, and achieve the goal of safe and efficient underground production2. With the rapid development of artificial intelligence technology, the abnormal situation of equipment, environment and personnel are expected to achieve real-time and accurate detection.

In a fully mechanized working face, hydraulic support is indispensable to the whole face’s safe production. As the core equipment for fully mechanized coal mining, hydraulic support can provide a safe working face, and to move the scraper conveyor and shearer in the working face3. It can also reliably and effectively support coal mine roof, isolate mined-out areas, prevent waste rock into the working face. In accordance with the coal mining process of the fully mechanized coal face, once the hydraulic support plate is not in place or not fully recovered during the working process, it may cause the movement interference between hydraulic support and shearer. Hence, it is necessary to find the status of the hydraulic support guard plate in time and deal with it accordingly. For the fully mechanized longwall mining face, large sized coal is easy to cause scraper conveyor blockage, retention and other abnormal state. It is necessary to automatically identify and track large coal, so as to timely judge and warn the abnormal state of large coal. Towline is used in fully mechanized mining face to ensure the power supply and stable operation of shearer. However, in the process of operation, the traction cable would be broken or be removed from the cable slot due to the stacking of cable clamps, and the cable may be torn off, resulting in underground electric leakage, which may eventually lead to electric shock, gas, coal dust explosion, fire and other major coal mine safety accidents. Therefore, it is necessary to conduct real-time status monitoring and intelligent analysis of the towline to ensure that the fault of the towline is detected and handled in time.

Aimed to protect the personnel safety of fully mechanized mining face, it is necessary to identify and track the mine personnel so as to judge whether the mine personnels are in a safe area. The personnel entering the dangerous area should be timely detected and positioned, the corresponding voice reminder processing should be carried out, and the operation of the corresponding equipment should be stopped at the same time. Except the mine workers entering dangerous areas, the coal miners will have a variety of different postures during work. In the complex working environment, the unsafe behaviours of miners will also easily lead to the increase of safety accidents in coal mine, and the abnormal behaviour of the downhole staff also needs attention at any time. Safety helmet is a kind of safety equipment that coal miners must wear at all times during their work. The area where the coal seam is extracted will cause the pressure to transfer from the hydraulic support to the coal wall, which may increase the pressure on the coal wall and eventually causes the phenomenon of coal wall spalling. The coal falling form roof and the collision between personnel and equipment may cause injury accidents. Hence, the safety helmets are related to the safety of coal miners in fully mechanized mining face, and the wearing of the safety helmet for the coal mine staff also needs real-time monitoring.

The above states of the hydraulic support guard plate, large coal, towline, mine worker detection, personal behaviour and the wearing condition of safety helmet are the key contents of abnormal detection and identification in fully mechanized longwall mining face. The monitoring video in fully mechanized mining face is numerous and updated quickly. The abnormal condition of the working face was judged by specialized personnel through real-time video surveillance in traditional production process, this may result in the abnormal condition not be found in time because the visual fatigue during the long-term work. Therefore, it is of great significance to apply artificial intelligence technology to the analysis, identification and warning of the abnormal state, which includes hydraulic support guard plate, large coal, towline, miners’ behaviour and the wearing condition of safety helmet. The object detection using intelligence data-mining is inseparable from datasets and a large number of samples are required for training to achieve better generalization4,5,6. Hence, it is very necessary to establish an image dataset to identify and warn the underground abnormal conditions of the fully mechanized longwall mining face. Considering that the downhole datasets are still blank at present, this work constructs image dataset DsLMF+ for intelligent recognition of abnormal condition in underground longwall mining face, which mainly consists of the hydraulic support guard plate, large coal, towline, mine safety helmet, coal miners and miners’ behaviour in the fully mechanized face.

Currently, datasets are widely used in automatic driving, object detection, face recognition, natural language processing, text detection, medical and other fields7,8,9,10. Some widely used object detection datasets are as follows: (1) COCO datasets with large-scale commonly used items as target detection objects11,12,13; (2) VOC datasets with people, common animals, traffic vehicles, indoor furniture objects as target detection objects14,15,16; (3) DOTA dataset with airplanes, ships, storage tanks, baseball stadiums, tennis courts, basketball courts, ground runways, ports, bridge as target detection objects17,18,19; (4) TT100K dataset with common vehicles as the target detection object20,21,22; (5) WIDER FACE dataset with facial expression, illumination and posture as target detection objects23,24,25; (6) YOLO format dataset that dedicated to the target detection26,27,28, etc. In addition to these common datasets, we can also customize the dataset through pytorch framework, but the custom dataset format is complex, diversified and poor sharing29. The downhole datasets are still blank at present, in order to construct and facilitate the promotion and application of image dataset of the fully mechanized face in the field of intelligent coal mining, the compatibility and practicability of the coal mine dataset should be taken into consideration.

On the basis of the analysis on the format and production method of the above commonly used object detection datasets, the production of the datasets in this work has been completed by personnels who are familiar with the fully mechanized mining face in coal mine. The Labelimg software has been used to complete the label annotation of datasets in the YOLO format30, which make it convenient to be used in the currently popular YOLO series target detection networks. At the same time, in order to extend the application range of this dataset, the label format of the dataset has also been converted into the COCO format through label format conversion script, and therefore it could be used in the currently popular COCO target detection methods. Of course, in addition to the COCO label format and the YOLO label format, the rest of data label format can also be converted through the tag conversion script.

The image dataset of the fully mechanized longwall mining face (DsLMF+) is of great significance for the application of object detection using intelligence data-mining in the field of coal mine, which is expected to be able to identify and warn the underground abnormal conditions, solving the problems of underground dangerous and inefficient work and thus accelerate the intellectualization of coal mine.

Methods

The construction process of the image dataset of underground longwall mining face (DsLMF+) is shown in Fig. 1, which is mainly divided into the following three steps: (1) Image data collection; (2) Image data filtering; (3) Data labeling.

Fig. 1
figure 1

Overview of the construction process for the DsLMF+ datasets.

Full size image

Image data collection

The original underground monitoring videos of the fully mechanized coal mining face were offered by several coal mines in Shaanxi Province of China, which were then screened and classified according to the different target object. We signed a scene authorization agreement with Shaanxi Coal and Chemical Industry Group Sunjiacha Longhua Mining Co.,LTD, so as to ensure that the dataset could be disclosed normally. Meanwhile, the agreement also included the authorization for the disclosure of the portraits of the mine personnel, so as to ensure that the miners who are photographed in the coal mine scene were aware of the disclosure of the dataset. The image acquisition equipment is composed of IVG-G5A network HD camera and Openmv IMX335(1/2.8”) lens. The lens focal length is 2.02 mm, and angle of field of view is 119.8°(D), 105.2°(H) and 87.2°(V). The camera can complete the image acquisition with a maximum resolution of 5 megapixels, the frame rate is 1~30FPS, and the used video formats are Flash video (FLV) and MPEG-4. The FFmpeg video processing software is used to process the needed classified videos31 and clip relevant images according to the different frame rate settings. The DsLMF+ datasets built in this work consists of 6 categories, which are respectively coal miners, large coal, towline, mine safety helmet, hydraulic support guard plate and miners’ behaviors. Considering that there is no target object to be annotated in some original images data, that is, the images do not include the mine personal, large coal, towline, hydraulic support guard plate and other target categories that need to be annotated. Therefore, some image frames have been removed and the other images are sorted according to the different categories, and the obtained images are used as the original image source of the DsLMF+ dataset.

Image data filtering

The original image source of the DsLMF+ dataset will then be screened. The DsLMF+ dataset collected in this work mainly includes the mine personnel, large coal and hydraulic support guard plate, towline, mine safety helmet and miners’ behaviors, on account of that some images in original datasets might be with no target, incomplete target, and poor image quality that makes it difficult to identify the target, hence those images where might exist some abnormal data should be all removed.

The abnormal images that need to be processed mainly include the following situations: 1) When the fully mechanized mining face is affected by severe environmental factors such as high dust and water mist, it is difficult to identify the coal miners, large coal and hydraulic support guard plate, towline, mine safety helmet and miners’ behaviors in the collected images. 2) Due to the limited field of view of a camera or the occlusion, the target acquisition is incomplete in the process of image acquisition, resulting in only local features of the target are included in the acquired images. 3) When the fully mechanized mining face has stopped working, the camera still continues to collect images, resulting in a large number of repeated images in the collected video images. 4) The target objects in the downhole video acquisition are in a moving state. In the process of converting these videos into pictures, a reasonable frame rate should be adopted according to the different moving speed. However, if the target moves too fast, the picture obtained by video conversion will inevitably be blurred. 5) Due to the influence of the downhole environment and the distance between the target from the camera, the target object at a far distance is difficult to distinguish from other equipment.

All the above abnormal video images need to be manually or automatically eliminated in the process of image dataset production. In order to make it reproducibility of the datasets, we used ResNet50 to build a tri-classification automatic filtering network model to deal with the low-quality images that affected by downhole environmental factors such as high dust, water mist, motion blur, etc. In this work, some high dust and water mist images, defocused and motion blurred image as well as clear image were selected from the collected raw images data, and constructed an image filtering dataset for the training and verification of the tri-classification automatic filter model. The obtained automatic filter model can be used to deal with the invalid images data automatically to increase the reproducibility of our datasets and enhance the chances for other researchers to collaborate with the datasets. The tri-classification automatic filter model has been provided along with the datasets, and its specific usage can be on reference on in its attached README file. In addition, the structural similarity index SSIM can be used to judge and automatically filter out the duplicate or similarity images. For the other cases, considering that it is easy to be affected by personal subjective factors in the process of screening images, the multiple people uniformly reviewed the controversial images in the dataset when removing images from the dataset, especially for those images that are difficult to distinguish.

Data labeling

Finally, the filtered original image datasets were annotated using LabelImg software and named the label, and here we provide an official open source download link (https://github.com/heartexlabs/labelImg) for the Labelimg software. The researchers can set the label in YOLO, VOC or CreateML format and annotate the images according to the instructions provided by the official. In the process of labeling diverse kinds of datasets, the label order needed be determined accordingly. Once the label order is determined, the label order cannot be changed when open the software to label next time. If the order is changed, the label order of the dataset will be automatically changed to the current label order, and the original labeled annotations will appear in the current order, resulting in label confusion in the dataset. The LabelImg tool was used to annotate the training set and validation set in accordance with YOLO format, in the meanwhile, we also converted the YOLO datasets into COCO datasets through script files and retain. This work includes the datasets of the mine personnel, towline, mine safety helmet and large coal with the single-label annotation, as well as the hydraulic support guard plate and miners’ behaviors with multi-label annotations. Figure 2 shows the label annotations of coal miners, large coal, towline, mine safety helmet, miners’ behavior and supporting state of the hydraulic support guard plate.

Fig. 2
figure 2

Label annotation for the dataset of fully mechanized longwall mining face: (a) Coal miner; (b) large coal; (c) mine safety helmet; (d) towline; (e) hydraulic support guard plate; (f) miners’ behaviors.

Full size image

The single-label datasets of the large coal, mine safety helmet, towline and mine personnel are named as large_coal, mine_safety_helmet, towline and coal miner, respectively. In order to judge whether there is movement interference between the shearer’s operation and the guard plate, the images are labeled according to the unfolding angle of the hydraulic support guard plate in this work, so as to obtain the support state information of the hydraulic support of the fully mechanized mining face. In the process of labeling the guard plate, the label types cover all angles of the hydraulic support guard plate. In order to ensure the accuracy of angle labeling, this work uses the built-in sensor in the hydraulic support of the fully mechanized mining face to detect and extract the angle information of the guard plate in real time. The extracted angle information is not only used to annotate the image of the guard plate in the dataset, but also to verify whether the annotated angle types of the guard plate are reasonable. Among which, In accordance with the different angle of unfolding of the hydraulic support guard plate, the supporting states of the hydraulic support guard plate are divided into eight kinds of type, which were respectively named as hydraulic_support_guard_plate_00, hydraulic_support_guard_plate_00_30, hydraulic_support_guard_plate_30_60, hydraulic_ support_guard_plate_60_90, hydraulic_support_guard_plate_90,hydraulic_support_guard_ plate_90_abnormal, hydraulic _support_guard_plate_90_120 and hydraulic_support_guard_ plate_abnormal. In order to judge whether there will be motion interference between the guard plate and the shearer, the label annotation for the image in which the shearer passing under the hydraulic support guard plate is also marked as Shearer. The involved dataset labels of the hydraulic support guard plate states are shown in Fig. 3.

Fig. 3
figure 3

The dataset label annotations for the hydraulic support guard plate states. (a) Shearer; (b) hydraulic_support_guard_plate_00; (c) hydraulic_support_guard_plate_00_30; (d) hydraulic_support_guard_plate_30_60; (e) hydraulic_support_guard_plate_60_90; (f) hydraulic_support_guard_plate_90; (g) hydraulic_support_guard_plate_abnormal; (h) hydraulic_support_guard_plate_90_abnormal; (i) hydraulic_support_guard_plate_90_120.

Full size image

Among them, hydraulic_support_guard_plate_00 state is the state when the guard plate is fully recovered and there is no interference with shearer operation. The numbers before and after the underline in hydraulic_support_guard_plate_00_30, hydraulic_support_guard_ plate_30_60 and hydraulic_support_guard_plate_60_90 respectively represent the angle range corresponding to the unfolding state of the guard plate. When the guard plate is in these three states, it will interfere with the shearer in operation. In hydraulic_support _guard_plate_90 state, when the unfolding angle corresponding to the state of the guard plate is 90°, the supporting plate is close to the coal wall, which can play a well supporting role on the coal wall and effectively prevent the occurrence of coal wall slab accident in the fully mechanized mining face. In hydraulic_support_guard_plate_abnormal state, there is a problem in the structure of the hydraulic support guard plate, which should be replaced in time. In hydraulic_support_guard_plate_90_abnormal state, the unfolding angle of the guard plate is 90°, and there is a small gap between the guard plate and the coal wall, so the support strength is not enough. In hydraulic_support_guard_plate_90_120 state, the unfolding angle of the guard plate is too large, which resulting in the gap between the guard plate and the coal wall is too large, and the support strength is not enough.

In order to ensure the universality and compatibility of this dataset, we collected the images of mine personnel, large coal, towline, mine safety helmet, miners’ behaviors and hydraulic support guard plate from multiple scenes, respectively. The image data of mine personnel came from 58 different scenes, the image data of large coal came from 18 different scenes, the image data of guard plate came from 159 different scenarios, the image data of towline images came from 65 different scenarios, the image data of mine safety helmet came from 85 different scenarios, and the image data of the miners’ behaviors came from 67 different scenarios. The DsLMF+ datasets built in this work are divided into training set and validation set at the ratio32 of 8:2. There are 30704 mine personnel images with 24563 images in training sets and 6141 in validation set, 21017 large coal images with 16813 images in training sets and 4204 in validation set, 21412 towline images with 17129 in training sets and 4283 in validation set, 20117 mine safety helmet images with 16093 in training set and 4024 in validation set, 24709 miners’ behavior images with 19767 in training sets and 4942 in validation set, and 20045 hydraulic support guard plates images with 16036 in training sets and 4009 in validation set. Tables 1–7 respectively describes the datasets of mine safety helmet, towline, coal miners, miners’ behavior, large coal and guard plate in multiple different scenarios.

Table 1 The summary of the training set and validation set for mine safety helmet.
Full size table
Table 2 The summary of the training set and validation set for towline.
Full size table
Table 3 The summary of the training set and validation set for coal miner.
Full size table
Table 4 The summary for the datasets of miners’ behavior.
Full size table
Table 5 The summary of the training set and validation set for large coal.
Full size table
Table 6 The summary for the datasets of hydraulic support guard plate that from scenario 1 to scenario 108.
Full size table
Table 7 The summary for the datasets of hydraulic support guard plate that from scenario 109 to scenario 159.
Full size table

Data Records

The DsLMF+ dataset of the coal mine image in the fully mechanized longwall mining face has been publicly available at the figshare data repository33. Data annotations include YOLO format and COCO format. Among them, the image and label files of the dataset in YOLO format are stored as follows: the folder names of each dataset in data2023_yolo are respectively coal_miner_data2023_yolo, large_coal_data2023_yolo, mine_safety_helmet_data2023_yolo, towline_data2023_yolo,miner_behavior_data2023_yolo and hydraulic_support_guard_plate _data2023_yolo. Each folder contains the picture folders and label folders that named as images and labels, in which respectively stores image data and label data. These folders also contain training set folders and verification set folders. The information contained in the label data mainly includes data type, number of labels and label coordinates.

The image and label files of the dataset in COCO format are stored as follows: the folder names of each dataset in data2023_coco are respectively coal_miner_data2023_coco, large_coal_data2023_coco, mine_safety_helmet_data2023_coco, towline_data2023_coco, miner_behavior_data2023_coco and hydraulic_support_guard_plate_data2023_coco. Each of these folders contains the training set image folder, verification set image folder and label folder respectively named as train2017, val2017 and annotations, which are used to store training set pictures, verification set pictures and label files. The information contained in COCO label files contains file name, image width and height, label category and label coordinates, etc.

In addition, the files coal_miner_DsLMF, large_coal_DsLMF, mine_safety_helmet_DsLMF, towine_DsLMF, miner_behavior_DsLMF and hydraulic_support_guard_plate_DsLMF are provided to be used to better distinguish the images of mine personnel, large coal, towline, miners’ behavior, mine safety helmet and guard plate in different scenarios in DsLMF+ datasets, and the image index corresponding to the different scenes are given in the files.

Technical Validation

To ensure the reliability of the DsLMF+ dataset in this work, we also conducted a comprehensive manual review of all images and their corresponding label annotation. The specific review method is as follows: five members with rich working experience in the coal mining field are selected to check the image dataset and label information one by one to see whether there are missing or wrong labels. At the same time, in order to ensure the quality and application effect of the dataset, the five members uniformly reviewed the controversial images in the dataset, such as the size threshold of large coal, the angle involved in the guard plate image and its label, the label veracity of the downhole towline, coal personnel behaviour and the mine safety helmet. Through the collective voting of the five members, the review work of the dataset was completed.

DsLMF+ dataset have provided two types of datasets formats of YOLO and COCO, which make it convenient to be applied for the currently popular top-ranked target detection neural networks. In order to verify the feasibility of the constructed dataset, this work selected YOLOv734, DETA35 and ViT-Adapter-L36 three top deep learning network from the COCO target detection ranking list, and conducted model training and verification on the DsLMF+ dataset. The access links of DETA, ViT-Adapter-L and YOLOv7 that used to verify the datasets are respectively https://github.com/jozhang97/deta, https://github.com/czczup/vit-adapter and https://github.com/wongkinyiu/yolov7. The DsLMF+ datasets were trained on a machine with Intel(R) Xeon(R) Gold 6330 CPU, RTX A5000 GPU and Ubantu18.04. The hyper-parameters of the above three target detection algorithms were on the reference to the recommended default values. To suit the dataset, some hyper-parameter values such as width, height, batch size, initial learning rate and Epochs are modified. This change was implemented in accordance to the recommendations from the initial YOLOv7, DETA, and ViT-Adapter-L research.

For the dataset verification, the coal miners, large coal, towline, mine safety helmet, hydraulic support guard plate and miners’ behaviours in the datasets are trained and evaluated. The image height and width of the input image are both resized to 640 in the network training. Table 8 presents the benchmark result of ViT-Adapter-L, DETA and YOLOv7 on the DsLMF+ datasets. Figure 4 shows the graphs of the three model’s performance during validation, the mAP value curves of each target detection network model. The mAP values of YOLOv7 detection model can respectively reach 0.986, 0.976, 0.978, 0.868, 0.913 and 0.997, the mAP values of DETA detection model can respectively reach 0.976, 0.960, 0.958, 0.815, 0.914 and 0.989, and the mAP values of ViT-Adapter-L detection model can respectively reach 0.966, 0.961, 0.963, 0.854, 0.928 and 0.989. The above mAP values indicate that the models have good performance, and the DsLMF+ dataset performs well under YOLOv7, DETA and ViT-Adapter-L. The deployed YOLOv7, DETA and ViT-Adapter-L have been respectively used to randomly extract and detect the 6 categories of images of coal miners, large coal, towline, mine safety helmet, hydraulic support guard plate and miners’ behaviours in the DsLMF+ dataset, and the identified target detection results are shown in Fig. 5, the detection effect and accuracy demonstrated the reliability and practicability of DsLMF+ datasets.

Table 8 Benchmark of ViT-Adapter-L, DETA and YOLOv7 performed on coal miners, large coal, towline, mine safety helmet, hydraulic support guard plate and miners’ behaviours in the datasets DsLMF+.
Full size table
Fig. 4
figure 4

The validation mAp value curve of the models ViT-Adapter-L, DETA and YOLOv7 on the coal miners, large coal, towline, mine safety helmet, hydraulic support guard plate and miners’ behaviours in the.

Full size image
Fig. 5
figure 5

The validation of the DsLMF+ datasets by using of the models ViT-Adapter-L, DETA and YOLOv7. (a) coal miners; (b) large coal;(c) towline; (d) mine safety helmet; (e) miners’ behaviours; (f) hydraulic support guard plate.

Full size image

Moreover, we will further expand the DsLMF+ dataset to make the dataset have better applicability and universality in the fully mechanized coal mining face. We also encourage other researchers in coal mine field to expand and improve the DsLMF+ dataset. The coal mine image dataset produced in this work is of great significance for the application of deep learning object detection algorithm for the intelligent identification and classification of abnormal conditions for underground mining, which aims to support further research and advancement of intelligence in the fully mechanized longwall mining face .

Table 9 The overview of the site-packages for ViT-Adapter-L, DETA and YOLOv7.
Full size table