Introduction

As a key equipment for deep sea and onshore oil drilling, drilling rigs are the key link affecting the oil supply chain, especially in the current world where oil and nature gas still account for a large proportion. The mining efficiency of drilling rigs and the development level of automation technology have become important factors in the supply and demand of oil and gas. Developing intelligent and efficient drilling systems is the only way to improve drilling technology.

The early pipe treatment system was mainly used in offshore platforms, and its market was mainly monopolized by several petroleum equipment companies in Europe and the USA. The German Bentee company applied a vertical drill pipe rig to the Norwegian-designed drilling platform. The system automatically completed the movement of the horizontal and vertical drill pipes, and the movement process was remotely controlled by the drilling control room. The pipe handling system of the Light Deep Well Marine Drill produced by Strachman and Henshaw in the UK can transport a variety of pipes such as drill pipes, casings, tubing and non-magnetic drill collars. The drill pipe was discharged in the correct drilling order by computer and control system (Murray et al. 2001). In 2008, Widford, Switzerland, developed an automatic derrick system for land rigs that can discharge drill pipes with an outer diameter of 88.9–203.2 mm and a lifting capacity of 68 kN. The STV automatic piping system developed by NOV Company of the USA was also suitable for land drilling, but it cannot increase the load of the drill pipe, only the upper end of the auxiliary moving drill pipe, which supports the outer diameter of the drill pipe from 73.0 to 203.2 mm. China’s Baoji Petroleum Machinery Co. Ltd. cooperated with Chuanqing Drilling Engineering Co. Ltd. to develop the TJG9 3//4-10 automatic derrick in 2011. The device realized the automatic emission of the two-story drill string, which is flexible and reliable. It can meet the drilling pipe emission requirements of various domestic land drilling.

The land rig pipe handling system, also known as the automatic tube manipulator, is a drilling automation device installed on a modified two-story platform. The drill pipe emission system was mainly composed of a two-story manipulator and a drill rig manipulator to work together to place the drill pipe and the drill collar into the two-story finger beam and the drill stand. It can realize the function of the second-floor discharge of the drill pipe and assisting the tripping. Drill pipe emission is an important part of the drilling process, which is characterized by long time, high strength, high risk and repeated operation. However, the level of automation in the drilling industry has been slow to develop, especially in land drilling operations, where the driller manually handles the manipulator to complete the discharge operation. The derrick workers worked on the second floor of the 20-m-high platform, which is labor intensive and the safety is not guaranteed. The IADC reported that based on 105 drilling contractors from the around the world in 2005, and as published in the July/August Drilling Contractor Magazine, the out of a total of 23 fatalities, 7 were Derrickman (the highest number by position), 5 were floor man, and 4 fatalities occurred during Tripping (Richard and Stroshein 2007). At present, the laws and regulations of various countries and regions in the world have continuously improved the HSE requirements for drilling workers. Therefore, in order to protect the life safety of drilling workers and improve drilling efficiency, it is practical to develop an intelligent and efficient automatic drill pipe emission control system. Because the derrick of the land rig has a special structure and cannot be directly compatible with the mature rig system of the ocean, we have a long way to go to develop a simple and reliable rig system (Saeed et al. 2012). Many scholars have done a lot of research on how to achieve drilling automation and how to control drilling systems (Godhavn 2009; Breyholtz and Nikolaou 2012; Macpherson et al. 2013).

According to the actual investigation of China’s domestic drilling vertical discharge system, the drill pipe is skewed in the drill string box due to mechanical or other interference factors. At this time, when the manipulator takes the drill pipe from the drill string box according to the planned path, the drill pipe cannot be accurately grasped, and the tripping operations cannot be completed. The current solution is to manually correct the clamping angle of the manipulator until the drill pipe can be accurately grasped. During the corrective operation, the worker needs to spend time to complete the alignment operation of the manipulator, which not only reduces the drilling efficiency, but also threatens the life safety of the drill floor worker. In order to solve this problem, this paper introduces the machine vision control system based on the traditional manipulator work of the land rig and develops and designs the automatic drill pipe emission system based on machine vision. In this paper, the centerline of the drill pipe is obtained in real time through visual sensors and linear regression algorithm. After calculating the deviation between the drill pipe and the mechanical arm, it is transmitted to the arm controller. The controller adjusts the posture of the mechanical arm in real time to accurately grasp the drill pipe.

System design

With the rapid development of science and technology, industrial robots are widely used in various fields, such as welding (Liu and Zhang 2017; Lopes et al. 2017), shipbuilding (Kermorgant 2018) and automotive (Michalos et al. 2018). In order to further improve the adaptability and application range of robots to the external environment, more and more people are beginning to study the visual servo of robots (Radcliffe et al. 2018; Chethan et al. 2018; Mendikute et al. 2018; Dong and Zhu 2019). Visual servo is the visual information of the target image collected by the visual sensor. After a series of algorithms, the control quantity is transmitted to the controller for closed-loop control. Visual servo control can improve production efficiency, improve product quality and rationally allocate labor resources.

In this paper, for the problem that the manipulator cannot grasp the drill pipe in the land drilling rig operation, a set of automatic drill pipe emission control system based on machine vision is developed. The system is mainly composed of a traveling mechanism, a rotating mechanism, a mechanical arm, a claw and a control system based on machine vision. The innovation of the system is to add a visual control device based on the traditional tube manipulator, which constitutes a hand–eye control system, allowing the tube manipulator to have a “eye.” In this system, the camera is fixed to the end of the mechanical arm and moves with the end of the mechanical arm during the movement. The camera can follow the end of the mechanical arm to measure near the target, improve the image accuracy and get the image information of the target for precise control. The servo control system first collects the drill string image information by using the camera, then processes the image based on the image information, then calculates the relative posture between the camera and the drill string target and transmits the deviation information to the controller for posture correction, to achieve accurate positioning and real-time tracking of the drill string target and then complete the drill pipe emission task intelligently and efficiently.

The hardware of the machine vision-based automatic drill pipe emission control system mainly includes a traveling mechanism, a rotating mechanism, a mechanical arm and a claw.

Figure 1 shows a schematic view of the overall structure of a drill pipe manipulator. Among them, the traveling mechanism includes a rail system and a travel carriage. The rail system is a combination of standard rails and standard pressure guiding blocks. The bottom of the guide rail adopts an integral steel plate structure to ensure the parallelism and fixed strength of the two guide rails. Mechanical limit switches are arranged at both ends of the guide rail to ensure that the travel carriage moves to a predetermined position and stops in time. Figure 2 shows a schematic structure view of travel carriage. The travel carriage includes the following parts: 1—fender; 2—drive wheel; 3—drive wheel reduction motor; 4—travel carriage frame; 5—four-point contact ball wheel bearing (inner gears); 6—plug electric cylinder; 7—driven wheel; 8—pin mechanism; 9—rotating mechanism: drive reduction servo motor and drive gear; 10—anti-roll plate.

Fig. 1
figure 1

Overall structure of a drill pipe manipulator

Full size image
Fig. 2
figure 2

Travel carriage

Full size image

The driving wheel and the driven wheel function to support the vehicle body and the mechanical arm. The role of the drive wheel reduction motor is to drive the carriage to walk. The function of the anti-rolling plate is to prevent the problem of rollover due to excessive overturning moment when the manipulator is working. When the travel carriage moves to the working position, the pin mechanism can ensure that the carriage is stationary when the mechanical arm is working.

The middle portion of Fig. 2 is a schematic structure view of a rotating mechanism. The rotating mechanism mainly includes four-point contact ball wheel bearing and servo motor reducer. The bearing adopts an internal tooth structure, and its function is to connect the travel carriage and the mechanical arm. It is capable of carrying rotational and overturning moments. The function of the servo motor reducer is to drive the pinion, then drive the internal tooth of the bearing and finally drive the mechanical arm to rotate. The servo motor has its own encoder for high-precision positioning. Precise control of the rotation speed can effectively reduce the vibration during the operation of the mechanical arm. During the operation of the drill pipe manipulator, the gear meshing gap and the arm structure clearance may cause positioning errors. In this paper, the influence of the gear meshing clearance is eliminated by setting the zero point of the mechanical arm circumferential position. At the same time, the influence of the mechanical arm clearance is eliminated by the laser profiler of the claw mechanism.

Figure 3 shows a schematic structure view of a mechanical arm. The mechanical arm adopts a four-link folding structure. This structure has the advantages of small footprint, large working range, short driving stroke and large gripping force. The drive system of the mechanical arm is controlled by a servo electric cylinder. This closed-loop servo control has high precision, low noise, energy saving, cleanness, impact resistance, long life and simple maintenance. Compared with the hydraulic system and the pneumatic system, the servo electric cylinder only needs to periodically inject grease in a complicated working environment, and there is no need to replace the wearing parts. Therefore, a lot of after-sales service costs are reduced.

Fig. 3
figure 3

Structure of mechanical arm

Full size image

Figure 4 shows a schematic structure view of a claw. The claw mainly includes mechanical limit switch, camera and servo electric cylinder. The function of the mechanical limit switch is to close the gripper after detecting that the drill pipe has reached the gripping position. The camera is equivalent to the eye of the mechanical arm. Its function is to collect the drill pipe image in real time. When the hand is extended to the theoretical position of the drill pipe according to the set path, the exact position of the drill pipe is found by the camera and the vision control system. Use the controller to adjust the attitude of the mechanical arm to complete the final grab task.

Fig. 4
figure 4

Structure of claw

Full size image

Algorithms

According to the actual investigation of the operation process of the land drilling rig, it is found that in the grasping operation of the drill pipe, the claw mechanism is required to align the centerline of the drill pipe. This paper has experiment with various methods and found that the linear regression method can solve this problem well.

In statistics, linear regression is a linear approach to modeling the relationship between a scalar response (or dependent variable) and one or more explanatory variables (or independent variable). The case of one explanatory variable is called simple linear regression. For more than one explanatory variable, the process is called multiple linear regression (Freedman 2005). This term is distinct from multivariate linear regression, where multiple correlated dependent variable is predicted, rather than a single scalar variable (Rencher and Christensen 2012). In the statistical analysis of data, the correlation study between variable is very important. Figure 5 shows a scatter plot in a Cartesian coordinate system. The statistics approximate a straight line.

Fig. 5
figure 5

Scatter plot

Full size image

Figure 6 shows the relationship between two variables using three straight lines in a scatter plot. Finding one from multiple linear equations best reflects the relationship between two variables. In general, linear regression can solve equations by least squares, which is a mathematical optimization technique. It finds the best function match of the data by the sum of the squares of the smallest errors. The least-squares method can be used to easily obtain unknown data while minimizing the sum of the squares of the errors between the data and the actual data.

Fig. 6
figure 6

Scatter plot (2)

Full size image

The mathematical formula for obtaining the best fit line by the least-squares method is derived as follows:

Assuming the best fit line is Eq. (1), for any sample point \((x_{i} ,y_{i} )\)\(i = 1,2,3, \ldots ,n\), when \(x\) takes the value \(x_{i} \;(i = 1,2,3, \ldots ,n)\), the error \(e\) (see Eq. 2) characterizes the degree of deviation of the actual value \(y_{i}\) from the ordinate of the corresponding point on the regression line.

$$y = ax + b$$
(1)
$$e = y_{i} - (ax_{i} + b)$$
(2)

In order for the best fit line to be close to the sample point, the total dispersion of the \(n\) dispersion needs to be minimal. The total dispersion cannot be expressed by the sum of \(n\) dispersions. Since the dispersion is positive or negative, the direct addition will cancel. The general approach is to use the sum of the squares of the dispersion as the total dispersion \(S\) (see Eqs. 3a, 3b).

$$S = \sum\limits_{i = 1}^{n} {e_{i}^{2} }$$
(3a)
$$S = \sum\limits_{i = 1}^{n} {(y_{i} - ax_{i} - b)^{2} }$$
(3b)

The total dispersion \(S\) is a first-order partial guide for \(a\) and \(b\), respectively. The result is shown in Eqs. (4a, 4b).

$$\frac{\partial S}{\partial b} = - 2\left( {\sum\limits_{i = 1}^{n} {y_{i} } - nb - a\sum\limits_{i = 1}^{n} {x_{i} } } \right)$$
(4a)
$$\frac{\partial S}{\partial b} = - 2\left( {\sum\limits_{i = 1}^{n} {x_{i} y_{i} } - b\sum\limits_{i = 1}^{n} {x_{i} } - a\sum\limits_{i = 1}^{n} {x_{i}^{2} } } \right)$$
(4b)

According to mathematical Eqs. (5a, 5b), let (Eqs. 4a, 4b) equal 0 and solve for the equation expressions for \(a\) and \(b\) (see Eqs. 6a, 6b).

$$n\bar{x} = \sum\limits_{i = 1}^{n} {x_{i} }$$
(5a)
$$n\bar{y} = \sum\limits_{i = 1}^{n} {y{}_{i}}$$
(5b)
$$a = \frac{{\sum\nolimits_{i = 1}^{n} {(x_{i} - \bar{x})} (y_{i} - \bar{y})}}{{\sum\nolimits_{i = 1}^{n} {(x_{i} - \bar{x})^{2} } }}$$
(6a)
$$b = \bar{y} - a\bar{x}$$
(6b)

The result can also be expressed as follows (see Eqs. 7a, 7b).

$$a = \frac{{n\sum {x_{i} y_{i} - \sum {x_{i} \sum {y_{i} } } } }}{{n\sum {x_{i}^{2} - (\sum {x_{i} } )^{2} } }}$$
(7a)
$$b = \frac{{\sum {x_{i}^{2} \sum {y_{i} } - \sum {x_{i} \sum {x_{i} y_{i} } } } }}{{n\sum {x_{i}^{2} - (\sum {x_{i} } )^{2} } }}$$
(7b)

In this paper, the linear regression of all threshold pixels in the image is calculated. According to the position coordinate data of the edge of the drill pipe obtained after the threshold is filtered, the centerline of the drill pipe is fitted by the least-squares method. The coordinate system is established to obtain the deviation between the drill pipe and the mechanical arm. The controller adjusts the grab angle of the mechanical arm in real time according to the deviation and completes the drill pipe emission task intelligently and efficiently.

Experiment

The mechanical arm controller and vision work together to produce visual servo control. The visual servo system generates control signals by acquiring and processing the drill pipe image and drives the mechanical arm to complete the servo task. Figure 7 shows a flowchart of a visual servo control system. The visual servo control system is mainly divided into four parts.

Fig. 7
figure 7

Flowchart of visual servo control system

Full size image
  1. 1.

    ROI The camera captures the image stream of the drill pipe and performs preprocessing (threshold and binarization) to obtain the outline of the drill pipe. Here is a brief description of setting the threshold. In the experiment, we used a 20-cm-long iron tube instead of the drill pipe to perform the mechanical arm correction experiment. When the value of the adjustment threshold can clearly distinguish the iron pipe and the background feature, the parameter at this time is the optimal threshold. This parameter remains the same unless the drill pipe test body is replaced again.

  2. 2.

    Linear regression Linear regression is computed for all threshold pixels of the image, and the centerline of the drill pipe is fitted using least squares.

  3. 3.

    Coordinate system A coordinate system is established for the image, and the deviation between the drill pipe and the mechanical arm is obtained according to the mathematical logic relationship.

  4. 4.

    Controller The controller adjusts the posture of the arm according to the deviation to realize the positioning and tracking of the drill pipe.

The difficulty of the visual servo control system is to obtain the expression of the centerline of the drill pipe and to solve the deviation between the drill pipe and the mechanical arm. In order to verify the accuracy of the core algorithm to obtain the centerline of the drill pipe, we built a miniature visual mechanical arm in the laboratory to simulate the gripper mechanism to grab the drill pipe. Figure 8 shows a schematic illustration of a miniature vision mechanical arm and an iron tubular tube. In the experiment, a 20-cm-long round iron tube was used instead of the drill pipe. The miniature vision mechanical arm mainly includes STM32F765VIT6 master chip, Tower Pro MG90s steering gear, distortion-free lens, lithium battery, DuPoint line and other components.

Fig. 8
figure 8

Experimental device schematic

Full size image

The master chip is used to process and extract the centerline of the iron tube and transmit the deviation information to the steering gear through the DuPoint line. The steering gear is composed of a steering wheel, a reduction gear set, a position feedback potentiometer, a DC motor and a control circuit. In this experiment, the steering gear is equivalent to the rotating mechanism of the drill pipe arm. It has the advantage of compact structure, easy installation and debugging, simple control, large torque and low cost. It is a position servo drive for control systems that require constant angle changes and can be maintained. In this paper, the PID algorithm is used to control the steering of the steering gear in real time to ensure that the lens always follows the direction of the iron tube swinging. The role of the distortion-free lens is to obtain an image of the iron tube. The lithium battery allows the micro-mechanical arm to operate offline. The role of the DuPoint line is to connect the steering gear to the master chip and transmit information.

Experiment process:

  1. 1.

    Find an optimal threshold based on the characteristics of the iron tube to enable the lens to capture the clearest image of the iron tube. The threshold is set to remain unchanged unless the drill pipe replacement is replaced again. The schematic diagram of the iron tube obtained by the distortion-free lens is shown in Fig. 9.

    Fig. 9
    figure 9

    Schematic diagram of the iron tube

    Full size image
  2. 2.

    The image is binarized to obtain the outline of the iron tube. Figure 10 shows a schematic view of the iron tube after binarization.

    Fig. 10
    figure 10

    A schematic view of the iron tube after binarization

    Full size image
  3. 3.

    After the first and second steps, a regression analysis is performed on the threshold pixel points that meet the condition, that is, a linear regression is calculated for all the pixels of the edge of the iron tube in Fig. 10. In this operation, the least-squares method fits the centerline based on the position data of the edge points of the iron tube obtained after the threshold filtering. When the linear regression analysis is completed, in addition to fitting the centerline of the iron tube in the image, the coordinate position information of the two vertices of the centerline is returned. The equation of the centerline can be obtained by two sets of coordinates. Figure 11 shows a schematic illustration of fitting the centerline of an iron circular tube.

    Fig. 11
    figure 11

    A schematic illustration of the centerline of an iron tube

    Full size image
  4. 4.

    Since the resolution of each image taken by the experimental undistorted lens is 160 pixels × 120 pixels (width × height), based on this, a coordinate system with x-axis right and y-axis downward established. The red point is the center point of the lens; the green point is the origin of the coordinate system; the yellow point and the purple point are the upper and lower endpoints of the centerline of the iron tube, respectively. The coordinate of the yellow point is \((x_{1} ,y_{1} )\), and the purple point coordinate is \((x_{2} ,y_{2} )\). The coordinate system and the resolution of the image give the expression of the \(y_{1}\) equation (see Eq. 8), and the equation \(y_{2}\) of the fitted line can be obtained from the coordinates of the two end points of the fitted line (see Eq. 9). These simultaneous equations \(y_{1}\) and \(y_{2}\) show the coordinate position of the gray intersection. Figure 12 shows a schematic illustration of a coordinate system.

    Fig. 12
    figure 12

    A schematic illustration of a coordinate system

    Full size image
$$y_{1} = - 60$$
(8)
$$\frac{{y - y_{1} }}{{y_{2} - y_{1} }} - \frac{{x - x_{1} }}{{x_{2} - x_{1} }} = 0\;\;\;x_{1} \ne x_{2}$$
(9)

In this paper, EX is defined as the difference between the abscissa of the center point of the lens and the abscissa of the gray intersection. The deviation EX is equal to the abscissa of the intersection minus the abscissa of the center point of the lens. When EX is positive, it indicates that the position of the drill pipe is shown in Fig. 12, and the iron tube is on the right side of the center point of the lens. In the process of the iron tube tracking experiment, the PID algorithm (Bingol et al. 2017) is used to control the steering gear to drive the lens movement to eliminate the deviation between the iron tube and the lens. The process of tracking the iron tube by the lens is like the process of automatically correcting the grab angle by the mechanical arm. By correcting the parameters of the PID, the response speed and response accuracy of the servo are gradually improved. After statistics, the time for the micro-mechanical arm to eliminate the deviation is about 0.23 s, and the average number of frames transmitted per second is 75. The result of the relevant information during the running of the program is shown in Fig. 13.

Fig. 13
figure 13

Program terminal result diagram

Full size image

A complete experimental video of the steering gear driving the lens following the iron tube swing has been uploaded to YouTube at https://www.youtube.com/watch?v=uhaqN4heDgA.

In addition to the linear regression method to extract the centerline of the iron tube, the quad detection, the line detection, and the line segment detection methods were used for comparison experiments. Figure 14 shows an effect of the quad detecting method. Figure 15 shows an effect of the line detecting method. Figure 16 shows an effect of the line segment detecting method. Figure 17 shows the effect of the linear regression method. In the experiment of the quad detection method, the efficiency and accuracy of each extraction rectangle are not very high, so the centerline line of the iron tube cannot be accurately obtained according to the extracted rectangle. Similarly, in the line detection and line segment detection method experiments, the purpose of both is to extract the edge line information of the iron tube to obtain the centerline of the iron tube. However, due to the unsatisfactory extraction effect of these two methods, the detection results often have deviations that directly affect the subsequent centerline extraction. In this paper, the coordinate data of the edge of the iron tube after the image threshold operation are used for regression analysis, and the centerline of the iron tube is fitted by the least-squares method. This method has the advantages of high stability, high speed and high accuracy. Table 1 compares the FPS data of the four methods during operation.

Fig. 14
figure 14

Quad detection

Full size image
Fig. 15
figure 15

Line detection

Full size image
Fig. 16
figure 16

Line segment detection

Full size image
Fig. 17
figure 17

Linear regression detection algorithm

Full size image
Table 1 FPS comparison
Full size table

Conclusions

This paper introduces the development and challenges of land and offshore oil rigs at home and abroad. Aiming at the problem that the land rig mechanical arm needs manual assistance to complete the drill pipe emission task, a set of automatic drill pipe emission control system based on machine vision has been developed. The system can change the current status of existing land rig drill pipe emissions, save manpower, improve drilling efficiency and ensure the safety of drilling workers.

In this paper, the method of linear regression to obtain the centerline of the drill string is proved by experiments. In the experiment, the position coordinate data of the pixel points at the edge of the drill pipe are obtained by threshold screening. The least-squares method is used to fit the centerline of the drill pipe using these data. The coordinate system is established to determine the deviation between the drill pipe and the mechanical arm, and the steering gear drives the lens movement according to the deviation information to ensure real-time tracking of the drill pipe by the lens.

In this paper, four methods for extracting the centerline of the drill pipe are compared through experiments. From the experimental results, the accuracy of the quad detection method for extracting rectangles is poor, and the centerline of the drill pipe cannot be further extracted. The accuracy of the line detection method is also relatively poor, and often only the edge of one side of the iron tube is detected. The accuracy of line segment detection is good, but the efficiency is too low, and the FPS is less than 5. The method of linear regression can be superior in terms of efficiency and accuracy, so this paper chooses linear regression method to obtain the centerline of the drill pipe. In this way, the problem of the mechanical arm quickly and accuracy aligning the centerline of the drill pipe can be solved.

Further work

According to the results of the laboratory, the next step is to apply the scheme to the actual land rig site and re-determine the threshold value and PID parameters in the field to ensure stable and smooth operation of automatic drill pipe emission control system.