1 Introduction

In the last decades, the concepts of fractional differentiation and integration have attracted the attention of many scholars from many fields due to their wider applicability. In many articles it has been pointed out that fractional modeling of some systems in science and engineering provides more efficient solutions than classical modeling of that problem using integer-order derivatives. Nevertheless, they were used to construct ordinary and partial differential equations that could depict real-world problems. Three classes could be identified, including differential operators with the power-law kernel, differential operators with exponential decay kernel, and differential operators with the generalized Mittag-Leffler kernel. Each of these mentioned operators has got success in some particular practical problems as one differential and integral operator cannot replicate all the real-world problems accurately. In particular, the differential operators with the generalized Mittag-Leffler kernel have been introduced very recently, and they have earned respect across many fields of science due to their wider applicability. Very recently, Ghanbari and Atangana [1] have extended the Atangana–Baleanu fractional integral to the framework of image processing to remove existing noise in a given image. They obtained outstanding results as the new fractional mask was able to denoise images with great success. In this work, we aim to further extend the use of the Atangana–Baleanu fractional integral in edge detection. Edge detection is a fundamental tool in image processing algorithms applied to medical imaging. The general structure of this article will be as follows. In Sect. 2, we have a brief overview of some of the well-known algorithms for determining the edges of an image. In the third section of the article, we utilize four categories of approximations to calculate the value of the Atangana–Baleanu fractional integration. Then, in each case, two general structures are employed for making the fracture masks in edge detecting. These results are the main achievements of this paper since they are presented for the first time. In Sect. 4, the performance of the proposed masks is tested using several numerical simulations. For this purpose, seven images with different structures are used. Finally, the last part of this article also states the conclusions.

2 A short review of some of the well-known methods

Edge detection is an important branch of image processing. It includes techniques used to identify pixels from a digital image in which the brightness intensity varies drastically compared to the adjacent pixels [2]. Many efficient algorithms have been proposed in the literature for edge detection. Most of these algorithms are designed based on first-order, such as the Sobel, Prewitt, and Roberts, operators and second-order differential operators such as Laplace operator. In practical applications, this feature may lead to the defective performance of the corresponding masks. Noise can also pose a challenge to the performance of these algorithms to extract edges in images [3].

One of the most popular filters used to determine the edges of an image is the Prewitt operator. This approach is based on the approximation of the first-order derivative by the central difference. The results of the method are obtained by convolving the image with the following two kernels:

$$ h_{x}= \textstyle\begin{array} {|c|c|c|} \hline -1 & 0 & 1 \\ \hline -1 & 0 & 1 \\ \hline -1 & 0 & 1 \\ \hline \end{array}\displaystyle \ ,\quad\quad h_{y}= \textstyle\begin{array} {|c|c|c|} \hline -1 & -1 & -1 \\ \hline 0 & 0 & 0 \\ \hline 1 & 1 & 1 \\ \hline \end{array}\displaystyle \ . $$
(1)

Another important filter is the Sobel operator, which is based on central finite differences. Unlike the Prewitt operator, the main focus of the method is to provide more partnerships with pixels closer to the center of the mask. The convolution kernels used in this method are as follows:

$$ h_{x}= \textstyle\begin{array} {|c|c|c|} \hline -1 & 0 & 1 \\ \hline -2 & 0 & 2 \\ \hline -1 & 0 & 1 \\ \hline \end{array}\displaystyle \ ,\quad\quad h_{y}= \textstyle\begin{array} {|c|c|c|} \hline -1 & -2 & -1 \\ \hline 0 & 0 & 0 \\ \hline 1 & 2 & 1 \\ \hline \end{array}\displaystyle \ . $$
(2)

The Sobel operator can provide more accurate edge direction information, but it will also detect many false edges with a coarse edge width. While the Prewitt operator is more sensitive to horizontal and vertical edges, the Sobel operator is more sensitive to the diagonal edges than to the horizontal and vertical edges. All the above-mentioned kernels are based on integral differential operators of integer-order operators. Many dramatic changes in this area have been made by taking the fractional differential concepts into account. In recent years, the use of fractional differential operators to improve image quality, image texture enhancement, image noise reduction, and image edge analysis have yielded stunning results [4–12].

One of the most important formulas for expanding of fractional differential operators in image processing is to use the following general form:

$$\begin{aligned} { \mathcal{T}} ^{ \sigma } \mathcal{K}(t) &\approx \rho _{0} \mathcal{K}(t)+\rho _{1} \mathcal{K} (t-1 )+\rho _{2} \mathcal{K} (t-2 )+\rho _{3}\mathcal{K} (t-3 )+ \cdots , \end{aligned}$$
(3)

where \(\rho _{1}\), \(\rho _{2}\), and \(\rho _{3}\) are the consecutive coefficients in the expansion of (3). Considering this definition, the expansion can be generalized to the two-dimensional space of the images in each of the x or y directions as follows:

$$ \begin{gathered} {}^{x} { \mathcal{T}}_{\mathrm{GL}}^{ \sigma } \mathcal{K}(x,y) \approx \rho _{0} \mathcal{K}(x,y)+\rho _{1} \mathcal{K} (x-1,y )+\rho _{2} \mathcal{K} (x-2,y )+\rho _{3}\mathcal{K} (x-3,y )+ \cdots , \\ {}^{y} { \mathcal{T}}_{\mathrm{GL}}^{ \sigma } \mathcal{K}(x,y) \approx \rho _{0} \mathcal{K}(x,y)+\rho _{1} \mathcal{K} (x,y-1 )+\rho _{2} \mathcal{K} (x,y-2 )+ \rho _{3}\mathcal{K} (x,y-3 )+\cdots . \end{gathered} $$
(4)

In the remainder of the paper, the following two structures will be used to make new fractional-order masks.

Mask 1

As the first main structure in this article, we construct a \(3\times 3\) fractional integral mask as follows:

$$ h_{x}= \textstyle\begin{array} {|c|c|c|} \hline -\rho _{0} & 0 & \rho _{0} \\ \hline -\rho _{1} & 0 & \rho _{1} \\ \hline -\rho _{2} & 0 & \rho _{2} \\ \hline \end{array}\displaystyle \ ,\quad\quad h_{y}= \textstyle\begin{array} {|c|c|c|} \hline \rho _{0} & \rho _{1} & \rho _{2} \\ \hline 0 & 0 & 0 \\ \hline -\rho _{0} & -\rho _{1} & -\rho _{2} \\ \hline \end{array}\displaystyle \ . $$
(5)

Mask 2

The second structure used is the same kernel as introduced in [13] is given by

$$ h_{x}= \textstyle\begin{array} {|c|c|c|} \hline \rho _{0} & \rho _{1} & \rho _{2} \\ \hline -\rho _{3} & 0 & \rho _{3} \\ \hline -\rho _{2} & -\rho _{1} & -\rho _{0} \\ \hline \end{array}\displaystyle \ ,\quad\quad h_{y}= \textstyle\begin{array} {|c|c|c|} \hline -\rho _{2} & -\rho _{3} & \rho _{0} \\ \hline -\rho _{1} & 0 & \rho _{1} \\ \hline -\rho _{0} & \rho _{3} & \rho _{2} \\ \hline \end{array}\displaystyle \ . $$
(6)

The design of these two kernels is such that the vertical, horizontal, left, and right diagonal pixels around the central pixel are applied. This crucial feature makes these two kernels an excellent tool for extracting image details, including texture and edges. After making these kernels, the absolute values are often used to calculate the gradient moduli approximately as follows:

$$ \vert H \vert \approx \vert H_{x} \vert + \vert H_{y} \vert , $$
(7)

where

$$ H_{x}=I(x,y)*h_{x}, \quad\quad H_{y}=I(x,y)*h_{y}, $$
(8)

and \(I(x,y)\) is the pixel value of the gray pixel of the given image.

3 New edge detection masks based on ABC-fractional

So far, many definitions have been provided for critical concepts such as fractional derivatives and integrals. The use of these definitions seems to be increasingly used in modeling of applied phenomena [14–39].

One of the most commonly used definitions for a fractional-order derivative is the definition given by Atangana and Baleanu in the Caputo sense (ABC) [40]:

$$ {\mathcal{D}}_{t}^{\sigma } \mathcal{K}(t) = \frac{\mathcal{M}(\sigma )}{1-\sigma } \int _{0}^{t} E_{\sigma } \biggl[- \sigma \frac{(t-\tau )^{\sigma }}{1-\sigma } \biggr]\dot{\mathcal{K}}( \tau )\,d\tau . $$
(9)

A prominent feature of this definition is the use of a non-local and non-singular kernel in the derivative definition. Besides, the derivative preserves a memory property that keeps the key function information from the starting point to the desired time. Another key feature of this definition is the use of the Mittag-Leffler function of index σ, which is defined as follows [41, 42]:

$$\begin{aligned} E_{ \sigma }(t)=\sum_{k=0}^{\infty } \frac{t^{k}}{\varGamma (\sigma k+1)}, \quad \sigma >0. \end{aligned}$$
(10)

Another function used in this definition is \(\mathcal{M}(\cdot )\), which is used as a normalization function, defined by

$$ \mathcal{M}(\sigma )=1-\sigma +\frac{\sigma }{\varGamma (\sigma )}. $$

For a function \(\mathcal{K} (t )\), the corresponding definition for the fractional integral of Atangana–Baleanu of order σ is also defined as follows:

$$\begin{aligned} {\mathcal{I}}_{t}^{ \sigma } \mathcal{K} (t ) &= \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \int _{0}^{t} \mathcal{K}(\tau ) (t-\tau )^{\sigma -1}\,d\tau . \end{aligned}$$
(11)

So far, using various ideas, a variety of numerical methods have been introduced to approximate these fractional operators. In what follows, we will use several different approaches to designing and making new masks.

3.1 The masks based on the Grunwald–Letnikov (GL) approximation

The starting point of this approach is taking into account the definition of the fractional integral in the sense of Atangana–Baleanu as follows:

$$\begin{aligned} {\mathcal{I}}_{\mathrm{GL}}^{ \sigma } \mathcal{K} (t ) = \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \int _{0}^{t} \mathcal{K}(\tau ) (t-\tau )^{\sigma -1}\,d\tau . \end{aligned}$$
(12)

Now we approximate the corresponding integral Grunwald–Letnikov [43]:

$$ \begin{aligned}[b] {\mathcal{I}}_{\mathrm{GL}}^{ \sigma } \mathcal{K} (t ) &\approx \int _{0}^{t }\frac{\mathcal{K}(\tau )}{ (t-\tau )^{1-\sigma }}\,d\tau \\ &=\lim_{h\rightarrow 0} h^{-\sigma } \biggl(\mathcal{K} (t )+ \sigma \mathcal{K} (t-h )+ \frac{ ( -\sigma ) ( -\sigma +1)}{2}\mathcal{K} (t-h ) \\ &\quad{} + \cdots +\frac{\varGamma (-\sigma +1)}{k!\varGamma (-\sigma -N+1)} \mathcal{K} (t-Nh ) \biggr). \end{aligned} $$
(13)

This definition is perhaps one of the most common definitions in discrete fractional calculus, and it has also been used in image processing.

Setting \(h=1\) in (13) along with (12), we have

$$\begin{aligned} \begin{aligned}[b] {\mathcal{I}}_{\mathrm{GL}}^{ \sigma } \mathcal{K} (t )\approx{}& \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\mathcal{M}(\sigma )} \biggl(\mathcal{K} (t )+ \sigma \mathcal{K} (t-h )+\frac{ \sigma ( \sigma -1)}{2} \mathcal{K} (t-2h ) \\ &{}+ \frac{ \sigma ( 1-\sigma ^{2})}{6} \mathcal{K} (t-3h )+\cdots \biggr). \end{aligned} \end{aligned}$$
(14)

The corresponding fractional GL-integral in x and y directions is obtained respectively as follows:

$$\begin{aligned}& \begin{aligned}[b] {}^{x} { \mathcal{I}}_{\mathrm{GL}}^{ \sigma } \mathcal{K}(x,y) \approx {}&\frac{1}{\mathcal{M}(\sigma )} \mathcal{K} (x,y )+ \frac{\sigma ^{2}}{\mathcal{M}(\sigma )} \mathcal{K} (x-1,y )+\frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} \mathcal{K} (x-2,y ) \\ &{}+ \frac{\sigma ^{2}-\sigma ^{4}}{6\mathcal{M}(\sigma )}\mathcal{K} (x-3,y )+\cdots , \end{aligned} \end{aligned}$$
(15)
$$\begin{aligned}& \begin{aligned}[b] {}^{y} { \mathcal{I}}_{\mathrm{GL}}^{ \sigma } \mathcal{K}(x,y) \approx {}&\frac{1}{\mathcal{M}(\sigma )} \mathcal{K} (x,y )+ \frac{\sigma ^{2}}{\mathcal{M}(\sigma )} \mathcal{K} (x,y-1 )+\frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} \mathcal{K} (x,y-2 ) \\ &{}+ \frac{\sigma ^{2}-\sigma ^{4}}{6\mathcal{M}(\sigma )}\mathcal{K} (x,y-3 )+\cdots . \end{aligned} \end{aligned}$$
(16)

In this way the required coefficients for the method are calculated as follows:

$$ \begin{aligned} \rho _{0}&= \frac{1}{\mathcal{M}(\sigma )}, \\ \rho _{1}&=\frac{\sigma ^{2}}{\mathcal{M}(\sigma )}, \\ \rho _{2}&=\frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )}, \\ \rho _{3}&=\frac{\sigma ^{2}-\sigma ^{4}}{6\mathcal{M}(\sigma )}. \end{aligned} $$
(17)

Using the general structures presented in (5) and (6), then utilizing the coefficients of σs obtained in (17), two new masks can be used to determine the edges of an image outlined in what follows.

Mask GL1

Using the values obtained in Eq. (17) and using them in (5), the following \(3\times 3\) masks are introduced:

$$\begin{aligned}& h_{x}= \textstyle\begin{array} {|c|c|c|} \hline - \frac{1}{\mathcal{M}(\sigma )} & 0 & \frac{1}{\mathcal{M}(\sigma )} \\ \hline -\frac{\sigma ^{2}}{\mathcal{M}(\sigma )} & 0 & \frac{\sigma ^{2}}{\mathcal{M}(\sigma )} \\ \hline -\frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} & 0 & \frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} \\ \hline \end{array}\displaystyle \ ,\quad\quad h_{y}= \textstyle\begin{array} {|c|c|c|} \hline \frac{1}{\mathcal{M}(\sigma )} & \frac{\sigma ^{2}}{\mathcal{M}(\sigma )} & \frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} \\ \hline 0 & 0 & 0 \\ \hline -\frac{1}{\mathcal{M}(\sigma )} & -\frac{\sigma ^{2}}{\mathcal{M}(\sigma )} & -\frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} \\ \hline \end{array}\displaystyle \ . \end{aligned}$$

Mask GL2

Similarly, by inserting the values into the structure presented in (6), the following σ fractional order-dependent mask is constructed:

$$\begin{aligned}& h_{x}= \textstyle\begin{array} {|c|c|c|} \hline \frac{1}{\mathcal{M}(\sigma )} & \frac{\sigma ^{2}}{\mathcal{M}(\sigma )} & \frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} \\ \hline -\frac{\sigma ^{2}-\sigma ^{4}}{6\mathcal{M}(\sigma )} & 0 & \frac{\sigma ^{2}-\sigma ^{4}}{6\mathcal{M}(\sigma )} \\ \hline -\frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} & -\frac{\sigma ^{2}}{\mathcal{M}(\sigma )} & -\frac{1}{\mathcal{M}(\sigma )} \\ \hline \end{array}\displaystyle \ ,\quad\quad h_{y}= \textstyle\begin{array} {|c|c|c|} \hline -\frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} & - \frac{\sigma ^{2}-\sigma ^{4}}{6\mathcal{M}(\sigma )} & \frac{1}{\mathcal{M}(\sigma )} \\ \hline -\frac{\sigma ^{2}}{\mathcal{M}(\sigma )} & 0 & \frac{\sigma ^{2}}{\mathcal{M}(\sigma )} \\ \hline -\frac{1}{\mathcal{M}(\sigma )} & \frac{\sigma ^{2}-\sigma ^{4}}{6\mathcal{M}(\sigma )} & \frac{\sigma ^{3}-\sigma ^{2}}{2\mathcal{M}(\sigma )} \\ \hline \end{array}\displaystyle \ . \end{aligned}$$

3.2 The masks based on Toufik–Atangana (TA) approach

Another approximate method is based on the use of function interpolation of \(\mathcal{K} (\tau )\) on \([t_{k},t_{k+1}]\) as follows:

$$ \mathcal{K}(\tau ) =\frac{\mathcal{K}(t_{k}) }{h} (\tau -t_{k-1} )+ \frac{\mathcal{K}(t_{k-1}) }{h} (\tau -t_{k} ). $$
(18)

On the other hand, setting \(t=t_{n}\) in ABC-fractional integral (11), one gets

$$\begin{aligned} {\mathcal{I}}_{t}^{ \sigma } \mathcal{K} (t_{n} ) =& \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t_{n} )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \int _{0}^{t_{n+1}} \mathcal{K}(\tau ) (t-\tau )^{\sigma -1}\,d\tau \end{aligned}$$
(19)
$$\begin{aligned} =&\frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \sum _{k=0}^{n} \int _{t_{k}}^{t_{k+1}} \mathcal{K}(\tau ) (t-\tau )^{\sigma -1}\,d\tau . \end{aligned}$$
(20)

Using (18) in (20) and doing some necessary calculations, we get [44]

$$\begin{aligned} {\mathcal{I}}_{t}^{ \sigma } \mathcal{K} (t_{n} ) &= \frac{1 -\sigma }{\mathcal{M}(\sigma ) } \mathcal{K} (t_{n} ) + \frac{\sigma h^{\sigma }}{\mathcal{M}(\sigma )\varGamma ( \sigma +2)} \\ &\quad{}\times \sum_{m=0}^{n} \bigl( { \mathcal{K} (t_{m} )} \bigl[ (n - m+ 1 )^{\sigma } (n - m + 2 + \sigma ) - (n-m)^{\sigma }(n - m + 2 + 2 \sigma ) \bigr] \\ &\quad{} - \mathcal{K} (t_{m-1} ) \bigl[ (n- m + 1 )^{\sigma +1} - (n -m)^{\sigma }(n - m + 1 + \sigma ) \bigr] \bigr). \end{aligned}$$
(21)

Equation (21) can be rewritten as

$$\begin{aligned} {\mathcal{I}}_{\mathrm{TA}}^{ \sigma } \mathcal{K} (t_{n} ) &= \biggl[{ \frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma {h}^{\sigma } ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (t_{n} ) \\ &\quad{}+ \biggl[{ \frac{\sigma {h}^{\sigma } ( { { ( \sigma +3 ) {2}^{\sigma }-2 \sigma -4 }{ }} )}{B ( \sigma ) \varGamma ( \sigma +2 )} } \biggr] \mathcal{K} (t_{n-1} ) \\ &\quad{}+ \biggl[{ \frac{\sigma {h}^{\sigma } ( ( \sigma +4 ) {3}^{\sigma }- (2\sigma +5 ) {2}^{\sigma } +\sigma +2 ) }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (t_{n-2} ) \\ &\quad{}+ \biggl[{ \frac{\sigma {h}^{\sigma } ( ( \sigma +5 ) {4}^{\sigma }- ( 2\sigma +6 ) {3}^{\sigma }+ ( \sigma +3 ) {2}^{\sigma } ) }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (t_{n-3} ) + \cdots. \end{aligned}$$
(22)

For \({h}=1\) in (22), the following expansions are obtained:

$$\begin{aligned}& {{}^{x} \mathcal{I}}_{\mathrm{TA}}^{ \sigma } \mathcal{K} (x,y ) \\& \quad = \biggl[{ \frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (x,y ) + \biggl[{ { \frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma }-2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } \biggr] \mathcal{K} (x-1,y ) \\& \quad \quad{}+ \biggl[{ \frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (x-2,y ) \\& \quad \quad{} + \biggl[{ \frac{ ( \sigma ^{2}+5\sigma ) {4}^{\sigma }- ( 2\sigma ^{2}+6\sigma ) {3}^{\sigma }+ ( \sigma ^{2}+3\sigma ) {2}^{\sigma } }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (x-3,y ) +\cdots, \end{aligned}$$
(23)
$$\begin{aligned}& {{}^{y} \mathcal{I}}_{\mathrm{TA}}^{ \sigma } \mathcal{K} (x,y ) \\& \quad = \biggl[{ \frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (x,y ) + \biggl[{ { \frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma }-2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } \biggr] \mathcal{K} (x,y-1 ) \\& \quad \quad{} + \biggl[{ \frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (x,y-2 ) \\& \quad \quad{} + \biggl[{ \frac{ ( \sigma ^{2}+5\sigma ) {4}^{\sigma }- ( 2\sigma ^{2}+6\sigma ) {3}^{\sigma }+ ( \sigma ^{2}+3\sigma ) {2}^{\sigma } }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \biggr] \mathcal{K} (x,y-3 ) +\cdots. \end{aligned}$$
(24)

Therefore, the coefficients for this type of fractional approximation are calculated as follows:

$$ \begin{aligned} \rho _{0}&={ \frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} , \\ \rho _{1}&={ { \frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} }, \\ \rho _{2}&={ \frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }}, \\ \rho _{3}&={ \frac{ ( \sigma ^{2}+5\sigma ) {4}^{\sigma }- ( 2\sigma ^{2}+6\sigma ) {3}^{\sigma }+ ( \sigma ^{2}+3\sigma ) {2}^{\sigma } }{ B ( \sigma ) \varGamma ( \sigma +2 ) }}. \end{aligned} $$
(25)

Mask TA1

In this case, using the results of (25), the following masks of fractional order σ are formed:

$$\begin{aligned}& h_{x}= \textstyle\begin{array} {|c|c|c|} \hline -{ \frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} & 0 & {\frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline -{ {\frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } & 0 & { {\frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } \\ \hline -{\frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} & 0 & {\frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline \end{array}\displaystyle \ , \\& h_{y}= \textstyle\begin{array} {|c|c|c|} \hline { \frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} & { {\frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } & {\frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline 0 & 0 & 0 \\ \hline -{\frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} & -{ {\frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } & -{ \frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline \end{array}\displaystyle \ . \end{aligned}$$

Mask TA3

The parameters obtained in (25) can also be used to determine new fractional masks in determining the edges of the images as follows:

$$\begin{aligned}& h_{x}= \textstyle\begin{array} {|c|c|c|} \hline { \frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} & { {\frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } & {\frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline -{\frac{ ( \sigma ^{2}+5\sigma ) {4}^{\sigma }- ( 2\sigma ^{2}+6\sigma ) {3}^{\sigma }+ ( \sigma ^{2}+3\sigma ) {2}^{\sigma } }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} & 0 & {\frac{ ( \sigma ^{2}+5\sigma ) {4}^{\sigma }- ( 2\sigma ^{2}+6\sigma ) {3}^{\sigma }+ ( \sigma ^{2}+3\sigma ) {2}^{\sigma } }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline -{\frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} & -{ {\frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } & -{ \frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline \end{array}\displaystyle \ , \\& h_{y}= \textstyle\begin{array} {|c|c|c|} \hline -{ \frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} & -{\frac{ ( \sigma ^{2}+5\sigma ) {4}^{\sigma }- ( 2\sigma ^{2}+6\sigma ) {3}^{\sigma }+ ( \sigma ^{2}+3\sigma ) {2}^{\sigma } }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} & {\frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline -{ {\frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } & 0 & { {\frac{ ( \sigma ^{2}+3 \sigma ) {2}^{\sigma } -2 \sigma ^{2}-4\sigma }{B ( \sigma )\varGamma ( \sigma +2 ) }} } \\ \hline -{\frac{ ( 1-\sigma ) \varGamma ( \sigma +2 ) + \sigma ( \sigma +2 ) }{B ( \sigma ) \varGamma ( \sigma +2 ) }} & {\frac{ ( \sigma ^{2}+5\sigma ) {4}^{\sigma }- ( 2\sigma ^{2}+6\sigma ) {3}^{\sigma }+ ( \sigma ^{2}+3\sigma ) {2}^{\sigma } }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} & {\frac{ ( \sigma ^{2}+4\sigma ) {3}^{\sigma }- (2\sigma ^{2}+5\sigma ) {2}^{\sigma } +\sigma ^{2}+2\sigma }{ B ( \sigma ) \varGamma ( \sigma +2 ) }} \\ \hline \end{array}\displaystyle \ . \end{aligned}$$

3.3 The masks based on the Euler method (Eu)

The following expansion is introduced in the reference [45] to approximate ABC-fractional integral (11) at \(t=t_{n}\) as follows:

$$ \begin{aligned} {\mathcal{I}}_{\mathrm{Eu}}^{ \sigma } \mathcal{K} (t_{n} ) &= \frac{1 -\sigma }{\mathcal{M}(\sigma ) } \mathcal{K} (t_{n} ) + \frac{\sigma h^{\sigma }}{\mathcal{M}(\sigma )\varGamma (\sigma +1) } \sum_{m=0}^{n-1} \theta _{n,m} \mathcal{K} (t_{m} ), \end{aligned} $$
(26)

where

$$ \theta _{n,m}= (n-m )^{\sigma }- (n-m-1 )^{\sigma }. $$
(27)

Equation (26) can be transformed into the following form:

$$ \begin{aligned}[b] {\mathcal{I}}_{\mathrm{Eu}}^{ \sigma } \mathcal{K} (t_{n} ) &= \biggl[{ \frac{ 1-\sigma }{B ( \sigma ) }} \biggr] \mathcal{K} (t_{n} ) + \biggl[{ \frac{\sigma {h}^{\sigma }}{B ( \sigma ) \varGamma ( \sigma +1 )}} \biggr] \mathcal{K} (t_{n-1} ) + \biggl[{ \frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \biggr] \mathcal{K} (t_{n-2} ) \\ &\quad{} + \biggl[{ \frac{\sigma {h}^{\sigma } ( {3}^{\sigma }-{2}^{\sigma } )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \biggr] \mathcal{K} (t_{n-3} ) +\cdots .\end{aligned} $$
(28)

So, we can write corresponding expressions related to x and y directions as follows:

$$\begin{aligned}& \begin{aligned}[b] {}^{x} { \mathcal{I}}_{\mathrm{Eu}}^{ \sigma } \mathcal{K}(x,y) &\approx \biggl[{ \frac{ 1-\sigma }{B ( \sigma ) }} \biggr] \mathcal{K} (x,y ) \\ &\quad{} + \biggl[{ \frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} \biggr] \mathcal{K} (x-1,y ) + \biggl[{ \frac{\sigma ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \biggr] \mathcal{K} (x-2,y ) \\ &\quad{}+ \biggl[{ \frac{\sigma ( {3}^{\sigma }-{2}^{\sigma } )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \biggr] \mathcal{K} (x-3,y ) +\cdots , \end{aligned} \end{aligned}$$
(29)
$$\begin{aligned}& \begin{aligned}[b] {}^{y} { \mathcal{I}}_{\mathrm{Eu}}^{ \sigma } \mathcal{K}(x,y) &\approx \biggl[{ \frac{ 1-\sigma }{B ( \sigma ) }} \biggr] \mathcal{K} (x,y ) \\ &\quad{} + \biggl[{ \frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} \biggr] \mathcal{K} (x,y-1 ) + \biggl[{ \frac{\sigma ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \biggr] \mathcal{K} (x,y-2 ) \\ &\quad{}+ \biggl[{ \frac{\sigma ( {3}^{\sigma }-{2}^{\sigma } )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \biggr] \mathcal{K} (x,y-3 ) +\cdots . \end{aligned} \end{aligned}$$
(30)

Then, the following sequence of required coefficients is derived:

$$ \begin{aligned} \rho _{0}&={\frac{ 1-\sigma }{B ( \sigma ) }} , \\ \rho _{1}&={ \frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}}, \\ \rho _{2}&={ \frac{\sigma ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \\ \rho _{3}&={ \frac{\sigma ( {3}^{\sigma }- {2}^{\sigma } )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }}. \end{aligned} $$
(31)

Mask EU1

The first form of \(3\times 3\) fractional masks based on the Euler method are defined as follows:

$$\begin{aligned}& h_{x}= \textstyle\begin{array} {|c|c|c|} \hline -{ \frac{ 1-\sigma }{B ( \sigma ) }} & 0 & {\frac{ 1-\sigma }{B ( \sigma ) }} \\ \hline -{\frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} & 0 & {\frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} \\ \hline -{\frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} & 0 & {\frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \\ \hline \end{array}\displaystyle \ ,\quad\quad h_{y}= \textstyle\begin{array} {|c|c|c|} \hline {\frac{ 1-\sigma }{B ( \sigma ) }} & { \frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} & {\frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \\ \hline 0 & 0 & 0 \\ \hline -{\frac{ 1-\sigma }{B ( \sigma ) }} & -{\frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} & -{ \frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \\ \hline \end{array}\displaystyle \ . \end{aligned}$$

Mask EU2

The second group of \(3\times 3\) fractional masks can be also constructed as follows:

$$\begin{aligned}& h_{x}= \textstyle\begin{array} {|c|c|c|} \hline { \frac{ 1-\sigma }{B ( \sigma ) }} & {\frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} & {\frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \\ \hline -{\frac{\sigma {h}^{\sigma } ( {3}^{\sigma }- {2}^{\sigma } )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} & 0 & {\frac{\sigma {h}^{\sigma } ( {3}^{\sigma }- {2}^{\sigma } )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \\ \hline -{\frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} & -{\frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} & -{ \frac{ 1-\sigma }{B ( \sigma ) }} \\ \hline \end{array}\displaystyle \ , \\& h_{y}= \textstyle\begin{array} {|c|c|c|} \hline -{\frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} & -{ \frac{\sigma {h}^{\sigma } ( {3}^{\sigma }- {2}^{\sigma } )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} & {\frac{ 1-\sigma }{B ( \sigma ) }} \\ \hline -{\frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} & 0 & {\frac{\sigma }{B ( \sigma ) \varGamma ( \sigma +1 )}} \\ \hline -{\frac{ 1-\sigma }{B ( \sigma ) }} & {\frac{\sigma {h}^{\sigma } ( {3}^{\sigma }- {2}^{\sigma } )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} & {\frac{\sigma {h}^{\sigma } ( {2}^{\sigma }-1 )}{ B ( \sigma ) \varGamma ( \sigma +1 ) }} \\ \hline \end{array}\displaystyle \ . \end{aligned}$$

3.4 The masks based on a middle point (MP) approach

This part also looks for a new structure for the fractional mask. To this end, we reconsider the ABC-fractional integral as

$$ {\mathcal{I}}^{ \sigma } \mathcal{K} (t ) = \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \int _{0}^{t} \frac{\mathcal{K}(\omega )}{(t-\omega )^{1-\sigma }}\,d\omega . $$
(32)

Applying the new variable of \(\tau =t-\omega \) in the integral (32), we have

$$ {\mathcal{I}}^{ \sigma } \mathcal{K} (t ) = \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \int _{0}^{t} \frac{\mathcal{K}(t-\tau )}{ \tau ^{1-\sigma }}\,d\tau . $$
(33)

Now, by dividing the integral in (33), we have

$$ {\mathcal{I}}_{t}^{ \sigma } \mathcal{K} (t ) = \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \sum _{m=0}^{n-1} \int _{t_{m}}^{t_{m+1}} \frac{\mathcal{K}(t-\tau )}{ \tau ^{1-\sigma }}\,d\tau . $$
(34)

The integrals in the above relation can be approximated as follows:

$$ \int _{t_{m}}^{t_{m+1}} \frac{ \mathcal{K}(\tau )}{\tau ^{1-\sigma }} \,d \tau \approx \frac{\mathcal{K}(t_{m})+\mathcal{K}(t_{m+1})}{2} \int _{t_{m}}^{t_{m+1}} \frac{d\tau }{\tau ^{1-\sigma }}. $$
(35)

Given this approximation, (34) will lead to the following form:

$$ \begin{aligned}[b] {\mathcal{I}}^{ \sigma } \mathcal{K} (t ) &= \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \sum _{m=0}^{n-1} \frac{\mathcal{K}(t-t_{m})+\mathcal{K}(t-t_{m+1})}{2} \int _{t_{m}}^{t_{m+1}} \frac{d\tau }{ \tau ^{1-\sigma }}, \\ &=\frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t )+ \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \sum _{m=0}^{n-1} \frac{\mathcal{K}(t-t_{m})+\mathcal{K}(t-t_{m+1})}{2\sigma } \bigl[t_{m+1}^{ \sigma }-t_{m}^{\sigma } \bigr]. \end{aligned} $$
(36)

Inserting \(t=t_{n}=nh\) in (34), one gets

$$ \begin{aligned}[b] {\mathcal{I}}^{ \sigma } \mathcal{K} (t_{n} )&= \frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t_{n} ) \\ &\quad{} + \frac{\sigma }{\varGamma (\sigma )\mathcal{M}(\sigma )} \sum_{m=0}^{n-1} \frac{\mathcal{K}(t_{n}-t_{m})+\mathcal{K}(t_{n}-t_{m+1})}{2\sigma } \bigl[{\bigl({(m+1)h}\bigr)}^{\sigma }-{({ m h})}^{\sigma } \bigr], \\ &=\frac{1-\sigma }{\mathcal{M}(\sigma )} \mathcal{K} (t_{n} )+\frac{h^{\sigma }}{\varGamma (\sigma )\mathcal{M}(\sigma )} \sum _{m=0}^{n-1} \frac{\mathcal{K}(t_{n-m})+\mathcal{K}(t_{n-m-1})}{ 2} \bigl[{({m+1})^{ \sigma }}-{m^{\sigma }} \bigr]. \end{aligned} $$
(37)

By performing a series of simple algebraic calculations, Eq. (37) can be transformed into the following equation:

$$ \begin{aligned}[b] {\mathcal{I}}_{\mathrm{MP}}^{ \sigma } \mathcal{K} (t_{n} )={}& \biggl[ \frac{1-\sigma }{\mathcal{M}(\sigma )}+ \frac{h^{\sigma }}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (t_{n} ) \\ & {}+ \biggl[{ \frac{ {h}^{\sigma }}{2B ( \sigma )\varGamma ( \sigma ) } \bigl( { { {2}^{\sigma } } { }} \bigr) } \biggr] \mathcal{K} (t_{n-1} ) + \biggl[{ \frac{ {h}^{\sigma }}{B ( \sigma )\varGamma ( \sigma ) } \biggl( { \frac{ {3}^{\sigma }-1 }{ 2 }} \biggr) } \biggr] \mathcal{K} (t_{n-2} ) \\ &{}+ \biggl[{ \frac{ {h}^{\sigma }}{B ( \sigma )\varGamma ( \sigma ) } \biggl( {\frac{ {4}^{\sigma }-{2}^{\sigma } }{ 2 }} \biggr) } \biggr] \mathcal{K} (t_{n-3} ) + \cdots. \end{aligned} $$
(38)

So, we can write corresponding expressions related to x and y directions as follows:

$$\begin{aligned}& \begin{aligned}[b] {}^{x} { \mathcal{I}}_{\mathrm{MP}}^{ \sigma } \mathcal{K}(x,y) &\approx \biggl[ \frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (x,y ) \\ &\quad{} + \biggl[ \frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (x-1,y ) + \biggl[ \frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (x-2,y ) \\ &\quad{} + \biggl[ \frac{4^{\sigma }-2^{\sigma }}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (x-3,y ) +\cdots , \end{aligned} \end{aligned}$$
(39)
$$\begin{aligned}& \begin{aligned}[b] {}^{y} { \mathcal{I}}_{\mathrm{MP}}^{ \sigma } \mathcal{K}(x,y) &\approx \biggl[ \frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (x,y ) \\ &\quad{} + \biggl[\frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (x,y-1 ) + \biggl[ \frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (x,y-2 ) \\ &\quad{}+ \biggl[ \frac{4^{\sigma }-2^{\sigma }}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \biggr] \mathcal{K} (x,y-3 ) +\cdots . \end{aligned} \end{aligned}$$
(40)

Finally, the coefficients needed to make two new fractional masks are as follows:

$$ \begin{aligned} \rho _{0}&= \frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} , \\ \rho _{1}&= \frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )}, \\ \rho _{2}&=\frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )}, \\ \rho _{3}&= \frac{4^{\sigma }-2^{\sigma }}{2\mathcal{M}(\sigma )\varGamma (\sigma )}. \end{aligned} $$
(41)

These coefficients will be used to construct the following fractional masks.

Mask MP1

The first fractional mask to determine the edges of an image takes the following structure:

$$\begin{aligned}& h_{x}= \textstyle\begin{array} {|c|c|c|} \hline - \frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & 0 & \frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline -\frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} & 0 & \frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline -\frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & 0 & \frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline \end{array}\displaystyle \ , \\& h_{y}= \textstyle\begin{array} {|c|c|c|} \hline \frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & \frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} & \frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline 0 & 0 & 0 \\ \hline -\frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & -\frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} & -\frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline \end{array}\displaystyle \ . \end{aligned}$$

Mask MP2

Taking (6) and (41) into account, the second structure for the mask is also proposed:

$$\begin{aligned}& h_{x}= \textstyle\begin{array} {|c|c|c|} \hline \frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & \frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} & \frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline -\frac{4^{\sigma }-2^{\sigma }}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & 0 & \frac{4^{\sigma }-2^{\sigma }}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline -\frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & -\frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} & -\frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline \end{array}\displaystyle \ , \\& h_{y}= \textstyle\begin{array} {|c|c|c|} \hline -\frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & - \frac{4^{\sigma }-2^{\sigma }}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & \frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline -\frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} & 0 & \frac{2^{\sigma -1}}{ \mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline -\frac{2\varGamma (\sigma )(1-\sigma )+1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & \frac{4^{\sigma }-2^{\sigma }}{2\mathcal{M}(\sigma )\varGamma (\sigma )} & \frac{3^{\sigma }-1}{2\mathcal{M}(\sigma )\varGamma (\sigma )} \\ \hline \end{array}\displaystyle \ . \end{aligned}$$

It is important to note that all the masks presented in this section are new, and they have not been reported in the corresponding previous literature.

4 Numerical simulations

The main criterion for measuring the efficiency of the proposed kernels in this paper is the use of peak signal to noise ratio (PSNR) definition. This index can be calculated using the following formula:

$$ \mathit{PSNR}=10 \log _{10} \frac{512\times 512}{\mathit{MSE}}, $$
(42)

where

$$ \mathit{MSE}= \frac{1}{512\times 512}\sum_{j=1}^{N} \sum_{i=1}^{M} \bigl[I^{*}(i,j)-I(i,j) \bigr]^{2}, $$
(43)

where \(I^{*}\) is the original image, and I denotes the values of the pixels in the comparing image.

To assess the performance of the proposed fractional-order masks, different input images are used: “flower”, “hotel”, “cottages”, and “fractional man” with a size of \(512 \times 512\) pixels are considered. The examined mask also include “GL1”, “GL2”, “TA1”, “TA2”, “Eu1”, “Eu2”, “MP1”, and “MP2”.

One of the main features of the designed masks in this paper is that their structures depend on knowing the order of the fractional parameter, say σ. It is important to note that determining the optimal value for σ, in general, is very difficult and complicated. In fact, for each block of the image, the appropriate value for this parameter should be determined considering the structure of the image in that block. In this paper, the performance of the algorithms is compared with the assumption that sigma is within a range of \([0.2, 2]\). Then, the value for σ is determined in such a way that the corresponding image has the largest possible PSNR value compared to the other values. In this case the corresponding image is considered as the output of the method for that image. The values obtained for σ and the resulting images are reported in Figs. 1–4. In these experiments, the performance of the proposed masks is compared with three known mask types, including the Canny, Prewitt, and Sobol masks. In Figs. 5–8, we have also plotted the graph of the calculated values of the PNSR for each image for σ in that interval. All simulations have been performed via MATLAB. The comparison results of PSNR of the proposed masks are compared with some well-known algorithms in Table 1. By comparing the results reported in Table 1, it is apparent that in any approximation considered for the fractional operator, the structures related to mask (6) have a much better performance than the masks obtained by (5). This feature can be seen in all four approximations considered in the article. Moreover, from the visual quality of the acquired images and their corresponding PSNRs, it seems that the best performance among the methods belongs to two corresponding TA masks, which in most cases have the highest possible value for PSNR. By comparing the methods presented in this paper with other well-known methods, it can be concluded that these masks have much better results than these algorithms with almost equal computation cost. These new masks can be a good alternative to classic masks in defining the edge of an image.

Figure 1
figure 1

Best PSNR obtained by different methods for the flower image

Full size image
Figure 2
figure 2

Best PSNR obtained by different methods for the hotel image

Full size image
Figure 3
figure 3

Best PSNR obtained by different methods for the cottages image

Full size image
Figure 4
figure 4

Best PSNR obtained by different methods for the fractional man image

Full size image
Figure 5
figure 5

Comparisons between PSNR of proposed masks for different σs in the flower image

Full size image
Figure 6
figure 6

Comparisons between PSNR of proposed masks for different σs in the hotel image

Full size image
Figure 7
figure 7

Comparisons between PSNR of proposed masks for different σs in the cottages image

Full size image
Figure 8
figure 8

Comparisons between PSNR of proposed masks for different σs in the fractional man image

Full size image
Table 1 Comparisons of PSNRs obtained by different masks
Full size table

5 Conclusions

Integrals and derivatives of fractional order have been used successfully in many research fields. One of these very important and practical fields, in which the fractional differential account has been used a lot, is image processing. Using basic concepts of fractional differential calculus, such as derivative and integral, has made significant progress role in many branches of image processing. This manuscript attempts to design several new edge detectors using the fractional definition for integral in the sense of Atangana–Baleanu operator. Using the advantages of fractional calculus, such as flexibility in choosing the fractional derivative order, one can overcome the existing problems in the well-known masks such as Canny, Prewitt, and Sobel. The empirical results prove that the new fractional kernels presented in this paper have better performance than other existing methods in improving edge information and preserving image quality. It is also concluded that they can be considered as new excellent alternative kernels to enhance edge information of an image. In fact, they can reveal more accurate information than traditional algorithms. The computational cost of each of the new fractional masks is the same as the computational cost of conventional fractional masks. It is important to note that the masks used in this article can also be easily applied to color images. The next step in this regard could be to provide an idea for calculating the optimal value for the order of fractional operator.