1 Introduction

Let \(\mathcal H\) be the class of all analytic functions in \(\mathbb D:= \left\{ z \in \mathbb C: |z|<1 \right\}\) and \(\mathcal A\) be the subclass of functions f normalized by \(f(0)=0\), \(f'(0)=1,\) i.e., of the form

$$\begin{aligned} f(z) = \sum _{n=1}^{\infty }a_{n}z^{n},\quad a_1:=1,\ z\in \mathbb D, \end{aligned}$$
(1)

and let \(\mathcal S\) be the subclass of \(\mathcal A\) of univalent functions. We denote by \(\mathcal P'\) the subfamily of \(\mathcal A\) of all functions f such that

$$\begin{aligned} {{\,\mathrm{Re}\,}}f'(z) > 0,\quad z\in \mathbb D. \end{aligned}$$
(2)

Functions in \(\mathcal P'\) are called functions of bounded turning (cf. [6, Vol. I, p. 101]), and in recent times many authors have denoted this class of functions by \({\mathcal {R}}\).

It is well-known [1] (cf. [6, Vol. I, p. 88]) that \(\mathcal P'\subset \mathcal S,\) and is a fundamental subfamily of univalent functions which has been extensively studied by many authors, e.g., [17, 18].

For \(q,n \in \mathbb {N},\) the Hankel determinant \(H_{q,n}(f)\) of function \(f \in {\mathcal A}\) of the form (1) is defined as

$$H_{{q,n}} (f): = \left| {\begin{array}{*{20}c} {a_{n} } & {a_{{n + 1}} } & \cdots & {a_{{n + q - 1}} } \\ {a_{{n + 1}} } & {a_{{n + 2}} } & \cdots & {a_{{n + q}} } \\ \vdots & \vdots & \vdots & \vdots \\ {a_{{n + q - 1}} } & {a_{{n + q}} } & \cdots & {a_{{n + 2(q - 1)}} } \\ \end{array} } \right|.$$

General results for Hankel determinants of any degree with their applications can be found in [4, 20, 21, 23]. For a subfamily \(\mathcal F\) of \(\mathcal A,\) q and n,  computing the upper bound of \(H_{q,n}(f)\) represents an interesting and important problem. Hayman [7] examined the second Hankel determinant of mean for univalent functions. Recently, many authors examined the second Hankel determinant \(H_{2,2}(f)=a_2a_4-a_3^2\) (see e.g., [5, 15] with further references). The problem of finding sharp estimating of the third Hankel determinant

$$\begin{aligned} H_{3,1}(f) = \left| {\begin{array}{*{20}c} a_1 & a_1 & a_3 \\ a_2 & a_3 & a_4 \\ a_3 & a_4 & a_5 \end{array} } \right| =a_3(a_2a_4 - a_3^2) - a_4(a_4 - a_2a_3) + a_5(a_3 - a_2^2) \end{aligned}$$
(3)

is technically much more difficult, and few sharp bounds have been obtained. However, sharp bounds of \(|H_{3,1}(f)|\) have been obtained for convex functions [12], starlike functions of order 1/2 [14], and functions \(f\in \mathcal A\) which satisfy the condition \({{\,\mathrm{Re}\,}}f(z)/z>\alpha ,\ z\in \mathbb D,\) in the case \(\alpha =0\) and \(\alpha =1/2\) [11], and functions \(f\in \mathcal A\) such that \(|(z/f(z))^2-1|<1\) for \(z\in \mathbb D\) [19].

When \(f\in \mathcal P'\) Janteng et al. [9] showed that \(|H_{3,1}(f)|\le 439/540=0.81296\dots\) which is not sharp, and originally proved in [2]. The proof used (3) and sharp estimates of expressions contained in (3), namely \(|a_2a_4-a_3^2|\le 4/9\) found in [8], \(|a_2a_3-a_4|\le 1/2\) in [2] and reproved in [9], \(|a_3-a_2^2|\le 2/3\) cited from [2] (see also [10], where this result was proved earlier) and \(|a_n|\le 2/n\) found in [18]. We also note that proofs in [2] contained some gaps, and new proofs were obtained in [9]. The aforementioned estimate of \(|H_{3,1}(f)|\) in \(\mathcal P'\) was improved in [24], where the author shown that \(|H_{3,1}(f)|\le 41/60=0.683\dots\), which is also not sharp, and discussed \(H_{3,1}(f)\) for subclasses \(\mathcal P'^{(2)}\) and \(\mathcal P'^{(3)}\) of \(\mathcal P'\) consisting of 2-fold and 3-fold symmetric functions, respectively, and obtained the following sharp bounds: \(|H_{3,1}(f)|\le 2\sqrt{6}/45=0.108\dots\) and \(|H_{3,1}(f)|\le 1/4\) for f in \(\mathcal P'^{(2)}\) and \(\mathcal P'^{(3)},\) respectively.

In this paper, we show that \(|H_{3,1}(f)|\le 1/4\) for \(f\in \mathcal P'\) and that the inequality is sharp.

Since the class \(\mathcal P'\) can be represented using the Carathéodory class \(\mathcal P\), i.e., the class of functions \(p \in {\mathcal H}\) of the form

$$\begin{aligned} p(z) = 1 + \sum _{n=1}^{\infty }c_{n}z^{n}, \quad z\in \mathbb D, \end{aligned}$$
(4)

having a positive real part in \(\mathbb D,\) the coefficients of functions in \(\mathcal P'\) can be expressed as coefficients of functions in \(\mathcal P.\) We then obtain the upper bound of \(|H_{3,1}(f)|,\) basing our analysis on the well-known formulas of coefficient \(c_2\) (e.g., [22, p. 166]), the formula \(c_3\) due to Libera and Zlotkiewicz [16, 17], and the formula for \(c_4\) recently found in [13].

2 Main result

The basis for proof of the main result is the following lemma. It contains the well known formula for \(c_2\) (e.g., [22, p. 166]). the formula for \(c_3\) due to Libera and Zlotkiewicz [16, 17] and the formula for \(c_4\) found in [13].

Lemma 1

If \(p \in {\mathcal P}\) is of the form (4) with \(c_1\ge 0,\) then

$$\begin{aligned} c_1= \, & {} 2\zeta _1, \end{aligned}$$
(5)
$$\begin{aligned} c_2= \, & {} 2\zeta _1^2+2(1-\zeta _1^2)\zeta _2, \end{aligned}$$
(6)
$$\begin{aligned} c_3= \, & {} 2\zeta _1^3+4(1-\zeta _1^2)\zeta _1\zeta _2-2(1-\zeta _1^2)\zeta _1\zeta _2^2+2(1-\zeta _1^2)(1-|\zeta _2|^2)\zeta _3 \end{aligned}$$
(7)

and

$$\begin{aligned} \begin{aligned} c_4= \, &2\zeta _1^4+2(1-\zeta _1^2)\zeta _2\left( \zeta _1^2\zeta _2^2-3\zeta _1^2\zeta _2+3\zeta _1^2+\zeta _2\right) \\&+2(1-\zeta _1^2)(1-|\zeta _2|^2)\zeta _3\left( 2\zeta _1-2\zeta _1\zeta _2-\overline{\zeta _2}\zeta _3\right) \\&+2(1-\zeta _1^2)(1-|\zeta _2|^2)(1-|\zeta _3|^2)\zeta _4, \end{aligned} \end{aligned}$$
(8)

for some \(\zeta _1\in [0,1]\) and \(\zeta _2,\zeta _3,\zeta _4\in \overline{\mathbb D}.\)

We now prove the main theorem of this paper.

Theorem 1

$$\begin{aligned} \max \{|H_{3,1}(f)|: f\in \mathcal P'\}= \frac{1}{4} \end{aligned}$$
(9)

with extreme function \(f_0\in \mathcal P'\) given by

$$\begin{aligned} f_0'(z):=\frac{1-z^3}{1+z^3},\quad z\in \mathbb D. \end{aligned}$$
(10)

Proof

Let \(f \in \mathcal P'\) and be given by (1). Then by (2),

$$\begin{aligned} f'(z)=p(z),\quad z\in \mathbb D, \end{aligned}$$
(11)

for some function \(p \in \mathcal P\) of the form (4). Since both the classes \(\mathcal P'\) and \(\mathcal P\) and the functional \(H_{3,1}(f)\) are invariant under the rotations, we may assume that \(c_1 \in [0,2]\) ( [3], see also [6, Vol. I, p. 80, Theorem 3]), i.e., in view of (5) that \(\zeta _1\in [0,1].\)

Substituting (1) and (4) into (11) and equating coefficients, we obtain

$$\begin{aligned} a_2= \frac{1}{2}c_1, \quad a_3=\frac{1}{3}c_2,\quad a_4 = \frac{1}{4}c_3,\quad a_5= \frac{1}{5}c_4. \end{aligned}$$

Hence, by (3), we have

$$\begin{aligned} \begin{aligned} H_{3,1}(f)=\frac{1}{2160}\left( 180c_1c_2c_3-80c_2^3-135c_3^2+144c_2c_4-108c_1^2c_4\right) . \end{aligned} \end{aligned}$$
(12)

Using (5)–(8) by straightforward algebraic computation we obtain

$$\begin{aligned} \begin{aligned} 180c_1c_2c_3= \,&1440\left[ \zeta _1^6+(1-\zeta _1^2)(3-\zeta _2)\zeta _1^4\zeta _2\right. \\&+(1-\zeta _1^2)^2(2-\zeta _2)\zeta _1^2\zeta _2^2+(1-\zeta _1^2)(1-|\zeta _2|^2)\zeta _1^3\zeta _3\\&\left. +(1-\zeta _1^2)^2(1-|\zeta _2|^2)\zeta _1\zeta _2\zeta _3\right] ,\\ -80c_2^3=&-640\left[ \zeta _1^6+3(1-\zeta _1^2)\zeta _1^4\zeta _2+3(1-\zeta _1^2)^2\zeta _1^2\zeta _2^2+(1-\zeta _1^2)^3\zeta _2^3\right] ,\\ -135c_3^2=&-540\left[ \zeta _1^6+2(1-\zeta _1^2)(2-\zeta _2)\zeta _1^4\zeta _2+(1-\zeta _1^2)^2(2-\zeta _2)^2\zeta _1^2\zeta _2^2\right. \\&+2(1-\zeta _1^2)(1-|\zeta _2|^2)\zeta _1^3\zeta _3+2(1-\zeta _1^2)^2(1-|\zeta _2|^2)(2-\zeta _2)\zeta _1\zeta _2\zeta _3\\&\left. +(1-\zeta _1^2)^2(1-|\zeta _2|^2)^2\zeta _3^2\right] \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned}&144c_2c_4-108c_1^2c_4\\&\quad =288\left[ -\zeta _1^6+(1-\zeta _1^2)(-\zeta _1^2\zeta _2-\zeta _1^2\zeta _2^3+3\zeta _1^2\zeta _2^2-\zeta _2^2)\zeta _1^2\right. \\&\qquad +2(1-\zeta _1^2)^2(3\zeta _1^2\zeta _2^2+\zeta _1^2\zeta _2^4-3\zeta _1^2\zeta _2^3+\zeta _2^3)\\&\qquad -(1-\zeta _1^2)(1-|\zeta _2|^2)\left( 2\zeta _1\zeta _3-2\zeta _1\zeta _2\zeta _3-\overline{\zeta _2}\zeta _3^2\right) \zeta _1^2\\&\qquad +2(1-\zeta _1^2)^2(1-|\zeta _2|^2)(2\zeta _1\zeta _2\zeta _3-2\zeta _1\zeta _2^2\zeta _3-|\zeta _2|^2\zeta _3^2)\\&\qquad -(1-\zeta _1^2)(1-|\zeta _2|^2)(1-|\zeta _3|^2)\zeta _1^2\zeta _4\\&\qquad \left. +2(1-\zeta _1^2)^2(1-|\zeta _2|^2)(1-|\zeta _3|^2)\zeta _2\zeta _4\right] . \end{aligned} \end{aligned}$$

Substituting the above expression into (12), we obtain

$$\begin{aligned} \begin{aligned} H_{3,1}(f) = \frac{1}{540}\left[ \gamma _1(\zeta _1,\zeta _2) + \gamma _2(\zeta _1,\zeta _2)\zeta _3 + \gamma _3(\zeta _1,\zeta _2)\zeta _3^2 + \gamma _4(\zeta _1,\zeta _2,\zeta _3)\zeta _4\right] , \end{aligned} \end{aligned}$$
(13)

where for \(\zeta _1\in [0,1]\) and \(\zeta _2,\zeta _3,\zeta _4\in \overline{{\mathbb {D}}},\)

$$\begin{aligned} \begin{aligned} \gamma _{1}(\zeta _1,\zeta _2) :=&-7\zeta _1^6-12(1-\zeta _1^2)\zeta _1^4\zeta _2+6(1-\zeta _1^2)(10-\zeta _1^2)\zeta _1^2\zeta _2^2\\&+4(1-\zeta _1^2)(5\zeta _1^4-19\zeta _1^2-4)\zeta _2^3+9(1-\zeta _1^2)^2\zeta _1^2\zeta _2^4,\\ \gamma _{2}(\zeta _1,\zeta _2) :=&-18(1-\zeta _1^2)(1-|\zeta _2|^2)\left[ 3\zeta _1^3-(2\zeta _1^2+6)\zeta _1\zeta _2+(1-\zeta _1^2)\zeta _1\zeta _2^2\right] ,\\ \gamma _{3}(\zeta _1,\zeta _2) := \, &9(1-\zeta _1^2)(1-|\zeta _2|^2)\left[ 8\zeta _1^2\overline{\zeta _2}-(1-\zeta _1^2)(|\zeta _2|^2+15)\right] \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \gamma _4(\zeta _1,\zeta _2,\zeta _3) := 72(1-\zeta _1^2)(1-|\zeta _2|^2)(1-|\zeta _3|^2)\left[ -\zeta _1^2+2(1-\zeta _1^2)\zeta _2\right] . \end{aligned}$$

Since \(|\zeta _4|\le 1,\) from (13) we obtain

$$\begin{aligned} \begin{aligned} |H_{3,1}(f)|\le \,&\frac{1}{540}\left[ |\gamma _1(\zeta _1,\zeta _2)|+|\gamma _2(\zeta _1,\zeta _2)||\zeta _3|\right. \\&\left. +|\gamma _3(\zeta _1,\zeta _2)||\zeta _3|^2+|\gamma _4(\zeta _1,\zeta _2,\zeta _3)|\right] \\ \le&\frac{1}{540}\max \{\varPhi (x,y,u): (x,y,u)\in D:=[0,1]^3\}, \end{aligned} \end{aligned}$$

where

$$\begin{aligned} \varPhi (x,y,u) := \, \varphi _1(x,y)+\varphi _4(x,y)+\varphi _2(x,y)u +\left[ \varphi _3(x,y)-\varphi _4(x,y)\right] u^2, \end{aligned}$$

with

$$\begin{aligned} \begin{aligned} \varphi _1(x,y):=&7x^6+12(1-x^2)x^4y+6(1-x^2)(10-x^2)x^2y^2\\&+4(1-x^2)(5x^2+1)(4-x^2)y^3+9(1-x^2)^2x^2y^4,\\ \varphi _2(x,y):=&18x\left[ 3x^2+2(3+x^2)y+(1-x^2)y^2\right] (1-x^2)(1-y^2),\\ \varphi _3(x,y):=&9\left[ 8x^2y+(1-x^2)(y^2+15)\right] (1-x^2)(1-y^2), \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \varphi _{4}(x,y):=72[x^2+2(1-x^2)y](1-x^2)(1-y^2). \end{aligned}$$

We will now show that

$$\begin{aligned} \max \{\varPhi (x,y,u): (x,y,u)\in D\}=135. \end{aligned}$$

A. On the vertices of D, we have

$$\begin{aligned} \begin{aligned}&\varPhi (0,0,0)=0,\quad \varPhi (0,0,1)=135,\quad \varPhi (0,1,0)=\varPhi (0,1,1)=16,\\&\varPhi (1,0,0)=\varPhi (1,0,1)=\varPhi (1,1,0)=\varPhi (1,1,1)=7. \end{aligned} \end{aligned}$$

B. We next consider the edges of D.

1. \(y=0,\ u=0.\) Then

$$\begin{aligned} \begin{aligned}&\varPhi (x,0,0)=7x^6-72x^4+72x^2\le \varPhi (x_1,0,0)\\&\quad =\frac{96}{49}\left( 17\sqrt{102}-162\right) \approx 18.988,\quad x\in (0,1), \end{aligned} \end{aligned}$$

where

$$\begin{aligned} x_1:=\sqrt{\frac{2}{7}(12-\sqrt{102})}\approx 0.737. \end{aligned}$$

2. \(y=1,\ u=0.\) Then,

$$\begin{aligned} \begin{aligned}&\varPhi (x,1,0)=30x^6-168x^4+129x^2+16\le \varPhi (x_2,1,0)\\&\quad =\frac{1}{225}\left( 923\sqrt{1846}-30028\right) \approx 42.795,\quad x\in (0,1), \end{aligned} \end{aligned}$$

where

$$\begin{aligned} x_2:=\sqrt{\frac{56-\sqrt{1846}}{30}}\approx 0.659. \end{aligned}$$

3. \(x=0,\ u=0.\) Then,

$$\begin{aligned} \varPhi (0,y,0)=-128y^3+144y\le \varPhi \left( 0,\frac{\sqrt{6}}{4},0\right) =24\sqrt{6}\approx 58.788,\quad y\in (0,1). \end{aligned}$$

4. \(x=1,\ u=0.\) Then,

$$\begin{aligned} \varPhi (1,y,0)=7,\quad y\in [0,1]. \end{aligned}$$

5. \(y=0,\ u=1.\) Then,

$$\begin{aligned} \varPhi (x,0,1)=7x^6-54x^5+135x^4+54x^3-270x^2+135\le 135,\quad x\in (0,1), \end{aligned}$$

which is equivalent to

$$\begin{aligned} 7x^4-54x^3+135x^2+54x-270\le 0,\quad x\in [0,1], \end{aligned}$$

and is easily seen to be true.

6. \(y=1,\ u=1.\) Since \(\varPhi (x,1,1)=\varPhi (x,1,0)\) for \(x\in (0,1),\) which reduces to case 2.

7. \(x=0,\ u=1.\) Then,

$$\begin{aligned} \varPhi (0,y,1)=-9y^4+16y^3-126y^2+135\le 135,\quad y\in (0,1), \end{aligned}$$

which is equivalent to

$$\begin{aligned} y^2(-9y^2+16y-126)\le 0,\quad y\in (0,1), \end{aligned}$$

and again is evidently true.

8. \(x=1,\ u=1.\) Then,

$$\begin{aligned} \varPhi (1,y,1)=7,\quad y\in (0,1). \end{aligned}$$

9. \(x=0,\ y=0.\) Then,

$$\begin{aligned} \varPhi (0,0,u)=135u^2\le 135,\quad u\in (0,1). \end{aligned}$$

10. \(x=0,\ y=1.\) Then,

$$\begin{aligned} \varPhi (0,1,u)=16,\quad u\in (0,1). \end{aligned}$$

11. \(x=1,\ y=0.\) Then,

$$\begin{aligned} \varPhi (1,0,u)=7,\quad u\in (0,1). \end{aligned}$$

12. \(x=1,\ y=1.\) Then,

$$\begin{aligned} \varPhi (1,1,u)=7,\quad u\in (0,1). \end{aligned}$$

C. We consider now the faces of D.

1. \(x=1.\) Then,

$$\begin{aligned} \varPhi (1,y,u)=7,\quad y,u\in (0,1). \end{aligned}$$

2. \(x=0.\) Then,

$$\begin{aligned} \varPhi (0,y,u)=114y-128y^3+9(1-y^2)(1-y)(15-y)u^2=:G_1(y,u),\quad y,u\in (0,1). \end{aligned}$$

Clearly \(G_1\) has no critical point in \((0,1)\times (0,1).\)

3. \(y=1.\) Since \(\varPhi (x,1,u)=\varPhi (x,1,0)\) for \(x,u\in (0,1),\) this case reduces to the case B.2.

4. \(y=0.\) Then,

$$\begin{aligned} \begin{aligned}&\varPhi (x,0,u)=7x^6-72x^4+72x^2+54x^3(1-x^2)u+9(1-x^2)(15-23x^2)u^2\\&\quad =:G_2(x,u),\quad x,u\in (0,1). \end{aligned} \end{aligned}$$

We have

$$\begin{aligned} \frac{\partial G_2}{\partial u}=54x^3(1-x^2)+18(1-x^2)(15-23x^2)u,\quad x,u\in (0,1). \end{aligned}$$

Clearly \(\partial G_2/\partial u\not =0\) for \(x=\sqrt{15/23}.\) For \(x\not =\sqrt{15/23}\) we see that \(\partial G_2/\partial u=0\) if, and only if,

$$\begin{aligned} u=\frac{3x^3}{23x^2-15}=:u(x). \end{aligned}$$

Note that \(u(x)\in (0,1)\) if, and only if, \(23x^2-15>0\) and \(3x^3-23x^2+15<0,\) i.e., if, and only if, \(x\in (x_3,1),\) where \(x_3\approx 0.857.\) Since

$$\begin{aligned} \frac{\partial G_2}{\partial x}(x,u(x))=0 \end{aligned}$$

if, and only if,

$$\begin{aligned} 1840x^8-27360x^6+46176x^4-27360x^2+5400=0 \end{aligned}$$

and the last equation has no root in \((x_3,1)\) (all real roots are the following: \(x\approx \pm \, 0.68045\) and \(x\approx \pm \, 3.60968)\)), we conclude that \(G_2\) has no critical point.

5. \(u=0.\) We have

$$\begin{aligned} \begin{aligned} \varPhi (x,y,0)= \, &7x^6-72x^4+72x^2+(-12x^6+156x^4-288x^2+144)y\\&+(6x^6+6x^4-12x^2)y^2+(20x^6-240x^4+348x^2-128)y^3\\&+(9x^6-18x^4-9x^2)y^4=:G_3(x,y),\quad x,y\in (0,1). \end{aligned} \end{aligned}$$

Now set \(t:=x^2\in (0,1).\) Then \(\partial G_3/\partial y=0\) if, and only if,

$$\begin{aligned} (y+1)^2(1-3y)t^2+(-12-2y+55y^2+3y^3)t+12-32y^2=0, \end{aligned}$$
(14)

and \(\partial G_3/\partial t=0\) if, and only if,

$$\begin{aligned} \begin{aligned}&(7-12y+6y^2+20y^3+9y^4)t^2+(-48+104y-4y^2-160y^3-12y^4)t\\&\quad +24-96y-4y^2+116y^3+3y^4=0. \end{aligned} \end{aligned}$$
(15)

(a) Let \(y=1/3.\) Then, the Eq. (14) reduces to \(t=76/58>1.\)

(b) Suppose that \(y\not =1/3.\) Then by (14),

$$\begin{aligned} \varDelta :=9y^6-54y^5+2373y^4-276y^3-948y^2+96y+96>0 \end{aligned}$$

for \(y\in (0,1)\setminus \{1/3\}\) (all real roots of \(\varDelta =0\) are: \(y\approx -0.5108\) and \(y\approx -0.3278\)).

Let

$$\begin{aligned} t_{1,2}:=\frac{12+2y-55y^2-3y^3\mp \sqrt{\varDelta }}{2(y+1)^2(1-3y)}. \end{aligned}$$

(i) Observe that \(t_1>1.\) Indeed, for \(0< y<1/3\) and \(1/3<y<1\) this inequality is equivalent to the obviously true inequalities

$$\begin{aligned} (y+1)^2(1-3y)(72y^2-12y+4)>0 \end{aligned}$$

and

$$\begin{aligned} (y+1)^2(1-3y)(72y^2-12y+4)<0, \end{aligned}$$

respectively.

(ii) Since \(12+2y-55y^2-3y^3>0\) for \(0<y<1/3,\) then \(t_2>0.\) Note that the inequality \(t_2>0\) is false for \(y\in (1/3,y_1),\) where \(y_1\approx 0.479\) is a unique positive solution of \(12+2y-55y^2-3y^3=0.\) For \(y\in (y_1,1)\) the inequality \(t_2>0\) is equivalent to

$$\begin{aligned} (y+1)^2(3y-1)(8y^2-3)>0, \end{aligned}$$

which is true for \(y\in (\sqrt{6}/4,1).\)

Further, the inequality \(t_2<1\) which is equivalent to \(\sqrt{\varDelta }<-10-4y+45y^2-3y^3\) is evidently false for \(y\in (0,1/3),\) and is true when \(y\in (1/3,y_2),\) where \(y_2\approx 0.528\) is a unique root in (0, 1) of the equation \(-10-4y+45y^2-3y^3=0.\) For \(y\in (y_2,1)\) the inequality \(t_2<1\) is equivalent to

$$\begin{aligned} (y+1)^2(3y-1)(18y^2-3y+1)>0, \end{aligned}$$

which is also true.

Summarising \(t_2\in (0,1)\) if, and only if, \(y\in (\sqrt{6}/4,1),\) and substituting \(t_2\) into (15) we obtain

$$\begin{aligned} \begin{aligned}&(7-12y+6y^2+20y^3+9y^4)t_2^2+(-48+104y+4y^2-160y^3-12y^4)t_2\\&\quad +24-96y-4y^2+116y^3+3y^4=0, \end{aligned} \end{aligned}$$
(16)

which is equivalent to

$$\begin{aligned} \begin{aligned} L(y)&:=54y^{10}+180y^9+4194y^8+30672y^7+27630y^6-46356y^5\\&\qquad +13754y^4+4360y^3-7000y^2+1008y+624\\&=(-18y^7+30y^6+600y^5+660y^4-702y^3+394y^2-44y-72)\sqrt{\varDelta }=:R(y). \end{aligned} \end{aligned}$$
(17)

Note that \(L(y)>0\) for \(y\in (y_3,1)\) and \(R(y)>0\) for \(y\in (y_4,1),\) where \(y_3\approx 0.5782\) and \(y_4\approx 0.5555\) are roots of L and \(R/\sqrt{\varDelta },\) respectively. Therefore both sides of (17) are positive for \(y\in (\sqrt{6}/4,1).\) Thus (17) is equivalent to

$$\begin{aligned} \begin{aligned}&L^2(y)-R^2(y)=(y+1)^4(3y-1)^2(5184y^{13}-34560y^{12}+791424y^{11}\\&\qquad +6098976y^{10}+3934080y^9-14940000y^8-1216512y^7+6150528y^6\\&\qquad -4316544y^5+2653696y^4-1009152y^3+743424y^2-64512y-108288)=0. \end{aligned} \end{aligned}$$
(18)

A numerical computation shows that all real solution of the above equation are as follow:

$$\begin{aligned} \begin{aligned}&y=-1,\quad y=\frac{1}{3},\quad y\approx -\,4.2848,\quad y\approx -\,2.3680,\quad y\approx -\,1.0721,\\&y\approx -\,0.2646,\quad y\approx 0.5478,\quad y\approx 0.5613,\quad y\approx 1.0977. \end{aligned} \end{aligned}$$

Thus, the Eq. (18), so (16) has no solution in \((\sqrt{6}/4,1),\) and so , \(G_3\) has no critical point.

6. \(u=1.\) We have

$$\begin{aligned} \begin{aligned} \varPhi (x,y,1)= \,&7x^6-54x^5+135x^4+54x^3-270x^2+135\\&+(-12x^6-36x^5-60x^4-72x^3+72x^2+108x)y\\&+(6x^6+72x^5-192x^4-90x^3+312x^2+18x-126)y^2\\&+(20x^6+36x^5-24x^4+72x^3-12x^2-108x+16)y^3\\&+(9x^6-18x^5-27x^4+36x^3+27x^2-18x-9)y^4=:G_4(x,y) \end{aligned} \end{aligned}$$

for \(x,y\in (0,1).\) Then, \(\partial G_4/\partial x=0\) if, and only if,

$$\begin{aligned} \begin{aligned}&7x^5-45x^4+90x^3+27x^2-90x\\&\quad +(-12x^5-30x^4-40x^3-36x^2+24x+108)y\\&\quad +(6x^5+60x^4-128x^3-45x^2+104x+3)y^2\\&\quad +(20x^5+30x^4-16x^3+36x^2-4x-18)y^3\\&\quad +(9x^5-15x^4-18x^3+18x^2+9x-3)y^4=0 \end{aligned} \end{aligned}$$
(19)

and \(\partial G_4/\partial y=0\) if, and only if,

$$\begin{aligned} \begin{aligned}&-(1-x^2)\left[ -x^4-3x^3-6x^2-9x+(x^4+12x^3-31x^2-3x+21)y\right. \\&\qquad \left. +(5x^4+9x^3-x^2+27x-4)y^2+(3x^4-6x^3-6x^2+6x+3)y^3\right] =0. \end{aligned} \end{aligned}$$
(20)

Another numerical computation shows that all real solutions of the system of Eqs. (19) and (20) are as follow:

$$\begin{aligned} \begin{aligned}&(x=0,y=0),\quad (x\approx -\,0.9436,y\approx 0.9093),\quad (x\approx -\,0.9005,y\approx 0.2632),\\&(x\approx -\,0.7232,y\approx -\,18.8987),\quad (x\approx -\,0.4941,y\approx 0.9093),\\&(x\approx 1.2834,y\approx 16.4015),\quad (x\approx 5.4293,y\approx -\,3.4718,\\&(x\approx 6.1058,y\approx -\,3.0055),\quad (x\approx 8.9567,y\approx 0.3360). \end{aligned} \end{aligned}$$

Therefore, \(G_4\) has no critical point.

D. It remains to consider the interior of D,  i.e., \((0,1)^3.\) We have

$$\begin{aligned} \begin{aligned} \varPhi (x,y,u)= \,&\varphi _1(x,y)+\varphi _4(x,y)+\varphi _2(x,y)u +\left[ \varphi _3(x,y)-\varphi _4(x,y)\right] u^2\\&7x^6+12x^2(1-x^2)y+6x^2(1-x^2)(10-x^2)y^2\\&+4(1-x^2)(1+5x^2)(4-x^2)y^3+9x^2(1-x^2)^2y^4\\&+72\left[ x^2+2(1-x^2)y\right] (1-x^2)(1-y^2)\\&+18x\left[ 3x^2+(6+2x^2)y+(1-x^2)y^2\right] (1-x^2)(1-y^2)u\\&+9\left[ 15-23x^2+(24x^2-16)y+(1-x^2)y^2\right] (1-x^2)(1-y^2)u^2 \end{aligned} \end{aligned}$$

for \((x,y,u)\in (0,1)^3.\)

(a) Suppose that \(\varphi _3(x,y)=\varphi _4(x,y)\) for \(x,y\in (0,1).\) Since \(\varphi _1(x,y)+\varphi _4(x,y)\ge 0\) and \(\varphi _2(x,y)\ge 0\) for \(x,y\in (0,1),\) we have

$$\begin{aligned} \begin{aligned} \varPhi (x,y,u) = \,&\varphi _1(x,y)+\varphi _4(x,y)+\varphi _2(x,y)u\\ \le&\varphi _1(x,y)+\varphi _4(x,y)+\varphi _2(x,y)\\ \le&\max \{\varPhi (x,y,1): x,y\in [0,1],\ \varphi _3=\varphi _4\}\\ \le&\max \{\varPhi (x,y,1): x,y\in [0,1]\}\le 135. \end{aligned} \end{aligned}$$

The last inequality follows from A.3, B.5-8 and C.6.

(b) Suppose that \(\varphi _3(x,y)\not =\varphi _4(x,y)\) for \(x,y\in (0,1).\) Then \(\partial \varPhi /\partial u=0\) if, and only if,

$$\begin{aligned} u=\frac{-x\left[ 3x^2+(6+2x^2)y+(1-x^2)y^2\right] }{15-23x^2+(24x^2-16)y+(1-x^2)y^2}=:u(x,y) \end{aligned}$$

for \((x,y)\in (0,1)^2\) such that \(\varphi _3(x,y)\not =\varphi _4(x,y).\) A numerical computation shows that all real and complex solutions of the system of equations \(\partial \varPhi /\partial x(x,y,u(x,y))=0\) and \(\partial \varPhi /\partial y(x,y,u(x,y))=0\) are the following:

$$\begin{aligned} \begin{aligned}&(x\approx \pm \, 7.3296,y\approx -\,3.9586),\quad (x\approx \pm \, 3.9891,y\approx -\, 2.2430),\\&(x\approx \pm \, 0.7783,y\approx -\,0.9349),\quad (x\approx \pm \, 2.4991,y\approx 1.1601),\\&(x\approx 4.0332\cdot 10^{14},y\approx 108.2175),\quad (x\approx \pm 2.6637\mathrm {i},y\approx 0.4482),\\&(x\approx 0.6713\mathrm {i},y\approx 0.5624),\quad (x\approx 56781.0229\mathrm {i},y\approx 10.8430),\\&(x\approx \pm 2.5455\cdot 10^6\mathrm {i},y\approx 16.3752). \end{aligned} \end{aligned}$$

Therefore, \(\varPhi\) has no critical point.

Summarising from parts A to C (9) follows.

To see that (9) is sharp consider the function \(f_0\in \mathcal A\) given by (10) which belongs \(\mathcal P',\) with \(a_2=a_3=a_5=0\) and \(a_4=-1/2,\) which completes the proof. \(\square\)