1 Introduction

It is well known that fixed point theorems have been widely applied to many areas such as mechanics, physics, transportation, control, economics, and optimization. Many important research results can be found in the literature in the past years (see [1–4], etc. and the references therein). In 1976, Kellogg et al. (see [5]) gave a constructive proof of the Brouwer fixed point theorem and hence presented a homotopy method for computing the fixed points of a twice continuously differentiable self-mapping \(\Phi(x)\). From then on, this method has become a powerful tool in dealing with fixed point problems (see [6–10], etc. and the references therein). In general, these results in the literature require certain convexity assumptions, and it is difficult to reduce or remove these assumptions. However, the general Brouwer fixed point theorem does not require the convexity of the subsets in \(R^{n}\), certainly it is also very interesting and important to give a constructive proof of it and hence solve fixed point problems numerically in general nonconvex subsets. In 1996, Yu and Lin [11] proposed a homotopy interior path following method to complete this work on a class of nonconvex subset satisfying the normal cone condition, which is a generalization of the convexity. In [12], by introducing \(C^{2}\) mappings \(\alpha(x)=(\alpha_{1}(x),\ldots,\alpha_{m}(x))\in R^{n \times m}\) and \(\beta(x)=(\beta_{1}(x),\ldots,\beta_{l}(x))\in R^{n \times l}\), we further extended the results in [11] to more general nonconvex sets with both inequality and equality constraint functions. More recent related work can be seen in [13–15]. Set \(X=\{x\in R^{n}: g_{i}(x)\leq 0, i=1,\ldots,m, h_{j}(x)= 0, j=1,\ldots,l\}\), \(X^{0}=\{x\in R^{n}: g_{i}(x)<0, i=1,\ldots,m, h_{j}(x)= 0, j=1,\ldots,l\}\), \(R_{+}^{m}=\{x \in R^{m}: x \geq0\}\), \(R_{++}^{m}=\{x \in R^{m}: x > 0\}\), and \(B(x)=\{i\in\{1,\ldots,m\}: g_{i}(x)=0\}\). Now we state the main result in [12] as follows:

Theorem 1.1

Suppose that all \(g_{i}(x)\), \(i=1,\ldots, m\), and \(h_{j}(x)\), \(j=1, \ldots, l\), are \(C^{3}\) functions and:

(A1):

\(X^{0}\) is nonempty and X is bounded;

(A2):

for any \(x\in X\), if

$$\sum_{i\in B(x)} \bigl(y_{i} \nabla g_{i}(x)+u_{i} \alpha_{i}(x)\bigr)+\sum _{j=1}^{l}z_{j} \beta_{j}(x)=0, \quad y_{i}\geq0, u_{i}\geq 0, z_{j}\in R^{1}, $$

then \(y_{i}=0\), \(u_{i}=0\), \(\forall i\in B(x)\), and \(z_{j}=0\), \(j=1,\ldots,l\);

(A3):

(the weak normal cone condition of X) for any \(x\in X\), we have

$$\biggl\{ x+\sum_{i\in B(x)} u_{i} \alpha_{i}(x) +\beta(x)z: u_{i}\geq 0, i\in B(x),\textit{and }z\in R^{l}\biggr\} \cap X=\{x\}; $$
(A4):

for any \(x\in X\), \(\nabla h(x)\) is of full column rank and \(\nabla h(x)^{T}\beta(x)\) is nonsingular.

Then for any \(C^{2}\) mapping \(\Phi(x)\): \(R^{n}\rightarrow R^{n}\) satisfying \(\Phi(X)\subset X\) and for almost all \((x^{(0)}, y^{(0)}, z^{(0)})\in X^{0}\times R_{++}^{m}\times R^{l}\), there exists a \(C^{1}\) curve \((w(s), \mu(s))\) of dimension 1 of the homotopy

$$\begin{aligned} H_{1}\bigl(P, P^{(0)}, \mu\bigr) = \left ( \begin{array}{@{}c@{}} (1-\mu)(x-F(x)+\mu\nabla g(x)y + \alpha(x)y^{2})+\beta(x)z+ \mu(x-x^{(0)})\\ h(x)\\ Yg(x)-\mu Y^{(0)}g(x^{(0)}) \end{array} \right )=0 \end{aligned}$$
(1)

such that the limit set \(T \subset X^{0}\times R_{+}^{m}\times R^{l} \times\{0\}\) is nonempty, and the x-component of any point in T is a fixed point of \(F(x)\) in X, where \(P=(x, y, z)\in R^{n+m+l}\), \(P^{(0)}=(x^{(0)}, y^{(0)}, z^{(0)} )\in R^{n}\times R_{++}^{m} \times R^{l}\), \(\alpha(x)=(\alpha_{1} (x),\ldots,\alpha_{m} (x))\in R^{n\times m}\), \(y^{2}=(y_{1}^{2},\ldots,y_{m}^{2})^{T}\in R^{m}\), \(g(x)=(g_{1}(x),\ldots,g_{m}(x))^{T}\), \(h(x)=(h_{1}(x),\ldots,h_{m}(x))^{T}\), \(\nabla g(x)=(\nabla g_{1}(x),\ldots,\nabla g_{m}(x)) \in R^{n\times m}\), \(Y=\operatorname{diag}(y)\in R^{m\times m}\), \(Y^{(0)}=\operatorname{diag}(y^{(0)})\in R^{m\times m}\), and \(\mu\in(0, 1]\).

In [12], the global convergence of the algorithm is obtained under the assumption that the subset is bounded. To remove the boundednesss assumption, we present a set of unbounded conditions, under which, we are able to solve fixed point problems in a class of unbounded nonconvex set. Section 2 is the main part, which exhibits a convergence proof by the interior path following method.

2 Convergence analysis

To enlarge the scope of choice of initial points, similar to the way in [16], in this section, we still apply proper perturbations to the constrained functions \(g_{i}(x)\), \(i=1, \ldots, m\), and introduce the parameter

$$\gamma_{i}= \left \{ \begin{array}{@{}l@{\quad}l} 1, & g_{i}(x^{(0)})\geq0,\\ 0, & g_{i}(x^{(0)})<0, \end{array} \quad i=1,\ldots,m, \right . $$

then let

$$\begin{aligned}[b] & X(\mu)=\bigl\{ x: g_{i}(x)-\mu\gamma_{i} \bigl(g_{i}\bigl(x^{(0)}\bigr)+1\bigr) \leq0, i=1,\ldots ,m, h(x)=0\bigr\} , \\ & X^{0}(\mu)=\bigl\{ x: g_{i}(x)-\mu\gamma_{i} \bigl(g_{i}\bigl(x^{(0)}\bigr)+1\bigr) < 0, i=1,\ldots ,m, h(x)=0\bigr\} , \\ &\partial X(\mu)=X(\mu)\backslash X^{0}(\mu),\qquad I(x,\mu)=\bigl\{ i : g_{i}(x)-\mu\gamma_{i} \bigl(g_{i} \bigl(x^{(0)}\bigr)+1\bigr) = 0, i=1,\ldots,m\bigr\} . \end{aligned} $$

Now we construct the homotopy equation as follows:

$$\begin{aligned} H\bigl(P, P^{(0)}, \mu\bigr) = \left ( \begin{array}{@{}c@{}} (1-\mu)(x-\Phi(x)+\mu\nabla g(x)y + \alpha(x)y^{2})+\beta(x)z+ \mu(x-x^{(0)})\\ h(x)\\ Y(g(x)-\mu\Upsilon(g(x^{(0)})+e))-\mu Y^{(0)}(g(x^{(0)})-\Upsilon(g(x^{(0)})+e)) \end{array} \right )=0, \end{aligned}$$
(2)

where \(e=(1,\ldots,1)^{T} \in R^{m}\) and \(\Upsilon=\operatorname{diag}(\gamma_{1},\ldots,\gamma_{m})\).

This section is devoted to solving fixed point problems in unbounded sets. To this end, we introduce the concept of infinite solutions in [17]. The fixed point problem is said to have a solution at infinity, if the sequences \(\{x^{(k)}\}\) satisfy the following conditions:

  1. (i)

    \(\{{x}^{(k)}\} \subset X(\mu)\);

  2. (ii)

    when \(k \rightarrow\infty\), \(\|x^{(k)}\| \rightarrow\infty\);

  3. (iii)

    for any given \(x \in X(\mu)\), there exist \(y^{(k)} \in R_{+}^{m}\) and \(z^{(k)} \in R^{l}\) such that

    $$\lim_{k \rightarrow\infty} \bigl(x-x^{(k)}\bigr)^{T} \bigl(\bigl(x^{(k)}-F\bigl(x^{(k)}\bigr) + \alpha\bigl(x^{(k)} \bigr) \bigl(y^{(k)}\bigr)^{2}\bigr)+\beta \bigl(x^{(k)}\bigr)z^{(k)}\bigr)\geq0. $$

Then we replace the original assumption (A1) by the following unboundedness assumption:

(\(\mathrm{A}_{1}^{\prime}\)):

\(X^{0}(\mu)\) is nonempty; fixed point problems have no infinite solutions and for any given \(\eta\in X(\mu)\),

$$(x-\eta)^{T} \nabla g(x)\geq \bigl(g(x)-\mu\Upsilon\bigl(g \bigl(x^{(0)}\bigr)+e\bigr)\bigr)^{T} -\bigl(g(\eta)-\mu \Upsilon \bigl(g\bigl(x^{(0)}\bigr)+e\bigr)\bigr)^{T} $$

and

$$(x-\eta)^{T} \beta(x)\geq h(x)^{T} -h(\eta)^{T}. $$

The corresponding assumptions to assumptions (A2)-(A4) are needed.

(\(\mathrm{A}_{2}^{\prime}\)):

For any \(x\in X(\mu)\), if

$$\sum_{i\in I(x, \mu)} \bigl(y_{i} \nabla g_{i}(x)+u_{i} \alpha_{i}(x)\bigr)+\sum _{j=1}^{l}z_{j} \beta_{j}(x)=0, \quad y_{i}\geq0, u_{i}\geq 0, z_{j}\in R^{1}, $$

then \(y_{i}=0\), \(u_{i}=0\), \(\forall i\in I(x, \mu)\), and \(z_{j}=0\), \(j=1,\ldots,l\);

(\(\mathrm{A}_{3}^{\prime}\)):

(the weak normal cone condition of \(X(1)\)) for any \(x\in X(1)\), we have

$$\biggl\{ x+\sum_{i\in I(x, 1)} u_{i} \alpha_{i}(x) +\beta(x)z: u_{i}\geq 0, i\in B(x),\mbox{and }z\in R^{l}\biggr\} \cap X(1)=\{x\}; $$
(\(\mathrm{A}_{4}^{\prime}\)):

for any \(x\in X(\mu)\), \(\nabla h(x)\) is of full column rank and \(\nabla h(x)^{T}\beta(x)\) is nonsingular.

For a given \(P^{(0)}\), rewrite \(H(P, P^{(0)}, \mu)\) as \(H_{P^{(0)}}(P, \mu)\). The zero-point set of \(H_{P^{0}}\) is

$$H_{P^{(0)}}^{-1}(0)=\bigl\{ (P, \mu)\in X(\mu)\times R_{+}^{m}\times R^{l} \times(0, 1]: H_{P^{(0)}}(P, \mu) = 0\bigr\} . $$

The inverse image theorem tells us that, if 0 is a regular value of the map \(H_{P^{(0)}}\), then \(H_{P^{(0)}}^{-1}(0)\) consists of some smooth curves. The regularity of \(H_{P^{(0)}}\) can be obtained by the following lemma.

Lemma 2.1

(Parameterized Sard theorem)

Let \(V\subset R^{n}\), \(U\subset R^{m}\) be open sets, and \(\Phi: V\times U\rightarrow R^{k}\) a \(C^{r}\) map, where \(r>\max\{0, m-k\}\). If \(0\in R^{k}\) is a regular value of Φ, then for almost all \(a\in V\), 0 is a regular value of \(\Phi_{a}\equiv\Phi(a, \cdot)\).

Lemma 2.2

Let H be defined as in (2), let \(g_{i}(x)\), \(i=1,\ldots, m\), and \(h_{j}(x)\), \(j=1, \ldots, l\), be \(C^{3}\) functions, let assumptions (\(\mathrm{A}_{1}^{\prime}\))-(\(\mathrm{A}_{4}^{\prime}\)) hold, and let \(\alpha_{i}(x)\), \(i=1,\ldots, m\), and \(\beta_{j}(x)\), \(j=1, \ldots, l\), be \(C^{2}\) functions. Then, for almost all \(P^{(0)}\), the projection of the smooth curve \(\Gamma_{P^{(0)}}\) onto the x-plane is bounded.

Proof

If not, then there exists a sequence of points \(\{(x^{(k)}, y^{(k)}, z^{(k)}, \mu_{k})\}_{k=1}^{\infty}\) such that \(\|x^{(k)}\|\rightarrow\infty\) as \(k\rightarrow\infty\).

It is easy to show that the following inequality holds:

$$\begin{aligned} \|x-\eta\|^{2}-\|x^{0}-\eta\|^{2}\leq2(x- \eta)^{T}\bigl(x-x^{0}\bigr). \end{aligned}$$
(3)

Then we have

$$\begin{aligned}& \begin{aligned}[b] &(1-\mu_{k}) \bigl(x^{(k)}-F \bigl(x^{(k)}\bigr)+\mu_{k}\nabla g\bigl(x^{(k)} \bigr)y^{(k)} + \alpha\bigl(x^{(k)}\bigr) \bigl(y^{(k)} \bigr)^{2}\bigr) \\ &\quad{}+\beta\bigl(x^{(k)}\bigr)z^{(k)}+ \mu_{k} \bigl(x^{(k)}-x^{(0)}\bigr)=0, \end{aligned} \end{aligned}$$
(4)
$$\begin{aligned}& h\bigl(x^{(k)}\bigr)=0, \end{aligned}$$
(5)
$$\begin{aligned}& Y\bigl(g(x)-\mu\Upsilon\bigl(g\bigl(x^{(0)}\bigr)+e\bigr)\bigr)-\mu Y^{(0)}\bigl(g\bigl(x^{(0)}\bigr)-\Upsilon \bigl(g \bigl(x^{(0)}\bigr)+e\bigr)\bigr). \end{aligned}$$
(6)

Multiplying (4) by \((x^{(k)}-\eta)^{T}\), we get

$$\begin{aligned} &(1-\mu_{k}) \bigl(x^{(k)}-\eta \bigr)^{T}\bigl(x^{(k)}-F\bigl(x^{(k)}\bigr)+ \mu_{k}\nabla g\bigl(x^{(k)}\bigr)y^{(k)} + \alpha \bigl(x^{(k)}\bigr) \bigl(y^{(k)}\bigr)^{2}\bigr) \\ &\quad{}+\bigl(x^{(k)}-\eta\bigr)^{T}\beta \bigl(x^{(k)}\bigr)z^{(k)}+ \mu_{k} \bigl(x^{(k)}-\eta\bigr)^{T}\bigl(x^{(k)}-x^{(0)} \bigr)=0, \end{aligned}$$
(7)

i.e.,

$$\begin{aligned} &\mu_{k}\bigl(x^{(k)}-\eta\bigr)^{T} \bigl(x^{(k)}-x^{0}\bigr) \\ &\quad=-\bigl(x^{(k)}-\eta\bigr)^{T}\beta \bigl(x^{(k)}\bigr)z^{(k)} \\ &\qquad{} -(1-\mu_{k}) \bigl(x^{(k)}-\eta \bigr)^{T}\bigl(x^{(k)}-F\bigl(x^{(k)}\bigr)+ \mu_{k}\nabla g\bigl(x^{(k)}\bigr)y^{(k)} + \alpha \bigl(x^{(k)}\bigr) \bigl(y^{(k)}\bigr)^{2}\bigr). \end{aligned}$$
(8)

So

$$\begin{aligned} &\mu_{k}\bigl(\bigl\| x^{(k)}-\eta\bigr\| ^{2}- \bigl\| x^{0}-\eta\bigr\| ^{2}\bigr) \\ &\quad\leq 2\mu_{k}\bigl(x^{(k)}-\eta\bigr)^{T} \bigl(x^{(k)}-x^{0}\bigr) \\ &\quad=-2(1-\mu_{k}) \bigl(x^{(k)}-\eta\bigr)^{T} \bigl(x^{(k)}-F\bigl(x^{(k)}\bigr)+\alpha \bigl(x^{(k)} \bigr) \bigl(y^{(k)}\bigr)^{2}\bigr) \\ &\qquad{} -2(1-\mu_{k})\mu_{k}\bigl(x^{(k)}-\eta \bigr)^{T}\nabla g\bigl(x^{(k)}\bigr)y^{(k)} -2 \bigl(x^{(k)}-\eta\bigr)^{T}\beta\bigl(x^{(k)} \bigr)z^{(k)} \\ &\quad=-2(1-\mu_{k}) \bigl(x^{(k)}-\eta\bigr)^{T} \bigl(x^{(k)}-F\bigl(x^{(k)}\bigr)+\alpha \bigl(x^{(k)} \bigr) \bigl(y^{(k)}\bigr)^{2}+\beta\bigl(x^{(k)} \bigr)z^{(k)}\bigr) \\ &\qquad{} -2(1-\mu_{k})\mu_{k}\bigl(x^{(k)}-\eta \bigr)^{T}\nabla g\bigl(x^{(k)}\bigr)y^{(k)} -2 \mu_{k} \bigl(x^{(k)}-\eta\bigr)^{T}\beta \bigl(x^{(k)}\bigr)z^{(k)} \\ &\quad\leq-2(1-\mu_{k}) \bigl(x^{(k)}-\eta \bigr)^{T}\bigl(x^{(k)}-F\bigl(x^{(k)}\bigr)+\alpha \bigl(x^{(k)}\bigr) \bigl(y^{(k)}\bigr)^{2}+\beta \bigl(x^{(k)}\bigr)z^{(k)}\bigr) \\ &\qquad{}+2(1-\mu_{k})\mu_{k}\bigl(g(\eta)-\mu\Upsilon \bigl(g\bigl(x^{(0)}\bigr)+e\bigr)\bigr)^{T}y^{(k)}+2 \mu_{k}\bigl(h(\eta)^{T}y^{(k)}-h(x)^{T} \bigr)z^{(k)} \\ &\qquad{}-2(1-\mu_{k})\mu_{k}\bigl(g(x)-\mu\Upsilon \bigl(g \bigl(x^{(0)}\bigr)+e\bigr)\bigr)^{T}y^{(k)} \\ &\quad\leq-2(1-\mu_{k}) \bigl(x^{(k)}-\eta \bigr)^{T}\bigl(x^{(k)}-F\bigl(x^{(k)}\bigr)+\alpha \bigl(x^{(k)}\bigr) \bigl(y^{(k)}\bigr)^{2}+\beta \bigl(x^{(k)}\bigr)z^{(k)}\bigr) \\ &\qquad{}-2(1-\mu_{k})\mu_{k}^{2} \bigl(g \bigl(x^{(0)}\bigr)-\Upsilon\bigl(g\bigl(x^{(0)}\bigr)+e\bigr) \bigr)^{T} y^{(0)}. \end{aligned}$$
(9)

By (9), we have

$$\begin{aligned} &\bigl(\eta-x^{(k)}\bigr)^{T} \bigl(x^{(k)}-F\bigl(x^{(k)}\bigr) +\alpha\bigl(x^{(k)} \bigr) \bigl(y^{(k)}\bigr)^{2}+\beta\bigl(x^{(k)} \bigr)z^{(k)}\bigr) \\ &\quad \geq \frac{\mu_{k}}{2(1-\mu_{k})}\bigl(\bigl\| x^{(k)}-\eta\bigr\| ^{2}- \bigl\| x^{0}-\eta\bigr\| ^{2}\bigr)+\mu_{k}^{2} \bigl(g\bigl(x^{(0)}\bigr)-\Upsilon\bigl(g\bigl(x^{(0)}\bigr)+e \bigr)\bigr)^{T} y^{(0)}. \end{aligned}$$
(10)

When \(\|x^{(k)}\|\rightarrow\infty\), by (10), we have

$$\begin{aligned} & \lim_{k \rightarrow\infty} \bigl(\eta-x^{(k)} \bigr)^{T}\bigl(x^{(k)}-F\bigl(x^{(k)}\bigr) +\alpha \bigl(x^{(k)}\bigr) \bigl(y^{(k)}\bigr)^{2}+\beta \bigl(x^{(k)}\bigr)z^{(k)}\bigr) \\ &\quad \geq\lim_{k \rightarrow\infty}\frac{\mu_{k}}{2(1-\mu_{k})}\bigl( \bigl\| x^{(k)}-\eta\bigr\| ^{2}-\bigl\| x^{0}-\eta\bigr\| ^{2} \bigr) +\mu_{k}^{2} \bigl(g\bigl(x^{(0)}\bigr)- \Upsilon\bigl(g\bigl(x^{(0)}\bigr)+e\bigr)\bigr)^{T} y^{(0)} \\ &\quad\geq0, \end{aligned}$$
(11)

which contradicts assumption (\(\mathrm{A}_{1}^{\prime}\)). □

With the preparation of the previous lemmas, we can prove the following main theorem on the existence and boundedness of a smooth path from a given point \(x^{(0)}\) in \(R^{n}\) to a fixed point. This proof implies the global convergence of the path following algorithm.

Theorem 2.1

Let H be defined as in (2), let \(g_{i}(x)\), \(i=1,\ldots, m\), and \(h_{j}(x)\), \(j=1, \ldots, l\), be \(C^{3}\) functions, let assumptions (\(\mathrm{A}_{1}^{\prime}\))-(\(\mathrm{A}_{4}^{\prime}\)) hold, and let \(\alpha_{i}(x)\), \(i=1,\ldots, m\), and \(\beta_{j}(x)\), \(j=1, \ldots, l\), be \(C^{2}\) functions. Then for any \(C^{2}\) mapping \(F(x): R^{n}\rightarrow R^{n}\) satisfying \(F(X)\subset X\):

  1. (1)

    (existence of the fixed point) \(F(x)\) has a fixed point in X;

  2. (2)

    for almost all \(P^{(0)} \in R^{n}\times R_{++}^{m}\times R^{l}\), there exists a \(C^{1}\) curve \((P(s), \mu(s))\) of dimension 1 such that

    $$H\bigl(P(s), P^{(0)}, \mu(s)\bigr)=0, \qquad \bigl(P(0), \mu(0)\bigr)= \bigl(P^{(0)}, 1\bigr). $$

And when \(\mu(s)\rightarrow0\), \(P(s)\) tends to a point \(P^{*}=(x^{*}, y^{*}, z^{*})\). In particular, the component \(x^{*}\) of \(P^{*}\) is a fixed point of \(F(x)\) in X.

Proof

Denoting the Jacobi matrix of \(H(P, P^{(0)}, \mu)\) by \(DH(P, P^{(0)}, \mu)\), \(\forall(P,\mu)\in R^{n+m+l}\times(0,1]\), we obtain

$$\begin{aligned} &\frac{\partial H(P, P^{(0)}, \mu)}{\partial (x,x^{(0)},y^{(0)})} \\ &\quad=\left ( \begin{array}{@{}c@{\quad}c@{\quad}c} A & -\mu I & 0\\ \nabla h(x)^{T} & 0 & 0\\ Y\nabla g(x)^{T} & -\mu Y B-\mu Y^{(0)}(\nabla g(x^{(0)})^{T}-B) & -\mu(G(x^{(0)})-\Upsilon(G(x^{(0)})+I)) \end{array} \right ), \end{aligned}$$

where

$$\begin{aligned} & A=(1-\mu) \Biggl(I-\nabla F(x)+\mu\sum_{i=1}^{m} \nabla^{2} g_{i}(x)y_{i} + \sum _{i=1}^{m}\nabla\alpha_{i}(x)y_{i}^{2} \Biggr)+\sum_{j=1}^{l}\nabla \beta_{j}(x)z_{j}+ \mu I, \\ & B=\Upsilon\nabla g\bigl(x^{(0)}\bigr)^{T},\qquad G \bigl(x^{(0)}\bigr)=\operatorname{diag}\bigl(g\bigl(x^{(0)} \bigr)\bigr). \end{aligned}$$

Since \(G(x^{(0)})-\Upsilon(G(x^{(0)})+I)< 0\) and \(\nabla h(x)^{T}\) is a matrix of full row rank, by the parameterized Sard theorem and the inverse image theorem, \(H_{P^{(0)}}^{-1}(0)\) consists of some smooth curves. Since \(H_{P^{(0)}}(P^{(0)}, 1)=0\), then there exists a \(C^{1}\) curve \((P(s), \mu(s))\) (denoted by \(\Gamma_{P^{(0)}}\)) of dimension 1 such that

$$H\bigl(P(s), P^{(0)}, \mu(s)\bigr)=0, \qquad \bigl(P(0), \mu(0)\bigr)= \bigl(P^{(0)}, 1\bigr). $$

Since

$$\frac{\partial H_{P^{(0)}}(P, \mu)}{\partial P}= \left ( \begin{array}{@{}c@{\quad}c@{\quad}c@{}} C & D & \beta(x)\\ \nabla h(x)^{T} & 0 & 0\\ Y \nabla g(x)^{T} &G(x)-\mu\Upsilon(G(x^{(0)})+I) & 0 \end{array} \right ), $$

where

$$\begin{aligned} & C=(1-\mu) \bigl(I- \nabla\Phi(x)+\mu\nabla^{2} g(x)y + \nabla \alpha(x)y^{2}\bigr)+\nabla\beta(x)z+ \mu I, \\ & D=(1-\mu)\mu\nabla g(x) + 2(1-\mu)\alpha(x)y. \end{aligned}$$

Then it is easy to show that \(\partial H_{P^{(0)}}(P^{(0)}, 1)/\partial P\) is nonsingular. This fact implies that \(\Gamma_{P^{(0)}}\) is diffeomorphic to a unit interval.

Let \((P^{*}, \mu^{*})\) be a limit point of \(\Gamma_{P^{(0)}}\), then the following cases may be possible:

  1. (a)

    \((P^{*}, \mu^{*})=(x^{*}, y^{*}, z^{*}, \mu^{*})\in X(\mu^{*})\times R_{+}^{m}\times R^{l}\times\{0\}\),

  2. (b)

    \((w^{*}, \mu^{*})=(x^{*}, y^{*}, z^{*}, \mu^{*})\in X^{0}(1)\times R_{++}^{m}\times R^{l}\times\{1\}\),

  3. (c)

    \((P^{*}, \mu^{*})=(x^{*}, y^{*}, z^{*}, \mu^{*})\in \partial(X(\mu^{*})\times R_{+}^{m}\times R^{l})\times(0, 1]\).

Note that

$$H\bigl(P, P^{(0)}, 1\bigr)= \left ( \begin{array}{@{}c@{}} \beta(x)z+ x-x^{(0)}\\ h(x)\\ Y(g(x)-\Upsilon(g(x^{(0)})+e))- Y^{(0)}(g(x^{(0)})-\Upsilon(g(x^{(0)})+e)) \end{array} \right ). $$

Then the equation \(H(P, P^{(0)}, 1)=0\) has a unique solution \((P^{(0)}, 1)\) in \(X(1) \times R_{++}^{m}\times R^{l}\times\{1\}\). This fact implies case (b) is not possible.

By the fact that \(X(\mu)\) and \((0, 1]\) being bounded, assumption (\(\mathrm{A}_{2}^{\prime}\)), and the first and third equations in (2), we see that the component z of \(\Gamma_{P^{(0)}}\) is bounded.

If case (c) holds, since \(X(\mu)\) and \((0,1]\) are bounded, hence there exists a subsequence of points (denoted also by \(\{(P^{(k)},\mu_{k})\}\)) such that \(x^{(k)}\rightarrow x^{*}\), \(y^{(k)}\rightarrow\infty\), \(z^{(k)}\rightarrow z^{*}\), and \(\mu_{k}\rightarrow\mu^{*}\) as \(k\rightarrow\infty\).

If \(\mu^{*}=1\), from the first equation in (2),

$$ \begin{aligned}[b] & \sum_{i\in I(x^{*}, 1)}(1-\mu_{k}) \bigl(\mu_{k}\nabla g_{i}\bigl(x^{(k)} \bigr)y_{i}^{(k)}+\alpha_{i}\bigl(x^{(k)} \bigr) \bigl(y_{i}^{(k)}\bigr)^{2}\bigr)+\beta \bigl(x^{(k)}\bigr)z^{(k)} +x^{(k)}-x^{(0)}\\ &\quad=-(1-\mu_{k}) \biggl(\sum_{i\notin I(x^{*}, 1)} \bigl(\mu_{k}\nabla g_{i}\bigl(x^{(k)} \bigr)y_{i}^{(k)}+\alpha _{i}\bigl(x^{(k)} \bigr) \bigl(y_{i}^{(k)}\bigr)^{2} \bigr)+x^{(0)}-F\bigl(x^{(k)}\bigr)\biggr). \end{aligned} $$
(12)

By the fact that \(y_{i}^{(k)}\) are bounded for \(i\notin I(x^{*}, 1)\) and Lemma 2.1, when \(k\rightarrow\infty\), we obtain

$$\begin{aligned} \lim_{k\rightarrow\infty}\biggl(\sum _{i\in I(x^{*}, 1)}(1-\mu_{k}) \bigl(\mu_{k}\nabla g_{i}\bigl(x^{(k)}\bigr)y_{i}^{(k)}+ \alpha_{i}\bigl(x^{(k)}\bigr) \bigl(y_{i}^{(k)} \bigr)^{2}\bigr) +\beta\bigl(x^{(k)}\bigr)z^{(k)}+x^{(k)} \biggr) =x^{(0)}. \end{aligned}$$
(13)

By assumptions (\(\mathrm{A}_{2}^{\prime}\)) and (13), we have

$$\begin{aligned} \lim_{k\rightarrow\infty}(1-\mu_{k}) \bigl(y_{i}^{(k)} \bigr)^{2}=\rho_{i}^{*} \quad \mbox{and}\quad \lim _{k\rightarrow\infty}(1-\mu_{k})y_{i}^{(k)}=0, \quad i\in I\bigl(x^{*}, 1\bigr), \end{aligned}$$
(14)

where \(\rho_{i}^{*}\geq0\). Therefore from (13) and (14), we get

$$\begin{aligned} x^{*}+\sum_{i\in I(x^{*}, 1)}\rho_{i}^{*} \alpha_{i}\bigl(x^{*}\bigr)+\beta\bigl(x^{*}\bigr)z^{*}=x^{(0)}, \end{aligned}$$
(15)

which contradicts assumption (\(\mathrm{A}_{3}^{\prime}\)).

If \(\mu^{*}<1\), when \(k\rightarrow\infty\), since \(X(\mu)\) and \(y_{i}^{(k)}\), \(i\notin I(x, \mu)\) are bounded, then the right-hand side of (12) is bounded. But by assumption (\(\mathrm{A}_{2}^{\prime}\)), if \(y_{i}^{(k)}\rightarrow\infty\), \(i\in I(x, \mu)\), then the left-hand side of (12) is infinite. This fact results in a contradiction.

By the above discussion, we obtain the result that case (a) is the unique possible case. Therefore \(P^{*}\) is a solution of the equation

$$ \begin{aligned} &x-F(x)+\alpha(x)y^{2}+\beta(x)z=0, \\ &h(x)=0, \\ &Yg(x)=0, \quad g(x)\leq0, \ y\geq0. \end{aligned} $$
(16)

By simple discussions, we conclude that \(x^{*}\) is a fixed point of \(F(x)\) in X. This completes the proof. □

For almost all \(P^{(0)}=(x^{(0)}, y^{(0)},0)\in X^{0}(1)\times R_{++}^{m}\times R^{l}\), by Theorem 2.1, the homotopy generates a \(C^{1}\) curve \(\Gamma_{P^{(0)}}\), and we get the following theorem.

Theorem 2.2

The homotopy path \(\Gamma_{P^{(0)}}\) is determined by the following initial value problem to the ordinary differential equation:

$$\begin{aligned} DH_{P^{(0)}}\bigl(P(s),\mu(s)\bigr)\left ( \begin{array}{@{}c@{}} \dot{P}(s)\\ \dot{\mu}(s) \end{array} \right )=0,\qquad \bigl(P(0), \mu(0)\bigr)=\bigl(P^{(0)}, 1 \bigr), \end{aligned}$$
(17)

where s is the arc length of the curve \(\Gamma_{P^{(0)}}\).

As for how to trace numerically the homotopy path, there have been many predictor-corrector algorithms; see [7], etc. for references. Hence we omit them.