1 Introduction and Notation

Let G be a group. We use standard commutator notation: for elements \(g_{1}, \ldots , g_{c}\) of G, we write \([g_{1}, g_{2}]= g_{1}^{-1}g_{2}^{-1}g_{1}g_{2}\) and, for \(c\ge 3\), \([g_{1}, \ldots , g_{c}] = [[g_{1}, \ldots , g_{c-1}], g_{c}]\). For a non-negative integer d, we write \([x, ~_{d}y] = [x, y, \ldots , y]\), where \(x, y \in G\) and y is repeated d times (for \(d = 0\), we write \([x,~ _{0}y] = x\)). For a non-empty set X, \(\langle X\rangle \) denotes the group generated by X and for subgroups \(A_{1}, \ldots , A_{c}\) of G, with \(c \ge 1\), \(\langle A_{1}, \ldots , A_{c}\rangle \) denotes the subgroup of G generated by \(A_{1}, \ldots , A_{c}\). For non-empty subsets \(X_{1}, \ldots , X_{c}\) of G, we write \([X_{1}, X_{2}] = \langle [g_{1}, g_{2}]: g_{1} \in X_{1}, g_{2} \in X_{2}\rangle \) and, for \(c \ge 3\), \([X_{1}, \ldots , X_{c}] = [[X_{1}, \ldots , X_{c-1}], X_{c}]\). For a positive integer d, we write \([X_{1}, ~_{d}X_{2}] = [X_{1}, X_{2}, \ldots , X_{2}]\), where \(X_{2}\) is repeated d times. For a positive integer c, let \(\gamma _{c}(G)\) be the cth term of the lower central series of G. We point out that \(\gamma _{2}(G) = G^{\prime }\), that is, \( G^{\prime }\) is the derived subgroup of G, and \(G^{\prime \prime } =(G^{\prime })^{\prime }\). We write \({\textrm{Aut}}(G)\) for the automorphism group of G.

A variety of groups is the class of all groups satisfying each one of a given set of laws. For further information concerning varieties of groups, see [8]. Let \(F_{n}\) be the free group of finite rank n, with \(n \ge 2\), freely generated by the set \(\{f_{1}, \ldots , f_{n}\}\). Let V be a variety of groups and let \(V(F_{n})\) be the verbal subgroup of \(F_{n}\) corresponding to V. Thus, the group \(F_{n}(V) = F_{n}/V(F_{n})\) is a relatively free group of rank n in V. The natural mapping from \(F_{n}\) onto \(F_{n}(V)\) induces a group homomorphism, say \(\pi _{n,V}\), from \({\textrm{Aut}}(F_{n})\) into \({\textrm{Aut}}(F_{n}(V)\). The image of \(\pi _{n,V}\) is denoted by \(T_{F_{n}(V)}\) and the automorphisms of \(F_{n}(V)\) which belong to \(T_{F_{n}(V)}\) are called tame; otherwise they are called non-tame (or wild). Since, by [7], \({\textrm{Aut}}(F_{n})\) is finitely generated, \(T_{F_{n}(V)}\) is also finitely generated.

For a positive integer n, with \(n \ge 2\), let \(G_{n} = F_{n}/\gamma _{3}(F_{n}^{\prime })\), that is, \(G_{n}\) is a free nilpotent of class 2-by-abelian group of rank n. We write \(x_{i} = f_{i}\gamma _{3}(F_{n}^{\prime })\) for \(i = 1, \ldots , n\). Thus, the set \(\{x_{1}, \ldots , x_{n}\}\) is a free generating set of \(G_{n}\). For \(n = 4\), it has been proved in [6] that \({\textrm{Aut}}(G_{4})\) is generated by \(T_{G_{4}}\) and an explicitly given set of non-tame automorphisms of \(G_{4}\), and this result has been extended in [9] for all \(n \ge 4\). For positive integers n and k, we write \(G_{n,k} = F_{n}/\gamma _{3}(F_{n}^{\prime })[F^{\prime \prime },~_{k}F_{n}]\). Since \(G_{n,k} \cong G_{n}/[G_{n}^{\prime \prime }, ~_{k}G_{n}]\), we may assume, without loss of generality, that \(G_{n,k} = F_{n}/\gamma _{3}(F_{n}^{\prime })[F^{\prime \prime },~_{k}F_{n}] = G_{n}/[G_{n}^{\prime \prime }, ~_{k}G_{n}]\). Write \(y_{i,k} = x_{i}[G^{\prime \prime }_{n}, ~_{k}G_{n}]\). Thus, \(G_{n,k}\) is freely generated by the set \(\{y_{1,k}, \ldots , y_{n,k}\}\). For our convenience, throughout this paper, we write \(y_{i}\) instead of \(y_{i, k}\) for \(i = 1, \ldots , n\). We write \(T_{n,k}\) for the group \(T_{G_{n,k}}\) of the tame automorphisms of \(G_{n,k}\).

Our work is motivated by [6, Concluding remarks]. In [6], C. K. Gupta and F. Levin have found a generating set for \({\textrm{Aut}}(G_{4})\). Since the natural group epimorphism from \(G_{n}\) onto \(G_{n,k}\) induces a group epimorphism from \({\textrm{Aut}}(G_{n})\) onto \({\textrm{Aut}}(G_{n,k})\) for all \(n \ge 4\) (see Sect. 2), they have also found a generating set for \({\textrm{Aut}}(G_{4,2})\). Then, for \(n \ge 5\), they claim to have shown that \({\textrm{Aut}}(G_{n,2})\) is generated by \(T_{n,2}\) and one more automorphism. Since a generating set of \({\textrm{Aut}}(G_{n,2})\), with \(n \ge 5\), was not known in [6], the aforementioned claim, at the time it was made, does not seem to be right. Furthermore, based on their aforementioned result for \({\textrm{Aut}}(G_{n,2})\), with \(n \ge 5\), the authors claim that for \(n \ge k +3 \ge 5\), \({\textrm{Aut}}(F_{n})/[F_{n}^{\prime \prime }, ~_{k}F_{n}]\) is generated by the tame automorphisms and one more non-tame automorphism. In this generality, this claim does not seem to may be proved by the work in [6]. Since \(G_{n,1} = F_{n}/[F_{n}^{\prime \prime }, F_{n}]\) and \(G_{n,2} = F_{n}/[F_{n}^{\prime \prime }, ~_{2}F_{n}]\), in the spirit of the claim in [6, Concluding remarks], we are motivated to study \({\textrm{Aut}}(G_{n,k})\), with \(n \ge 4\) and \(k \ge 2\). Since a generating set of \({\textrm{Aut}}(G_{n})\), with \(n \ge 4\), is given by A. Papistas [9], we may prove that \({\textrm{Aut}}(G_{n,2})\) is generated by \(T_{n,2}\) and one more non-tame automorphism for all \(n \ge 4\) (see Proposition 4). Furthermore, for \(n \ge k+2 \ge 5\), we may prove that \({\textrm{Aut}}(G_{n,k})\) is generated by \(T_{n,k}\) and one more non-tame automorphism.

Theorem 1

For positive integers n and k, with \(n \ge k+2\ge 4\), let \(G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})[F^{\prime \prime }_{n},~_{k}F_{n}]\) be freely generated by the set \(\{y_{1}, \ldots , y_{n}\}\). Then, \({\textrm{Aut}}(G_{n,k}) = \langle T_{n,k}, {\overline{w}}\rangle \), where \({\overline{w}}\) is the non-tame automorphism of \(G_{n,k}\) satisfying the conditions \({\overline{w}}(y_{1}) = y_{1}[y_{1}, y_{2}, [y_{3}, y_{4}]]\) and \({\overline{w}}_{k}(y_{j}) = y_{j}\), \(j = 2, \ldots , n\).

Remark 1

 

  1. (1)

    The group \(G_{n,1} = F_{n}/[F^{\prime \prime }_{n}, F_{n}]\) is a free centre-by-metabelian group of rank n. For \(n \ge 4\), it has been proved in [10] that \({\textrm{Aut}}(G_{n,1})\) is generated by the tame automorphisms and one more automorphism which, by [5], is non-tame.

  2. (2)

    Let \(F_{\infty }\) be the free group of infinite rank and, for a positive integer k, let \(G_{\infty , k} = F_{\infty }/\gamma _{3}(F_{\infty }^{\prime })[F_{\infty }^{\prime \prime }, ~_{k}F_{\infty }]\). It follows from [4, Theorem 2] that every automorphism of \(G_{\infty , k}\) is tame.

2 Preliminaries

For positive integers c and n, with \(n \ge 2\), let \(V_{c}\) be the variety of all nilpotent of class c-by-abelian groups and let \(G_{n,V_{c}} = F_{n}/V_{c}(F_{n})\), that is, \(G_{n,V_{c}} = F_{n}/\gamma _{c+1}(F_{n}^{\prime })\). For \(c = 1\), we write \(M_{n}\) for \(G_{n,V_{1}}\), that is, \(M_{n} = F_{n}/F_{n}^{\prime \prime }\) is a free metabelian group of rank n. For the rest of the paper, we write \(G_{n}\) for \(G_{n,V_{2}}\) and \(x_{i} = f_{i}V_{2}(F_{n})\) for \(i = 1, \ldots , n\). Thus, \(G_{n}\) is a free nilpotent of class 2-by-abelian group freely generated by the set \(\{x_{1}, \ldots , x_{n}\}\). Furthermore, we write \(T_{n}\) for the group \(T_{G_{n,V_{2}}}\) of the tame automorphisms of \(G_{n}\). By results of [1, 3, 10], it is known that \({\textrm{Aut}}(G_{n})\) is not finitely generated for \(2 \le n \le 3\).

For positive integers n and c, with \(n \ge 4\) and \(c \ge 2\), let W be a variety of groups, with \(V_{1} \subseteq W \subseteq V_{c}\), and let \(W_{n}\) be a relatively free group of finite rank n in W. For the proof of the following result, see [3].

Lemma 1

For positive integers n and c, with \(n, c \ge 2\), let W be a variety of groups, with \(V_{1} \subseteq W \subseteq V_{c}\), and let \(W_{n}\) be a relatively free group of finite rank n in W. Then:

  1. (1)

    An endomorphism \(\phi \) of \(F_{n}\) induces an automorphism of \(W_{n}\) if and only if \(\phi \) induces an automorphism of the free metabelian group \(M_{n}\).

  2. (2)

    If every automorphism of \(M_{n}\) is induced by an automorphism of \(F_{n}\), then every automorphism of \(W_{n}\) is a product of a tame automorphism and an automorphism which induces the identity mapping on \(W_{n}/W^{\prime \prime }_{n}\).

Since, by [2], every automorphism of \(M_{n}\), with \(n \ge 4\), is induced by an automorphism of \(F_{n}\), the next result follows from Lemma 1.

Corollary 1

For positive integers n and k, with \(n \ge 4\), let \(M_{n}\) be a free metabelian group of rank n, let \(G_{n} = F_{n}/\gamma _{3}(F^{\prime }_{n})\) and let \(G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})\times [F^{\prime \prime }_{n},~_{k}F_{n}]\). Then:

  1. (1)

    An endomorphism of \(G_{n}\) is an automorphism of \(G_{n}\) if and only if it induces an automorphism of the free metabelian group \(M_{n}\).

  2. (2)

    An endomorphism of \(G_{n,k}\) is an automorphism of \(G_{n,k}\) if and only if it induces an automorphism of the free metabelian group \(M_{n}\).

  3. (3)

    The natural group epimorphism from \(G_{n}\) onto \(G_{n,k}\) induces a group epimorphism from \({\textrm{Aut}}(G_{n})\) onto \({\textrm{Aut}}(G_{n,k})\).

Let \(\psi _{1}\) and \(\psi _{2}\) be the endomorphisms of \(G_{n}\) satisfying the conditions \(\psi _{1}(x_{i}) = x_{i}u_{i}\), \(i = 1, \ldots , n\), for some \(u_{1}, \ldots , u_{n} \in G_{n}^{\prime \prime }\) and \(\psi _{2}(x_{i}) = x_{i}v_{i}\), \(i = 1, \ldots , n\), for some \(v_{1}, \ldots , v_{n} \in G_{n}^{\prime \prime }\). By Corollary 1 (2), \(\psi _{1}\) and \(\psi _{2}\) are automorphisms of \(G_{n}\). We point out that, by [9, section 2], \(\psi _{1}(w) = w\) for all \(w \in G_{n}^{\prime \prime }\) and \(\psi _{1}\psi _{2}(x_{i}) = x_{i}u_{i}v_{i}\), \(i = 1, \ldots , n\).

Let \(\phi _{1}\) and \(\phi _{2}\) be the endomorphisms of \(G_{n,k}\) satisfying the conditions \(\phi _{1}(y_{i}) = y_{i}u_{i}\), \(i = 1, \ldots , n\), for some \(u_{1}, \ldots , u_{n} \in G_{n,k}^{\prime \prime }\) and \(\phi _{2}(y_{i}) = y_{i}v_{i}\), \(i = 1, \ldots , n\), for some \(v_{1}, \ldots , v_{n} \in G_{n,k}^{\prime \prime }\). By Corollary 1 (3), \(\phi _{1}\) and \(\phi _{2}\) are automorphisms of \(G_{n,k}\). Since \(\gamma _{3}(G_{n,k}^{\prime }) = \{1\}\), it follows from the above observations that \(\phi _{1}(w) = w\) for all \(w \in G_{n,k}^{\prime \prime }\) and \(\phi _{1}\phi _{2}(y_{i}) = y_{i}u_{i}v_{i}\), \(i = 1, \ldots , n\). We point out that these facts will be used repeatedly in Sect. 4 without further comment.

3 The Automorphism Group of \(G_{n}\)

3.1 Identities in Nilpotent-By-Abelian Groups

For a group G and elements a and b in G, we write \(a^{b} = b^{-1}ab\). For positive integers k and l, with \(k \ge l\), we write \(\left( {\begin{array}{c}k\\ l\end{array}}\right) = \frac{k!}{l!(k-l)!}\).

We prove some identities in nilpotent-by-abelian groups, which we need in the following. For many other identities in nilpotent-by-abelian groups, see [9, Lemma 2.1].

Lemma 2

Let G be a group such that \(\gamma _{3}(G) = \{1\}\). Then:

  1. (1)

    For any \(g_{1}, \ldots , g_{r} \in G\), with \(r \ge 3\), and \(u \in G^{\prime \prime }\),

    $$\begin{aligned} {[}u, g_{1}g_{2}, g_{3}, \ldots , g_{r}] = [u, g_{2}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{2}, g_{3}, \ldots , g_{r}]. \end{aligned}$$
  2. (2)

    For any \(g_{1}, \ldots , g_{r} \in G\), with \(r \ge 2\), \(u \in G^{\prime \prime }\) and \(\pi \) a permutation of \(\{1, \ldots , r\}\),

    $$\begin{aligned} {[}u, g_{1}, \ldots , g_{r}] = [u, g_{\pi (1)}, \ldots , g_{\pi (r)}]. \end{aligned}$$
  3. (3)

    For any \(g \in G\) and \(u \in G^{\prime \prime }\), \([u, g^{m}] = \prod _{k=1}^{m}[u, ~_{k}g]^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }\) for all \(m \ge 1\).

Proof

 

  1. (1)

    We use induction on r. Let \(r = 3\). By the group commutator identities \([ab,c] = [a,c]^{b}[a,c] = [a,c][a,c,b][b,c]\) and \([a,bc] = [a,c][a,b]^{c} = [a,c][a,b][a,b,c]\) and since \(\gamma _{3}(G) = \{1\}\),

    $$\begin{aligned} \begin{aligned} {[}u, g_{1}g_{2}, g_{3}]&= [[u, g_{2}][u, g_{1}][u, g_{1}, g_{2}], g_{3}]\\&= [u, g_{2}, g_{3}]^{[u, g_{1}][u, g_{1}, g_{2}]}[u, g_{1}, g_{3}]^{[u, g_{1}, g_{2}]}[u, g_{1}, g_{2}, g_{3}]\\&= [u, g_{2}, g_{3}][u, g_{1}, g_{3}][u, g_{1}, g_{2}, g_{3}], \end{aligned} \end{aligned}$$

    that is, the result is true for \(r = 3\). Assume that the result is true for some \(r \ge 3\), that is, assume that

    $$\begin{aligned} {[}u, g_{1}g_{2}, g_{3}, \ldots , g_{r}] = [u, g_{2}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{2}, g_{3}, \ldots , g_{r}] \end{aligned}$$

    for some \(r \ge 3\). Then, by our inductive argument,

    $$\begin{aligned}{} & {} {[}u, g_{1}g_{2}, g_{3}, \ldots , g_{r}, g_{r+1}] \\{} & {} \quad = [[u, g_{2}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{2}, g_{3}, \ldots , g_{r}], g_{r+1}]. \end{aligned}$$

    Hence, by similar arguments as in the case \(r = 3\),

    $$\begin{aligned}{} & {} {[}u, g_{1}g_{2}, g_{3}, \ldots , g_{r+1}] \\{} & {} \quad = [u, g_{2}, g_{3}, \ldots , g_{r+1}][u, g_{1}, g_{3}, \ldots , g_{r+1}][u, g_{1}, g_{2}, g_{3}, \ldots , g_{r+1}] \end{aligned}$$

    and so, we obtain the required result.

  2. (2)

    By the Hall-Witt identity in the form \([a, b, c^{a}][c, a, b^{c}][b, c, a^{b}] = 1\), we have (for \(a = u\), \(b = g_{1}\) and \(c = g_{2}\)) \([u, g_{1}, g_{2}^{u}][g_{2}, u, g_{1}^{g_{2}}][g_{1}, g_{2}, u^{g_{1}}] = 1\). Since \(\gamma _{3}(G) = \{1\}\), we get

    $$\begin{aligned} {[}u, g_{1}, g_{2}^{u}]= & {} [u, g_{1}, g_{2}[g_{2}, u]] = [u, g_{1}, g_{2}], \\ {[}g_{2}, u, g_{1}^{g_{2}}]= & {} [g_{2}, u, g_{1}[g_{1},g_{2}]] = [g_{2}, u, g_{1}] = [[u, g_{2}]^{-1}, g_{1}] = [u, g_{2}, g_{1}]^{-1} \end{aligned}$$

    and

    $$\begin{aligned} {[}g_{1}, g_{2}, u^{g_{1}}] = 1. \end{aligned}$$

    Hence, it follows from the Hall-Witt identity that \([u, g_{1}, g_{2}] = [u, g_{2}, g_{1}]\) and the required result may be easily deduced.

  3. (3)

    We use induction on m. For \(m = 1\), the result is trivially true and so, we may assume that the result is true for some \(m \ge 1\). That is, we assume that for some \(m \ge 1\), \([u, g^{m}] = \prod _{k=1}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\) for all \(g \in G_{n}\) and \(u \in G_{n}^{\prime \prime }\). Since

    $$\begin{aligned}{}[u, g^{m+1}] = [u, g^{m}g] = [u, g][u, g^{m}]^{g}, \end{aligned}$$

    by our inductive argument we get

    $$\begin{aligned} \begin{aligned} {[}u, g^{m+1}]&= [u, g]\prod _{k=1}^{m}([u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }})^{g} = [u, g]\prod _{k=1}^{m}([u, ~_{k}g]^{g})^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\\&= [u, g]\prod _{k=1}^{m}([u, ~_{k}g][u, ~_{k+1}g])^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }} . \end{aligned} \end{aligned}$$

    Since \(\gamma _{3}(G^{\prime }) = \{1\}\), the elements \([u, ~_{k}g]\) and \([u, ~_{k+1}g]\) commute and hence, by the above equation,

    $$\begin{aligned} \begin{aligned} {[}u, g^{m+1}]&= [u, g]\prod _{k=1}^{m}\left( [u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}[u, ~_{k+1}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\right) \\ {}&= [u, g]\prod _{k=1}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\prod _{k=1}^{m}[u, ~_{k+1}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }} \\&= [u, g]\prod _{k=1}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\prod _{k=2}^{m+1}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k-1\end{array}}\right) }}\\&= [u, g][u,g]^{^{\left( {\begin{array}{c}m\\ 1\end{array}}\right) }}\prod _{k=2}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) +\left( {\begin{array}{c}m\\ k-1\end{array}}\right) }}[u, ~_{m+1}g]^{^{\left( {\begin{array}{c}m\\ m\end{array}}\right) }}. \end{aligned} \end{aligned}$$

    Since \(\left( {\begin{array}{c}m\\ k\end{array}}\right) +\left( {\begin{array}{c}m\\ k-1\end{array}}\right) = \left( {\begin{array}{c}m+1\\ k\end{array}}\right) \), we get

    $$\begin{aligned} \begin{aligned} {[}u, g^{m+1}]&= [u, g]^{m+1}\prod _{k=2}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m+1\\ k\end{array}}\right) }}[u, ~_{m+1}g] = \prod _{k=1}^{m+1}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m+1\\ k\end{array}}\right) }} \end{aligned} \end{aligned}$$

    and so, it follows that the result is true for all \(m \ge 1\).

\(\square \)

3.2 Generating Sets for \(\textrm{Aut}(G_{n})\)

For non-negative integers m and k, let \(a_{m,k}\) and \(\beta _{m}\) be the endomorphisms of \(G_{n}\) satisfying the conditions

$$\begin{aligned} \begin{aligned} a_{(m,k)}(x_{1})&= x_{1}[[x_{1}, x_{2}]^{x_{1}^{m}x_{2}^{k}}, [x_{3}, x_{4}]], a_{(m,k)}(x_{j}) = x_{j}, j = 2, \ldots , n,\\ \beta _{m}(x_{1})&= x_{1}[[x_{1}, x_{2}], [x_{3},x_{4}],x_{2}^{m}], \beta _{m}(x_{j}) = x_{j}, j = 2, \ldots , n. \end{aligned} \end{aligned}$$

We point out that, by Corollary 1 (1), \(a_{(m,k)}\) and \(\beta _{m}\) are automorphisms of \(G_{n}\) for all \(m, k \ge 0\).

It has been proved in [6, Theorem 1] that \({\textrm{Aut}}(G_{4}) = \langle T_{4}, a_{(m,0)}: m \ge 0\rangle \). Furthermore, it has been proved in [6, Lemma 3 (i)] that \(\langle T_{4}, a_{(m,0)}: m \ge 0\rangle = \langle T_{4}, a_{(0,m)}\rangle \) and in [6, Corollary] that \(\langle T_{4}, a_{(0,m)}: m \ge 0\rangle = \langle T_{4}, a_{(0,0)}, \beta _{m}: m \ge 1\rangle \). Since the arguments given in [6, proof of Lemma 3 (i)] and in [6, proof of Corollary] are valid for all \(n \ge 4\), we get the following result.

Corollary 2

For a positive integer n, with \(n \ge 4\), let \(G_{n} = F_{n}/\gamma _{3}(F_{n}^{\prime })\). Then,

$$\begin{aligned} \langle T_{n}, a_{(m,0)} : m \ge 0\rangle = \langle T_{n}, a_{(0,m)} : m \ge 0\rangle = \langle T_{n}, a_{(0,0)}, \beta _{m} : m \ge 1\rangle . \end{aligned}$$

In [9], it has been given a generating set for \({\textrm{Aut}}(G_{n})\), independent upon n, for all \(n \ge 4\). In particular, the next result has been proved in [9, proof of Theorem 1.1].

Proposition 1

For a positive integer n, with \(n \ge 4\), let \(G_{n} = F_{n}/\gamma _{3}(F_{n}^{\prime })\). Then, \({\textrm{Aut}}(G_{n}) = \langle T_{n}, \alpha _{(m,0)}: m \ge 0\rangle \).

We give a different generating set for \({\textrm{Aut}}(G_{n})\), which we need for the proof of Theorem 1.

Proposition 2

For a positive integer n, with \(n \ge 4\), let \(G_{n} = F_{n}/\gamma _{3}(F_{n}^{\prime })\). Then, \({\textrm{Aut}}(G_{n}) = \langle T_{n}, w_{m}: m \ge 0\rangle \), where \(w_{m}\) is the automorphism of \(G_{n}\) satisfying the conditions \(w_{m}(x_{1}) = x_{1}[[x_{1}, x_{2}], [x_{3},x_{4}], ~_{m}x_{2}]\) and \(w_{m}(x_{j}) = x_{j}\), \(j = 2, \ldots , n\).

Proof

It follows from Proposition 1 and Corollary 2 that \({\textrm{Aut}}(G_{n}) = \langle T_{n}, a_{(0,0)}, \beta _{m}: m \ge 1\rangle \). We point out that, by Corollary 1 (1), \(w_{m}\) is an automorphism of \(G_{n}\) for all \(m \ge 0\). Hence, to complete the proof, it suffices to show that \(\alpha _{(0, 0)}, \beta _{k} \in \langle T_{n}, w_{m}: m \ge 0\rangle \) for all \(k \ge 1\). Observe that \(\alpha _{(0,0)} = w_{0} \in \langle T_{n}, w_{m}: m \ge 0\rangle \). Fix some \(k \ge 1\). Since, by Lemma 2 (3),

$$\begin{aligned} {[[}x_{1}, x_{2}], [x_{3},x_{4}],x_{2}^{k}] = \prod _{t=1}^{k}[[x_{1}, x_{2}], [x_{3},x_{4}], ~_{t}x_{2}]^{\left( {\begin{array}{c}k\\ t\end{array}}\right) } \end{aligned}$$

and for all \(m \ge 1\) the automorphisms \(w_{m}\) commute, it follows that

$$\begin{aligned} \begin{aligned} \beta _{k}(x_{1})&= x_{1}[[x_{1}, x_{2}], [x_{3},x_{4}],x_{2}^{k}] = x_{1}\prod _{t=1}^{k}[[x_{1}, x_{2}], [x_{3},x_{4}], ~_{t}x_{2}]^{\left( {\begin{array}{c}k\\ t\end{array}}\right) }\\ {}&= \prod _{t=1}^{k}w_{t}^{\left( {\begin{array}{c}k\\ t\end{array}}\right) }(x_{1}). \end{aligned} \end{aligned}$$

Furthermore, \(\beta _{k}(x_{j}) = \prod _{t=1}^{k}w_{t}^{\left( {\begin{array}{c}k\\ t\end{array}}\right) }(x_{j}) = x_{j}\) for all \(j = 2, \ldots , n\). Therefore, \(\beta _{k} = \prod _{t=1}^{k}w_{t}^{\left( {\begin{array}{c}k\\ t\end{array}}\right) }\). Hence, \(\beta _{k} \in \langle T_{n}, w_{m}: m \ge 0\rangle \) and so, we get the required result. \(\square \)

4 The Automorphism Group of \(G_{n,k}\)

4.1 Notation

For positive integers n and k, with \(n \ge 4\) and \(k \ge 2\), recall that \(G_{n,k} = F_{n}/\gamma _{3}(F_{n}^{\prime })[F^{\prime \prime },~_{k}F_{n}]\) is freely generated by the set \(\{y_{1}, \ldots , y_{n}\}\). From now on, we use the following notation. For \(i, j \in \{1, \ldots , n\}\), with \(i \ne j\), let \(\sigma _{i,j}\) and \(\tau _{i,j}\) be the automorphisms of \(G_{n,k}\) satisfying the conditions

$$\begin{aligned} \sigma _{i,j}(y_{i})= & {} y_{j}, \sigma _{i,j}(y_{j}) = y_{i}, \sigma _{i,j}(y_{r}) = y_{r} ~\text {for all }r \in \{1, \ldots , n\} \setminus \{i, j\}, \\ \tau _{i,j}(y_{i})= & {} y_{i}y_{j}, \tau _{i,j}(y_{r}) = y_{r} ~\text {for all }r \in \{1, \ldots , n\} \setminus \{i\}. \end{aligned}$$

Let \(\rho _{1}\) be the automorphism of \(G_{n,k}\) satisfying the conditions

$$\begin{aligned} \rho _{1}(y_{1}) = y_{2}y_{1}, \rho _{1}(y_{r}) = y_{r}, r = 2, \ldots , n. \end{aligned}$$

It is readily verified that \(\sigma _{i,j}\), \(\tau _{i,j}\) and \(\rho _{1}\) are tame automorphisms of \(G_{n,k}\).

We write \(\rho _{2}\) for the endomorphism of \(G_{n,k}\) satisfying the conditions

$$\begin{aligned} \rho _{2}(y_{1}) = y_{1}^{-1}, \rho _{2}(y_{2}) = y_{2}y_{1}, \rho _{2}(y_{r}) = y_{r}, r = 3, \ldots , n. \end{aligned}$$

We point out that \(\rho _{2}\) is a tame automorphism of \(G_{n,k}\): it is straightforward to verify that \(\rho _{2} = \sigma _{1,2}\tau _{1,2}\sigma _{1,2}\mu \), where \(\mu \) is the tame automorphism of \(G_{n,k}\) satisfying the conditions \(\mu (y_{1}) = y_{1}^{-1}\), \(\mu (y_{r}) = y_{r}\), \(r = 2, \ldots , n\), and \(\rho _{2}^{-1} = \rho _{2}\).

We write \({\overline{w}}\) for the endomorphism of \(G_{n,k}\) satisfying the conditions

$$\begin{aligned} {\overline{w}}(y_{1}) = y_{1}[y_{1}, y_{2}, [y_{3}, y_{4}]], {\overline{w}}(y_{j}) = y_{j}, j = 2, \ldots , n. \end{aligned}$$

We point out that, by Corollary 1 (2), \({\overline{w}}\) is an automorphism of \(G_{n,k}\).

Let \(j \in \{1, \ldots , k-1\}\). For positive integers \(m_{1}, \ldots , m_{j}\) and \(i_{1}, \ldots , i_{j} \in \{1, \ldots , n\}\), with \(i_{1}< i_{2}< \ldots < i_{j}\), we write \({\overline{w}}_{(y_{i_{1}}^{m_{1}}y_{i_{2}}^{m_{2}}\ldots y_{i_{j}}^{m_{j}})}\) for the endomorphism of \(G_{n,k}\) satisfying the conditions

$$\begin{aligned} \begin{aligned} {\overline{w}}_{(y_{i_{1}}^{m_{1}}y_{i_{2}}^{m_{2}}\ldots y_{i_{j}}^{m_{j}})}(y_{1})&= y_{1}[[y_{1},y_{2}], [y_{3}, y_{4}], ~_{m_{1}}y_{i_{1}}, \ldots , ~_{m_{j}}y_{i_{j}}],\\ {\overline{w}}_{(y_{i_{1}}^{m_{1}}y_{i_{2}}^{m_{2}}\ldots y_{i_{j}}^{m_{j}})}(y_{r})&= y_{r}, r = 2, \ldots , n. \end{aligned} \end{aligned}$$

In the case that some \(m_{r} \in \{m_{1}, \ldots , m_{j}\}\) is 1, we write \({\overline{w}}_{(y_{i_{1}}^{m_{1}}\ldots y_{i_{r-1}}^{m_{r-1}}y^{}_{i_{r}}y_{i_{r+1}}^{m_{r+1}}\ldots y_{i_{j}}^{m_{j}})}\) instead of \({\overline{w}}_{(y_{i_{1}}^{m_{1}}\ldots y_{i_{r-1}}^{m_{r-1}}y_{i_{r}}^{^{1}}y_{i_{r+1}}^{m_{r+1}}\ldots y_{i_{j}}^{m_{j}})}\). We point out that, by Corollary 1 (2), \({\overline{w}}_{(y_{i_{1}}^{m_{1}}y_{i_{2}}^{m_{2}}\ldots y_{i_{j}}^{m_{j}})}\) is an automorphism of \(G_{n,k}\). For our convenience, for a positive integer m, we write \({\overline{w}}_{m}\) instead of \({\overline{w}}_{(y_{2}^{m})}\), that is, \({\overline{w}}_{m}\) satisfies the conditions

$$\begin{aligned} {\overline{w}}_{m}(y_{1}) = y_{1}[[y_{1},y_{2}], [y_{3}, y_{4}], ~_{m}y_{2}], {\overline{w}}_{m}(y_{r}) = y_{r}, r = 2, \ldots , n. \end{aligned}$$

We point out that for \(m \ge k\), \({\overline{w}}_{m}\) is the identity automorphism of \(G_{n,k}\).

4.2 A Generating Set of \({\textrm{Aut}}(G_{n,k})\)

By Corollary 1 (3), the natural group epimorphism from \(G_{n}\) onto \(G_{n,k}\) induces a group epimorphism, say \(\phi _{n,k}\), from \({\textrm{Aut}}(G_{n})\) onto \({\textrm{Aut}}(G_{n,k})\). Since for \(n \ge 4\), by Proposition 1, \({\textrm{Aut}}(G_{n}) = \langle T_{n}, \alpha _{(m,0)}: m \ge 0\rangle \), it follows that \({\textrm{Aut}}(G_{n,k}) = \langle \phi _{n,k}(T_{n}), \phi _{n,k}(\alpha _{(m,0)}): m \ge 0\rangle \). Hence, we have an infinite generating set for \({\textrm{Aut}}(G_{n,k})\) which is independent upon n. Our first aim is to give a finite generating set for \({\textrm{Aut}}(G_{n,k})\), with \(n \ge 4\), which we need in the following.

Proposition 3

For positive integers n and k, with \(n \ge 4\) and \(k \ge 2\), let \(G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})[F^{\prime \prime }_{n},~_{k}F_{n}]\) be freely generated by the set \(\{y_{1}, \ldots , y_{n}\}\). Then, \({\textrm{Aut}}(G_{n,k}) = \langle T_{n,k}, {\overline{w}}, {\overline{w}}_{m}: m = 1, \ldots , k-1\rangle \).

Proof

Since \(\phi _{n,k}\) is a group epimorphism from \({\textrm{Aut}}(G_{n})\) onto \({\textrm{Aut}}(G_{n,k})\) and, by Proposition 2, \({\textrm{Aut}}(G_{n}) = \langle T_{n}, w_{m}: m \ge 0\rangle \), we get \({\textrm{Aut}}(G_{n,k}) = \langle \phi _{n,k}(T_{n}), \phi _{n,k}(w_{m}): m \ge 0\rangle \). Since, by our notation, \(\phi _{n,k}(T_{n}) = T_{n, k}\), \(\phi _{n,k}(w_{0}) = {\overline{w}}\), \(\phi _{n,k}(w_{m}) = {\overline{w}}_{m}\) for \(m = 1, \ldots , k-1\) and \(\phi _{n,k}(w_{m})\) is the identity automorphism for \(k \ge m\), we have the required result. \(\square \)

4.3 The Case \(k = 2\)

Since \(\gamma _{3}(F_{n}^{\prime }) \subseteq [F_{n}^{\prime \prime }, F_{n}, F_{n}]\), \(G_{n,2} = F_{n}/[F_{n}^{\prime \prime }, F_{n}, F_{n}]\), that is, \(G_{n,2}\) is a free centre-by-(centre-by-metabelian) group of rank n.

Proposition 4

For a positive integer n, with \(n \ge 4\), let \(G_{n,2} = F_{n}/[F_{n}^{\prime \prime }, F_{n}, F_{n}]\) be a free centre-by-(centre-by-metabelian) group of rank n freely generated by the set \(\{y_{1}, \ldots , y_{n}\}\). Then, \({\textrm{Aut}}(G_{n,2}) = \langle T_{n,2}, {\overline{w}}\rangle \), where \({\overline{w}}\) is the non-tame automorphism of \(G_{n,2}\) satisfying the conditions \({\overline{w}}(y_{1}) = y_{1}[[y_{1}, y_{2}], [y_{3}, y_{4}]]\) and \({\overline{w}}(y_{j}) = y_{j}\), \(j = 2, \ldots , n\).

Proof

For simplicity, throughout the proof we write \(A = [y_{1}, y_{2}]\) and \(B = [y_{3}, y_{4}]\). It follows from Proposition 3 that \({\textrm{Aut}}(G_{n,2}) = \langle T_{n,2}, {\overline{w}}, {\overline{w}}_{1}\rangle \). Hence, to prove the required result, it suffices to show that \({\overline{w}}_{1} \in \langle T_{n,2}, {\overline{w}}\rangle \). We show at first that \({\overline{w}}_{(y_{1})} \in \langle T_{n,2}, {\overline{w}}\rangle \) (recall that, by our notation, \({\overline{w}}_{(y_{1})}\) is the automorphism of \(G_{n,2}\) satisfying the conditions \({\overline{w}}_{(y_{1})}(y_{1}) = y_{1}[A, B, y_{1}]\), \({\overline{w}}_{(y_{1})}(y_{r}) = y_{r}\), \(r = 2, \ldots , n\)). We claim that \(\sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1} = {\overline{w}}_{(y_{1})}\). By direct calculations, we get

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{1}) = [[y_{1}^{-1}, y_{2}y_{1}], B]^{-1}y_{1}[A, B]^{-1}. \end{aligned}$$

Using the standard group commutator identities \([a, bc] = [a,c][a,b]^{c}\), \([a^{-1},b] = ([a, b]^{-1})^{a^{-1}}\), \(ab = ba[a,b]\) and since \(\gamma _{3}(G_{n,k}^{\prime }) = \{1\}\), we get

$$\begin{aligned} \begin{aligned} {[[}y_{1}^{-1}, y_{2}y_{1}], B]^{-1}y_{1}&= [[y_{1}^{-1}, y_{2}]^{y_{1}}, B]^{-1}y_{1} = [A^{-1}, B]^{-1}y_{1}\\&= [A, B]y_{1} = y_{1}[A, B][A, B,y_{1}] = y_{1}[A, B,y_{1}][A, B]. \end{aligned} \end{aligned}$$

and hence, it follows that

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{1}) = y_{1}[A, B,y_{1}] = {\overline{w}}_{(y_{1})}(y_{1}). \end{aligned}$$

Similarly, we have

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{2}) = y_{2}[A^{-1}, B]^{-1}[[y_{1}^{-1},y_{2}y_{1}], B]. \end{aligned}$$

Since, by the above calculations, \([A^{-1}, B]^{-1} = [A,B]\) and \([[y_{1}^{-1},y_{2}y_{1}], B] = [A, B]^{-1}\), we get

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{2}) = y_{2}[A, B][A, B]^{-1} = y_{2} = {\overline{w}}_{(y_{1})}(y_{2}). \end{aligned}$$

Furthermore, it is straightforward to verify that

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{r}) = {\overline{w}}_{(y_{1})}(y_{r}) = y_{r}, r = 3, \ldots , n. \end{aligned}$$

Hence, our claim is true, that is, \(\sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1} = {\overline{w}}_{(y_{1})}\), and thus, \({\overline{w}}_{(y_{1})} \in \langle T_{n,2}, {\overline{w}}\rangle \).

We show now that \({\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1} = {\overline{w}}_{1}\). By direct calculations, we get

$$\begin{aligned} {\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1}(y_{1}) = y_{1}[[A, B, y_{1}]^{-1}[[y_{2}y_{1},y_{2}], B, y_{2}y_{1}]. \end{aligned}$$

By the group commutator identities \([ab, c] = [a,c]^{b}[b,c]\) and \([a, bc] = [a, c][a,b][a, b, c]\) and since \([G_{n,2}^{\prime \prime }, G_{n,2}, G_{n,2}] = \{1\}\), we get

$$\begin{aligned} \begin{aligned}&{[}A, B, y_{1}]^{-1}[[y_{2}y_{1},y_{2}], B, y_{2}y_{1}] = [A, B, y_{1}]^{-1}[A, B, y_{2}y_{1}] \\&\quad = [A, B, y_{1}]^{-1}[A, B, y_{1}][A, B, y_{2}][A, B, y_{2},y_{1}] = [A, B, y_{2}]. \end{aligned} \end{aligned}$$

Hence,

$$\begin{aligned} {\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1}(y_{1}) = y_{1}[[A, B, y_{2}] = {\overline{w}}_{1}(y_{1}). \end{aligned}$$

Furthermore, it is straightforward to verify that

$$\begin{aligned} {\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1}(y_{r}) = {\overline{w}}_{1}(y_{r}) = y_{r}, r = 2, \ldots , n. \end{aligned}$$

Therefore, \({\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1} = {\overline{w}}_{1}\). Since, as shown above, \({\overline{w}}_{(y_{1})} \in \langle T_{n,2}, {\overline{w}}\rangle \), it follows that \({\overline{w}}_{1} \in \langle T_{n,2}, {\overline{w}}\rangle \). We point out that, by [6, proof of Theorem 2], the automorphism \({\overline{w}}\) is non-tame. \(\square \)

4.4 The Case \(k \ge 3\)

From now on, let n and k be positive integers, with \(n \ge k+2\ge 5\). The next result is a crucial step for the proof of Theorem 1.

Proposition 5

For positive integers n and k, with \(n \ge k+2\ge 5\), let \(G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})[F^{\prime \prime }_{n},~_{k}F_{n}]\) be freely generated by the set \(\{y_{1}, \ldots , y_{n}\}\) and let \(H = \langle T_{n.k}, {\overline{w}}\rangle \), where \({\overline{w}}\) is the automorphism of \(G_{n,k}\) satisfying the conditions \({\overline{w}}(y_{1}) = y_{1}[[y_{1}, y_{2}], [y_{3}, y_{4}]]\) and \({\overline{w}}(y_{j}) = y_{j}\), \(j = 2, \ldots , n\). Then:

  1. (1)

    \({\overline{w}}_{(y_{5})} \in H\).

  2. (2)

    \({\overline{w}}_{(y_{5}\ldots y_{j})} \in H\) for all \(j = 5, \ldots , k+2\).

  3. (3)

    \({\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})} \in H\).

  4. (4)

    \({\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})} \in H\) for all \(r = 1, \ldots , k-2\).

  5. (5)

    \({\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{j})} \in H\) for all \(r = 1, \ldots , k-2\) and \(j = 5, \ldots , k+2\), with \(6 \le r+j \le k+3\).

Proof

Throughout the proof, we write \(A = [y_{1}, y_{2}]\), \(B = [y_{3}, y_{4}]\) and \(W = [A, B]\).

  1. (1)

    Let \(\nu \) be the tame automorphism of \(G_{n,k}\) satisfying the conditions

    $$\begin{aligned} \nu (y_{3}) = y_{5}^{-1}y_{3}y_{5}, \nu (y_{4}) = y_{5}^{-1}y_{4}y_{5}, \nu (y_{r}) = y_{r}, r \in \{1, \ldots , n\} \setminus \{3, 4\}. \end{aligned}$$

    We show that \({\overline{w}}^{-1}(\nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1})( \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1}) = {\overline{w}}_{(y_{5})}\). By direct calculations, we get

    $$\begin{aligned} \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1}(y_{1}) = y_{1}[[y_{1}, y_{2}y_{5}], B^{y_{5}}]. \end{aligned}$$

    By the group commutator identities \([a, bc] = [a, c][a,b]^{c}\) and \([a^{c}, b^{c}] = [a,b][a,b,c]\) and since \(\gamma _{3}(G_{n,k}^{\prime }) = \{1\}\), we get

    $$\begin{aligned} \begin{aligned} {[[}y_{1}, y_{2}y_{5}], B^{y_{5}}]&= [[y_{1},y_{5}]A^{y_{5}}, B^{y_{5}}] = [[y_{1},y_{5}], B^{y_{5}}][A^{y_{5}}, B^{y_{5}}]\\ {}&= [[y_{1},y_{5}], B^{y_{5}}]W[W, y_{5}] \end{aligned} \end{aligned}$$

    and hence,

    $$\begin{aligned} \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1}(y_{1}) = y_{1}[[y_{1},y_{5}], B^{y_{5}}]W[W, y_{5}]. \end{aligned}$$

    Furthermore, by direct calculations, we get

    $$\begin{aligned} \nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1}(y_{1}) = y_{1}[[y_{1},y_{5}], B^{y_{5}}]^{-1}. \end{aligned}$$

    Hence,

    $$\begin{aligned} \begin{aligned}&{\overline{w}}^{-1}(\nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1})( \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1})(y_{1}) \\&\quad =y_{1}W^{-1}[[y_{1},y_{5}], B^{y_{5}}]^{-1}[[y_{1},y_{5}], B^{y_{5}}]W[W, y_{5}] = y_{1}[W, y_{5}] = {\overline{w}}_{(y_{5})}(y_{1}). \end{aligned} \end{aligned}$$

    Furthermore, it is straightforward to verify that

    $$\begin{aligned} {\overline{w}}^{-1}(\nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1})( \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1})(y_{r}) = {\overline{w}}_{(y_{5})}(y_{r}) = y_{r}, r = 2, \ldots , n. \end{aligned}$$

    Therefore, \({\overline{w}}^{-1}(\nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1})( \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1}) = {\overline{w}}_{(y_{5})}\) and hence, \({\overline{w}}_{(y_{5})} \in H\).

  2. (2)

    We use induction on j. For \(j = 5\), the result has been proved in Proposition 5 (1). Assume that the result is true for some \(5 \le j < k+2\), that is, assume that \({\overline{w}}_{(y_{5}\ldots y_{j})} \in H\). We prove that

    $$\begin{aligned} \begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1} = \sigma _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\sigma _{j,j+1}^{-1}{\overline{w}}_{(y_{5}\ldots y_{j+1})}. \end{aligned} \end{aligned}$$

    We show the above equation for \(j > 5\). The case \(j = 5\) may be treated similarly. By direct calculations, we get

    $$\begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1}(y_{1}) \!=\! y_{1}[W, y_{5}, \ldots , y_{j}]^{-1}[W, y_{5}, \ldots , y_{j-1}, y_{j}y_{j+1}]. \end{aligned}$$

    Using the standard group commutator identity \([a, bc] = [a,c][a,b][a,b,c]\) and since \(\gamma _{3}(G_{n,k}^{\prime }) = \{1\}\), we get

    $$\begin{aligned} \begin{aligned}&{[}W, y_{5}, \ldots , y_{j}]^{-1}[[W, y_{5}, \ldots , y_{j-1}, y_{j}y_{j+1}] \\&= {[}W, y_{5}, \ldots , y_{j}]^{-1}[W, y_{5}, \ldots , y_{j-1}, y_{j+1}][W, y_{5}, \ldots , y_{j}][W, y_{5}, \ldots , y_{j+1}] \\&= {[}W, y_{5}, \ldots , y_{j-1}, y_{j+1}][W, y_{5}, \ldots , y_{j+1}]. \end{aligned} \end{aligned}$$

    Hence,

    $$\begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1}(y_{1}) = y_{1}[W, y_{5}, \ldots , y_{j-1}, y_{j+1}][W, y_{5}, \ldots , y_{j+1}]. \end{aligned}$$

    It is straightforward to verify that

    $$\begin{aligned} \begin{aligned} \sigma _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\sigma _{j,j+1}^{-1}{\overline{w}}_{(y_{5}\ldots y_{j+1})}(y_{1})&= [W, y_{5}\ldots y_{j-1}, y_{j+1}][W, y_{5}\ldots y_{j+1}] \\&= {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1}(y_{1}) \end{aligned} \end{aligned}$$

    and

    $$\begin{aligned} \begin{aligned} \sigma _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\sigma _{j,j+1}^{-1}{\overline{w}}_{(y_{5}\ldots y_{j+1})}(y_{r})&= {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1}(y_{r}) \\ {}&= y_{r}, r = 2, \ldots , n. \end{aligned} \end{aligned}$$

    Hence,

    $$\begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1} = \sigma _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\sigma _{j,j+1}^{-1}{\overline{w}}_{(y_{5}\ldots y_{j+1})}. \end{aligned}$$

    Since, by our inductive argument, \({\overline{w}}_{(y_{5}\ldots y_{j})} \in H\), it follows from the above equation that \({\overline{w}}_{(y_{5}\ldots y_{j+1})} \in H\) and hence, we get the required result.

  3. (3)

    We claim that

    $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1} = {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}. \end{aligned}$$

    By similar arguments as in the proof of Proposition 4, we have

    $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1}(y_{1}) = y_{1}[W, y_{5}, \ldots , y_{k+2}, y_{1}]. \end{aligned}$$

    Since, by Lemma 2 (2), \([W, y_{5}, \ldots , y_{k+2}, y_{1}] = [W, y_{1}, y_{5}, \ldots , y_{k+2}]\), we get

    $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1}(y_{1}) = {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}(y_{1}). \end{aligned}$$

    By similar arguments as in the proof of Proposition 4, we have

    $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1}(y_{2}) = y_{2}. \end{aligned}$$

    Furthermore, it is straightforward to verify that

    $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1}(y_{r}) = y_{r}, r = 3, \ldots , n. \end{aligned}$$

    Therefore, our claim is true, that is,

    $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1} = {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}. \end{aligned}$$

    Since, by Proposition 5 (2) (for \(j = k+2\)), \({\overline{w}}_{(y_{5}\ldots y_{k+2})} \in H\), it follows from the above equation that \({\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})} \in H\).

  4. (4)

    We use induction on r. At first, we prove the required result for \(r = 1\). We show that \({\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}^{-1}\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1} = {\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})}\). By direct calculations, we get

    $$\begin{aligned} \begin{aligned} \rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{1})&= y_{1}[[y_{2}y_{1}, y_{2}], B, y_{2}y_{1}, y_{5}, \ldots , y_{k+2}]\\&= y_{1}[W, y_{2}y_{1}, y_{5}, \ldots , y_{k+2}]. \end{aligned} \end{aligned}$$

    Since \(\gamma _{3}(G_{n,k}^{\prime }) = \{1\}\), it follows from Lemma 2 (1) that

    $$\begin{aligned} \begin{aligned}&\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{1}) \\&\quad =y_{1}[W, y_{1}, y_{5}, \ldots , y_{k+2}][W, y_{2}, y_{5}, \ldots , y_{k+2}][W, y_{2}, y_{1}, y_{5}, \ldots , y_{k+2}]. \end{aligned} \end{aligned}$$

    Since \([G_{n,k}^{\prime \prime }, ~_{k}G_{n,k}] = \{1\}\), we have \([W, y_{2}, y_{1}, y_{5}, \ldots , y_{k+2}] = 1\) and hence,

    $$\begin{aligned} \rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{1}) = y_{1}[W, y_{1}, y_{5}, \ldots , y_{k+2}][W, y_{2}, y_{5}, \ldots , y_{k+2}]. \end{aligned}$$

    Therefore,

    $$\begin{aligned} {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}^{-1}\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{1})= & {} y_{1}[W, y_{2}, y_{5}, \ldots , y_{k+2}] \\= & {} {\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})}(y_{1}). \end{aligned}$$

    It is straightforward to verify that

    $$\begin{aligned} {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}^{-1}\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{r}) = {\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})}(y_{r}) = y_{r} , r = 2, \ldots , n. \end{aligned}$$

    Therefore, \({\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}^{-1}\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1} = {\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})}\). Since, by Proposition 5 (3), \({\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})} \in H\), it follows from the above equation that \({\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})} \in H\), that is, the required result is true for \(r = 1\).

    Assume that the result is true for some \(r \ge 1\), that is, assume that \({\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})} \in H\) for some \(1 \le r < k-2\). We show that

    $$\begin{aligned} {\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1} = {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}. \end{aligned}$$

    Since

    $$\begin{aligned} \begin{aligned}&{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1}(y_{j}) \\&= {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}(y_{j})=y_{j},j = 2, \ldots , n, \end{aligned} \end{aligned}$$

    it suffices to show that

    $$\begin{aligned} {\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1}(y_{1}) \!=\! {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}(y_{1}). \end{aligned}$$

    By similar calculations as in the proof of Proposition 5 (2) and since \([G_{n,k}^{\prime \prime }, ~_{k}G_{n,k}] = \{1\}\), we get

    $$\begin{aligned} \begin{aligned}&{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1}(y_{1}) \\ {}&\quad = y_{1}[W, ~_{r}y_{2}, y_{5}, \ldots , y_{k+2-r}, y_{2}] \end{aligned} \end{aligned}$$

    and hence, by Lemma 2 (2),

    $$\begin{aligned} \begin{aligned}&{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1}(y_{1}) \\ {}&\quad = y_{1}[W, ~_{r+1}y_{2}, y_{5}, \ldots , y_{k+2-r}] = {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}(y_{1}). \end{aligned} \end{aligned}$$

    Thus,

    $$\begin{aligned} {\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1} = {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}. \end{aligned}$$

    Since, by our inductive argument, \({\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})} \in H\), it follows from the above equation that \({\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})} \in H\) and hence, we get the required result.

  5. (5)

    For \(k = 3\) (and hence, \(r = 1\) and \(j = 5\)), it follows from Proposition 5 (4) that \({\overline{w}}_{(y_{2}y_{5})} \in H\) and so, in what follows, we may assume that \(k > 3\). We use inverse induction on \(t = r+j\). For \(t = k+3\) (that is, \(j = k+3-r\)), the required result has been proved in Proposition 5 (4). Assume that the result is true for some \(6 < t \le k+3\), that is, assume that \({\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r})} \in H\) for all \(r = 1, \ldots , t-5\). Let \(r+j = t-1\). We consider the following cases:

    1. (a)

      \(k \ge 4\) and \(t = 7\). In this case, we have to show that \({\overline{w}}_{(y_{2}y_{5})} \in H\). By similar calculations as in the proof of Proposition 5 (4), we get

      $$\begin{aligned} {\overline{w}}_{(y_{5}y_{6})}^{-1}\tau _{6, 2}{\overline{w}}_{(y_{5}y_{6})}\tau _{6,2}^{-1} = {\overline{w}}_{(y_{2}y_{5})}{\overline{w}}_{(y_{2}y_{5}y_{6})}. \end{aligned}$$

      Since, by Proposition 5 (2), \({\overline{w}}_{(y_{5}y_{6})} \in H\) and, by our inductive argument, \({\overline{w}}_{(y_{2}y_{5}y_{6})} \in H\), from the above equation we get the required result.

    2. (b)

      \(k > 4\) and \(t > 7\). In this case, we have to show that \({\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})} \in H\) for all \(r = 1, \ldots , t-6\). By similar calculations as in the proof of Proposition 5 (4), we get

      $$\begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{t-1})}^{-1}\tau _{t-1, 2}{\overline{w}}_{(y_{5}\ldots y_{t-1})}\tau _{t-1,2}^{-1} = {\overline{w}}_{(y_{2}y_{5}\ldots y_{t-2})}{\overline{w}}_{(y_{2}y_{5}\ldots y_{t-1})} \end{aligned}$$

      and, for \(r = 1, \ldots , t - 7\),

      $$\begin{aligned} \begin{aligned}&{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})}^{-1}\tau _{t-r-1, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})}\tau _{t-r-1,2}^{-1}\\ {}&\quad = {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{t-r-2)})}{\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{t-r-1})}. \end{aligned} \end{aligned}$$

      Since, by our inductive argument, \({\overline{w}}_{(y_{2}y_{5}\ldots y_{t-1})}, {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{t-r-1})} \in H\), it follows from the above equations that

      $$\begin{aligned} {\overline{w}}_{(y_{2}y_{5}\ldots y_{t-2})}^{-1}{\overline{w}}_{(y_{5}\ldots y_{t-1})}^{-1}\tau _{t-1, 2}{\overline{w}}_{(y_{5}\ldots y_{t-1})}\tau _{t-1,2}^{-1} \in H \end{aligned}$$
      (1)

      and

      $$\begin{aligned} {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{t-r-2})}^{-1}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})}^{-1}\tau _{t-r-1, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})}\tau _{t-r-1,2}^{-1} \in H. \end{aligned}$$
      (2)

      Since, by Proposition 5 (2), \({\overline{w}}_{(y_{5}\ldots y_{t-1})} \in H\), by equation (1) we get \({\overline{w}}_{(y_{2}y_{5}\ldots y_{t-2})} \in H\). Hence, by equation (2) and an obvious induction on r, we get \({\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})} \in H\) for all \(r = 1, \ldots , t-6\).

    By the above cases, we get the required result.

\(\square \)

4.5 Proof of Theorem 1

We are now in position to give a proof of Theorem 1.

Proof of Theorem 1

For \(k = 2\), the result has been proved in Proposition 4 and hence, we may assume that \(n \ge k+2 \ge 5\). We write \(H = \langle T_{n,k}, {\overline{w}}\rangle \). We point out that, by [6, proof of Theorem 2], the automorphism \({\overline{w}}\) is non-tame. Since, by Proposition 3, \({\textrm{Aut}}(G_{n,k}) = \langle T_{n,k}, {\overline{w}}, {\overline{w}}_{m}: m = 1, \ldots , k-1\rangle \), to complete the proof, it suffices to show that \({\overline{w}}_{m} \in H\) for all \(m = 1, \ldots , k-1\). Assume at first that \(m = 1\). By similar calculations as in the proof of Proposition 5 (2), we get

$$\begin{aligned} {\overline{w}}_{(y_{5})}^{-1}\tau _{5,2}{\overline{w}}_{(y_{5})}\tau _{5,2}^{-1} = {\overline{w}}_{1}{\overline{w}}_{(y_{2}y_{5})}. \end{aligned}$$

Since, by Proposition 5 (1), \({\overline{w}}_{(y_{5})} \in H\) and, by Proposition 5 (5) (for \(r = 1\) and \(j =5\)), \({\overline{w}}_{(y_{2}y_{5})} \in H\), it follows from the above equation that \({\overline{w}}_{1} \in H\). Now, fix some \(2 \le m \le k-1\). By similar calculations as in the proof of Proposition 5 (2), we get

$$\begin{aligned} {\overline{w}}_{(y_{2}^{m-1}y_{5})}^{-1}\tau _{5,2}{\overline{w}}_{(y_{2}^{m-1}y_{5})}\tau _{5,2}^{-1} = {\overline{w}}_{m}{\overline{w}}_{(y_{2}^{m}y_{5})}. \end{aligned}$$

Since, by Proposition 5 (5), \({\overline{w}}_{(y_{2}^{m}y_{5})} \in H\) for all \(m = 2, \ldots , k-2\) and \({\overline{w}}_{(y_{2}^{k-1}y_{5})}\) is the identity mapping, it follows from the above equation that \({\overline{w}}_{m} \in H\) for all \(m = 2, \ldots , k-1\). Hence, we get the required result. \(\square \)

4.6 Concluding Remarks

  1. (1)

    It would be interesting to study the case \(4 \le n < k+2\), with \(k \ge 3\). By Proposition 3, \({\textrm{Aut}}(G_{n,k}) = \langle T_{n,k}, {\overline{w}}, {\overline{w}}_{m}: m = 1, \ldots , k-1\rangle \). Write \(d_{n,k}\) for the minimum number of generators of \({\textrm{Aut}}(G_{n,k})\). It would be interesting to know if there exists a number d such that \(d_{n,k} \le d\) for all n and d, with \(4 \le n < k+2\).

  2. (2)

    It has been conjectured by A. Papistas in [9] that \({\textrm{Aut}}(G_{n})\) is not finitely generated. If such an aforementioned number d for the minimum number of generators of \({\textrm{Aut}}(G_{n,k})\) does not exist, then the conjecture is true. (Assume that such a number d does not exist and suppose, on the contrary, that the conjecture is not true, that is, suppose that \({\textrm{Aut}}(G_{n})\) is finitely generated. Then, by Proposition 2, \({\textrm{Aut}}(G_{n})\) is generated by \(T_{n}\) and a fixed finite subset of the set \(\{w_{m}: m \ge 0\}\). Hence, by Proposition 3, for sufficiently large k, the number of generators of \({\textrm{Aut}}(G_{n,k})\) is independent upon n and k, which is the required contradiction.)

  3. (3)

    For positive integers n and k, with \(n \ge 4\) and \(k \ge 3\), it is still an open problem to find a generating set for \({\textrm{Aut}}(F_{n})/[F_{n}^{\prime \prime }, ~_{k}F_{n}]\).