Automorphisms in Certain Nilpotent-by-Abelian Varieties of Groups

Abstract

For positive integers n and k, with $n \ge 4$, let $F_{n}$ be the free group of rank n and let $G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})[F^{\prime \prime }_{n},~_{k}F_{n}]$. We show that for sufficiently large n, the automorphism group ${\textrm{Aut}}(G_{n,k})$ of $G_{n,k}$ is generated by the tame automorphisms and one more non-tame automorphism.

1 Introduction and Notation

Let G be a group. We use standard commutator notation: for elements $g_{1}, \ldots , g_{c}$ of G, we write $[g_{1}, g_{2}]= g_{1}^{-1}g_{2}^{-1}g_{1}g_{2}$ and, for $c\ge 3$, $[g_{1}, \ldots , g_{c}] = [[g_{1}, \ldots , g_{c-1}], g_{c}]$. For a non-negative integer d, we write $[x, ~_{d}y] = [x, y, \ldots , y]$, where $x, y \in G$ and y is repeated d times (for $d = 0$, we write $[x,~ _{0}y] = x$). For a non-empty set X, $\langle X\rangle$ denotes the group generated by X and for subgroups $A_{1}, \ldots , A_{c}$ of G, with $c \ge 1$, $\langle A_{1}, \ldots , A_{c}\rangle$ denotes the subgroup of G generated by $A_{1}, \ldots , A_{c}$. For non-empty subsets $X_{1}, \ldots , X_{c}$ of G, we write $[X_{1}, X_{2}] = \langle [g_{1}, g_{2}]: g_{1} \in X_{1}, g_{2} \in X_{2}\rangle$ and, for $c \ge 3$, $[X_{1}, \ldots , X_{c}] = [[X_{1}, \ldots , X_{c-1}], X_{c}]$. For a positive integer d, we write $[X_{1}, ~_{d}X_{2}] = [X_{1}, X_{2}, \ldots , X_{2}]$, where $X_{2}$ is repeated d times. For a positive integer c, let $\gamma _{c}(G)$ be the cth term of the lower central series of G. We point out that $\gamma _{2}(G) = G^{\prime }$, that is, $G^{\prime }$ is the derived subgroup of G, and $G^{\prime \prime } =(G^{\prime })^{\prime }$. We write ${\textrm{Aut}}(G)$ for the automorphism group of G.

A variety of groups is the class of all groups satisfying each one of a given set of laws. For further information concerning varieties of groups, see [8]. Let $F_{n}$ be the free group of finite rank n, with $n \ge 2$, freely generated by the set $\{f_{1}, \ldots , f_{n}\}$. Let V be a variety of groups and let $V(F_{n})$ be the verbal subgroup of $F_{n}$ corresponding to V. Thus, the group $F_{n}(V) = F_{n}/V(F_{n})$ is a relatively free group of rank n in V. The natural mapping from $F_{n}$ onto $F_{n}(V)$ induces a group homomorphism, say $\pi _{n,V}$, from ${\textrm{Aut}}(F_{n})$ into ${\textrm{Aut}}(F_{n}(V)$. The image of $\pi _{n,V}$ is denoted by $T_{F_{n}(V)}$ and the automorphisms of $F_{n}(V)$ which belong to $T_{F_{n}(V)}$ are called tame; otherwise they are called non-tame (or wild). Since, by [7], ${\textrm{Aut}}(F_{n})$ is finitely generated, $T_{F_{n}(V)}$ is also finitely generated.

For a positive integer n, with $n \ge 2$, let $G_{n} = F_{n}/\gamma _{3}(F_{n}^{\prime })$, that is, $G_{n}$ is a free nilpotent of class 2-by-abelian group of rank n. We write $x_{i} = f_{i}\gamma _{3}(F_{n}^{\prime })$ for $i = 1, \ldots , n$. Thus, the set $\{x_{1}, \ldots , x_{n}\}$ is a free generating set of $G_{n}$. For $n = 4$, it has been proved in [6] that ${\textrm{Aut}}(G_{4})$ is generated by $T_{G_{4}}$ and an explicitly given set of non-tame automorphisms of $G_{4}$, and this result has been extended in [9] for all $n \ge 4$. For positive integers n and k, we write $G_{n,k} = F_{n}/\gamma _{3}(F_{n}^{\prime })[F^{\prime \prime },~_{k}F_{n}]$. Since $G_{n,k} \cong G_{n}/[G_{n}^{\prime \prime }, ~_{k}G_{n}]$, we may assume, without loss of generality, that $G_{n,k} = F_{n}/\gamma _{3}(F_{n}^{\prime })[F^{\prime \prime },~_{k}F_{n}] = G_{n}/[G_{n}^{\prime \prime }, ~_{k}G_{n}]$. Write $y_{i,k} = x_{i}[G^{\prime \prime }_{n}, ~_{k}G_{n}]$. Thus, $G_{n,k}$ is freely generated by the set $\{y_{1,k}, \ldots , y_{n,k}\}$. For our convenience, throughout this paper, we write $y_{i}$ instead of $y_{i, k}$ for $i = 1, \ldots , n$. We write $T_{n,k}$ for the group $T_{G_{n,k}}$ of the tame automorphisms of $G_{n,k}$.

Our work is motivated by [6, Concluding remarks]. In [6], C. K. Gupta and F. Levin have found a generating set for ${\textrm{Aut}}(G_{4})$. Since the natural group epimorphism from $G_{n}$ onto $G_{n,k}$ induces a group epimorphism from ${\textrm{Aut}}(G_{n})$ onto ${\textrm{Aut}}(G_{n,k})$ for all $n \ge 4$ (see Sect. 2), they have also found a generating set for ${\textrm{Aut}}(G_{4,2})$. Then, for $n \ge 5$, they claim to have shown that ${\textrm{Aut}}(G_{n,2})$ is generated by $T_{n,2}$ and one more automorphism. Since a generating set of ${\textrm{Aut}}(G_{n,2})$, with $n \ge 5$, was not known in [6], the aforementioned claim, at the time it was made, does not seem to be right. Furthermore, based on their aforementioned result for ${\textrm{Aut}}(G_{n,2})$, with $n \ge 5$, the authors claim that for $n \ge k +3 \ge 5$, ${\textrm{Aut}}(F_{n})/[F_{n}^{\prime \prime }, ~_{k}F_{n}]$ is generated by the tame automorphisms and one more non-tame automorphism. In this generality, this claim does not seem to may be proved by the work in [6]. Since $G_{n,1} = F_{n}/[F_{n}^{\prime \prime }, F_{n}]$ and $G_{n,2} = F_{n}/[F_{n}^{\prime \prime }, ~_{2}F_{n}]$, in the spirit of the claim in [6, Concluding remarks], we are motivated to study ${\textrm{Aut}}(G_{n,k})$, with $n \ge 4$ and $k \ge 2$. Since a generating set of ${\textrm{Aut}}(G_{n})$, with $n \ge 4$, is given by A. Papistas [9], we may prove that ${\textrm{Aut}}(G_{n,2})$ is generated by $T_{n,2}$ and one more non-tame automorphism for all $n \ge 4$ (see Proposition 4). Furthermore, for $n \ge k+2 \ge 5$, we may prove that ${\textrm{Aut}}(G_{n,k})$ is generated by $T_{n,k}$ and one more non-tame automorphism.

Theorem 1

For positive integers n and k, with $n \ge k+2\ge 4$, let $G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})[F^{\prime \prime }_{n},~_{k}F_{n}]$ be freely generated by the set $\{y_{1}, \ldots , y_{n}\}$. Then, ${\textrm{Aut}}(G_{n,k}) = \langle T_{n,k}, {\overline{w}}\rangle$, where ${\overline{w}}$ is the non-tame automorphism of $G_{n,k}$ satisfying the conditions ${\overline{w}}(y_{1}) = y_{1}[y_{1}, y_{2}, [y_{3}, y_{4}]]$ and ${\overline{w}}_{k}(y_{j}) = y_{j}$, $j = 2, \ldots , n$.

Remark 1

 

• (1) The group $G_{n,1} = F_{n}/[F^{\prime \prime }_{n}, F_{n}]$ is a free centre-by-metabelian group of rank n. For $n \ge 4$, it has been proved in [10] that ${\textrm{Aut}}(G_{n,1})$ is generated by the tame automorphisms and one more automorphism which, by [5], is non-tame.

• (2) Let $F_{\infty }$ be the free group of infinite rank and, for a positive integer k, let $G_{\infty , k} = F_{\infty }/\gamma _{3}(F_{\infty }^{\prime })[F_{\infty }^{\prime \prime }, ~_{k}F_{\infty }]$. It follows from [4, Theorem 2] that every automorphism of $G_{\infty , k}$ is tame.

2 Preliminaries

For positive integers c and n, with $n \ge 2$, let $V_{c}$ be the variety of all nilpotent of class c-by-abelian groups and let $G_{n,V_{c}} = F_{n}/V_{c}(F_{n})$, that is, $G_{n,V_{c}} = F_{n}/\gamma _{c+1}(F_{n}^{\prime })$. For $c = 1$, we write $M_{n}$ for $G_{n,V_{1}}$, that is, $M_{n} = F_{n}/F_{n}^{\prime \prime }$ is a free metabelian group of rank n. For the rest of the paper, we write $G_{n}$ for $G_{n,V_{2}}$ and $x_{i} = f_{i}V_{2}(F_{n})$ for $i = 1, \ldots , n$. Thus, $G_{n}$ is a free nilpotent of class 2-by-abelian group freely generated by the set $\{x_{1}, \ldots , x_{n}\}$. Furthermore, we write $T_{n}$ for the group $T_{G_{n,V_{2}}}$ of the tame automorphisms of $G_{n}$. By results of [1, 3, 10], it is known that ${\textrm{Aut}}(G_{n})$ is not finitely generated for $2 \le n \le 3$.

For positive integers n and c, with $n \ge 4$ and $c \ge 2$, let W be a variety of groups, with $V_{1} \subseteq W \subseteq V_{c}$, and let $W_{n}$ be a relatively free group of finite rank n in W. For the proof of the following result, see [3].

Lemma 1

For positive integers n and c, with $n, c \ge 2$, let W be a variety of groups, with $V_{1} \subseteq W \subseteq V_{c}$, and let $W_{n}$ be a relatively free group of finite rank n in W. Then:

• (1) An endomorphism $\phi$ of $F_{n}$ induces an automorphism of $W_{n}$ if and only if $\phi$ induces an automorphism of the free metabelian group $M_{n}$.

• (2) If every automorphism of $M_{n}$ is induced by an automorphism of $F_{n}$, then every automorphism of $W_{n}$ is a product of a tame automorphism and an automorphism which induces the identity mapping on $W_{n}/W^{\prime \prime }_{n}$.

Since, by [2], every automorphism of $M_{n}$, with $n \ge 4$, is induced by an automorphism of $F_{n}$, the next result follows from Lemma 1.

Corollary 1

For positive integers n and k, with $n \ge 4$, let $M_{n}$ be a free metabelian group of rank n, let $G_{n} = F_{n}/\gamma _{3}(F^{\prime }_{n})$ and let $G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})\times [F^{\prime \prime }_{n},~_{k}F_{n}]$. Then:

• (1) An endomorphism of $G_{n}$ is an automorphism of $G_{n}$ if and only if it induces an automorphism of the free metabelian group $M_{n}$.

• (2) An endomorphism of $G_{n,k}$ is an automorphism of $G_{n,k}$ if and only if it induces an automorphism of the free metabelian group $M_{n}$.

• (3) The natural group epimorphism from $G_{n}$ onto $G_{n,k}$ induces a group epimorphism from ${\textrm{Aut}}(G_{n})$ onto ${\textrm{Aut}}(G_{n,k})$.

Let $\psi _{1}$ and $\psi _{2}$ be the endomorphisms of $G_{n}$ satisfying the conditions $\psi _{1}(x_{i}) = x_{i}u_{i}$, $i = 1, \ldots , n$, for some $u_{1}, \ldots , u_{n} \in G_{n}^{\prime \prime }$ and $\psi _{2}(x_{i}) = x_{i}v_{i}$, $i = 1, \ldots , n$, for some $v_{1}, \ldots , v_{n} \in G_{n}^{\prime \prime }$. By Corollary 1 (2), $\psi _{1}$ and $\psi _{2}$ are automorphisms of $G_{n}$. We point out that, by [9, section 2], $\psi _{1}(w) = w$ for all $w \in G_{n}^{\prime \prime }$ and $\psi _{1}\psi _{2}(x_{i}) = x_{i}u_{i}v_{i}$, $i = 1, \ldots , n$.

Let $\phi _{1}$ and $\phi _{2}$ be the endomorphisms of $G_{n,k}$ satisfying the conditions $\phi _{1}(y_{i}) = y_{i}u_{i}$, $i = 1, \ldots , n$, for some $u_{1}, \ldots , u_{n} \in G_{n,k}^{\prime \prime }$ and $\phi _{2}(y_{i}) = y_{i}v_{i}$, $i = 1, \ldots , n$, for some $v_{1}, \ldots , v_{n} \in G_{n,k}^{\prime \prime }$. By Corollary 1 (3), $\phi _{1}$ and $\phi _{2}$ are automorphisms of $G_{n,k}$. Since $\gamma _{3}(G_{n,k}^{\prime }) = \{1\}$, it follows from the above observations that $\phi _{1}(w) = w$ for all $w \in G_{n,k}^{\prime \prime }$ and $\phi _{1}\phi _{2}(y_{i}) = y_{i}u_{i}v_{i}$, $i = 1, \ldots , n$. We point out that these facts will be used repeatedly in Sect. 4 without further comment.

3 The Automorphism Group of $G_{n}$

3.1 Identities in Nilpotent-By-Abelian Groups

For a group G and elements a and b in G, we write $a^{b} = b^{-1}ab$. For positive integers k and l, with $k \ge l$, we write $\left( {\begin{array}{c}k\\ l\end{array}}\right) = \frac{k!}{l!(k-l)!}$.

We prove some identities in nilpotent-by-abelian groups, which we need in the following. For many other identities in nilpotent-by-abelian groups, see [9, Lemma 2.1].

Lemma 2

Let G be a group such that $\gamma _{3}(G) = \{1\}$. Then:

• (1) For any $g_{1}, \ldots , g_{r} \in G$, with $r \ge 3$, and $u \in G^{\prime \prime }$,

  $$\begin{aligned} {[}u, g_{1}g_{2}, g_{3}, \ldots , g_{r}] = [u, g_{2}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{2}, g_{3}, \ldots , g_{r}]. \end{aligned}$$

• (2) For any $g_{1}, \ldots , g_{r} \in G$, with $r \ge 2$, $u \in G^{\prime \prime }$ and $\pi$ a permutation of $\{1, \ldots , r\}$,

  $$\begin{aligned} {[}u, g_{1}, \ldots , g_{r}] = [u, g_{\pi (1)}, \ldots , g_{\pi (r)}]. \end{aligned}$$

• (3) For any $g \in G$ and $u \in G^{\prime \prime }$, $[u, g^{m}] = \prod _{k=1}^{m}[u, ~_{k}g]^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }$ for all $m \ge 1$.

Proof

 

• (1) We use induction on r. Let $r = 3$. By the group commutator identities $[ab,c] = [a,c]^{b}[a,c] = [a,c][a,c,b][b,c]$ and $[a,bc] = [a,c][a,b]^{c} = [a,c][a,b][a,b,c]$ and since $\gamma _{3}(G) = \{1\}$,

  $$\begin{aligned} \begin{aligned} {[}u, g_{1}g_{2}, g_{3}]&= [[u, g_{2}][u, g_{1}][u, g_{1}, g_{2}], g_{3}]\\&= [u, g_{2}, g_{3}]^{[u, g_{1}][u, g_{1}, g_{2}]}[u, g_{1}, g_{3}]^{[u, g_{1}, g_{2}]}[u, g_{1}, g_{2}, g_{3}]\\&= [u, g_{2}, g_{3}][u, g_{1}, g_{3}][u, g_{1}, g_{2}, g_{3}], \end{aligned} \end{aligned}$$

  that is, the result is true for $r = 3$. Assume that the result is true for some $r \ge 3$, that is, assume that

  $$\begin{aligned} {[}u, g_{1}g_{2}, g_{3}, \ldots , g_{r}] = [u, g_{2}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{2}, g_{3}, \ldots , g_{r}] \end{aligned}$$

  for some $r \ge 3$. Then, by our inductive argument,

  $$\begin{aligned}{} & {} {[}u, g_{1}g_{2}, g_{3}, \ldots , g_{r}, g_{r+1}] \\{} & {} \quad = [[u, g_{2}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{3}, \ldots , g_{r}][u, g_{1}, g_{2}, g_{3}, \ldots , g_{r}], g_{r+1}]. \end{aligned}$$

  Hence, by similar arguments as in the case $r = 3$,

  $$\begin{aligned}{} & {} {[}u, g_{1}g_{2}, g_{3}, \ldots , g_{r+1}] \\{} & {} \quad = [u, g_{2}, g_{3}, \ldots , g_{r+1}][u, g_{1}, g_{3}, \ldots , g_{r+1}][u, g_{1}, g_{2}, g_{3}, \ldots , g_{r+1}] \end{aligned}$$

  and so, we obtain the required result.

• (2) By the Hall-Witt identity in the form $[a, b, c^{a}][c, a, b^{c}][b, c, a^{b}] = 1$, we have (for $a = u$, $b = g_{1}$ and $c = g_{2}$) $[u, g_{1}, g_{2}^{u}][g_{2}, u, g_{1}^{g_{2}}][g_{1}, g_{2}, u^{g_{1}}] = 1$. Since $\gamma _{3}(G) = \{1\}$, we get

  $$\begin{aligned} {[}u, g_{1}, g_{2}^{u}]= & {} [u, g_{1}, g_{2}[g_{2}, u]] = [u, g_{1}, g_{2}], \\ {[}g_{2}, u, g_{1}^{g_{2}}]= & {} [g_{2}, u, g_{1}[g_{1},g_{2}]] = [g_{2}, u, g_{1}] = [[u, g_{2}]^{-1}, g_{1}] = [u, g_{2}, g_{1}]^{-1} \end{aligned}$$

  and

  $$\begin{aligned} {[}g_{1}, g_{2}, u^{g_{1}}] = 1. \end{aligned}$$

  Hence, it follows from the Hall-Witt identity that $[u, g_{1}, g_{2}] = [u, g_{2}, g_{1}]$ and the required result may be easily deduced.

• (3) We use induction on m. For $m = 1$, the result is trivially true and so, we may assume that the result is true for some $m \ge 1$. That is, we assume that for some $m \ge 1$, $[u, g^{m}] = \prod _{k=1}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}$ for all $g \in G_{n}$ and $u \in G_{n}^{\prime \prime }$. Since

  $$\begin{aligned}{}[u, g^{m+1}] = [u, g^{m}g] = [u, g][u, g^{m}]^{g}, \end{aligned}$$

  by our inductive argument we get

  $$\begin{aligned} \begin{aligned} {[}u, g^{m+1}]&= [u, g]\prod _{k=1}^{m}([u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }})^{g} = [u, g]\prod _{k=1}^{m}([u, ~_{k}g]^{g})^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\\&= [u, g]\prod _{k=1}^{m}([u, ~_{k}g][u, ~_{k+1}g])^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }} . \end{aligned} \end{aligned}$$

  Since $\gamma _{3}(G^{\prime }) = \{1\}$, the elements $[u, ~_{k}g]$ and $[u, ~_{k+1}g]$ commute and hence, by the above equation,

  $$\begin{aligned} \begin{aligned} {[}u, g^{m+1}]&= [u, g]\prod _{k=1}^{m}\left( [u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}[u, ~_{k+1}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\right) \\ {}&= [u, g]\prod _{k=1}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\prod _{k=1}^{m}[u, ~_{k+1}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }} \\&= [u, g]\prod _{k=1}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) }}\prod _{k=2}^{m+1}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k-1\end{array}}\right) }}\\&= [u, g][u,g]^{^{\left( {\begin{array}{c}m\\ 1\end{array}}\right) }}\prod _{k=2}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m\\ k\end{array}}\right) +\left( {\begin{array}{c}m\\ k-1\end{array}}\right) }}[u, ~_{m+1}g]^{^{\left( {\begin{array}{c}m\\ m\end{array}}\right) }}. \end{aligned} \end{aligned}$$

  Since $\left( {\begin{array}{c}m\\ k\end{array}}\right) +\left( {\begin{array}{c}m\\ k-1\end{array}}\right) = \left( {\begin{array}{c}m+1\\ k\end{array}}\right)$, we get

  $$\begin{aligned} \begin{aligned} {[}u, g^{m+1}]&= [u, g]^{m+1}\prod _{k=2}^{m}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m+1\\ k\end{array}}\right) }}[u, ~_{m+1}g] = \prod _{k=1}^{m+1}[u, ~_{k}g]^{^{\left( {\begin{array}{c}m+1\\ k\end{array}}\right) }} \end{aligned} \end{aligned}$$

  and so, it follows that the result is true for all $m \ge 1$.

$\square$

3.2 Generating Sets for $\textrm{Aut}(G_{n})$

For non-negative integers m and k, let $a_{m,k}$ and $\beta _{m}$ be the endomorphisms of $G_{n}$ satisfying the conditions

$$\begin{aligned} \begin{aligned} a_{(m,k)}(x_{1})&= x_{1}[[x_{1}, x_{2}]^{x_{1}^{m}x_{2}^{k}}, [x_{3}, x_{4}]], a_{(m,k)}(x_{j}) = x_{j}, j = 2, \ldots , n,\\ \beta _{m}(x_{1})&= x_{1}[[x_{1}, x_{2}], [x_{3},x_{4}],x_{2}^{m}], \beta _{m}(x_{j}) = x_{j}, j = 2, \ldots , n. \end{aligned} \end{aligned}$$

We point out that, by Corollary 1 (1), $a_{(m,k)}$ and $\beta _{m}$ are automorphisms of $G_{n}$ for all $m, k \ge 0$.

It has been proved in [6, Theorem 1] that ${\textrm{Aut}}(G_{4}) = \langle T_{4}, a_{(m,0)}: m \ge 0\rangle$. Furthermore, it has been proved in [6, Lemma 3 (i)] that $\langle T_{4}, a_{(m,0)}: m \ge 0\rangle = \langle T_{4}, a_{(0,m)}\rangle$ and in [6, Corollary] that $\langle T_{4}, a_{(0,m)}: m \ge 0\rangle = \langle T_{4}, a_{(0,0)}, \beta _{m}: m \ge 1\rangle$. Since the arguments given in [6, proof of Lemma 3 (i)] and in [6, proof of Corollary] are valid for all $n \ge 4$, we get the following result.

Corollary 2

For a positive integer n, with $n \ge 4$, let $G_{n} = F_{n}/\gamma _{3}(F_{n}^{\prime })$. Then,

$$\begin{aligned} \langle T_{n}, a_{(m,0)} : m \ge 0\rangle = \langle T_{n}, a_{(0,m)} : m \ge 0\rangle = \langle T_{n}, a_{(0,0)}, \beta _{m} : m \ge 1\rangle . \end{aligned}$$

In [9], it has been given a generating set for ${\textrm{Aut}}(G_{n})$, independent upon n, for all $n \ge 4$. In particular, the next result has been proved in [9, proof of Theorem 1.1].

Proposition 1

For a positive integer n, with $n \ge 4$, let $G_{n} = F_{n}/\gamma _{3}(F_{n}^{\prime })$. Then, ${\textrm{Aut}}(G_{n}) = \langle T_{n}, \alpha _{(m,0)}: m \ge 0\rangle$.

We give a different generating set for ${\textrm{Aut}}(G_{n})$, which we need for the proof of Theorem 1.

Proposition 2

For a positive integer n, with $n \ge 4$, let $G_{n} = F_{n}/\gamma _{3}(F_{n}^{\prime })$. Then, ${\textrm{Aut}}(G_{n}) = \langle T_{n}, w_{m}: m \ge 0\rangle$, where $w_{m}$ is the automorphism of $G_{n}$ satisfying the conditions $w_{m}(x_{1}) = x_{1}[[x_{1}, x_{2}], [x_{3},x_{4}], ~_{m}x_{2}]$ and $w_{m}(x_{j}) = x_{j}$, $j = 2, \ldots , n$.

Proof

It follows from Proposition 1 and Corollary 2 that ${\textrm{Aut}}(G_{n}) = \langle T_{n}, a_{(0,0)}, \beta _{m}: m \ge 1\rangle$. We point out that, by Corollary 1 (1), $w_{m}$ is an automorphism of $G_{n}$ for all $m \ge 0$. Hence, to complete the proof, it suffices to show that $\alpha _{(0, 0)}, \beta _{k} \in \langle T_{n}, w_{m}: m \ge 0\rangle$ for all $k \ge 1$. Observe that $\alpha _{(0,0)} = w_{0} \in \langle T_{n}, w_{m}: m \ge 0\rangle$. Fix some $k \ge 1$. Since, by Lemma 2 (3),

$$\begin{aligned} {[[}x_{1}, x_{2}], [x_{3},x_{4}],x_{2}^{k}] = \prod _{t=1}^{k}[[x_{1}, x_{2}], [x_{3},x_{4}], ~_{t}x_{2}]^{\left( {\begin{array}{c}k\\ t\end{array}}\right) } \end{aligned}$$

and for all $m \ge 1$ the automorphisms $w_{m}$ commute, it follows that

$$\begin{aligned} \begin{aligned} \beta _{k}(x_{1})&= x_{1}[[x_{1}, x_{2}], [x_{3},x_{4}],x_{2}^{k}] = x_{1}\prod _{t=1}^{k}[[x_{1}, x_{2}], [x_{3},x_{4}], ~_{t}x_{2}]^{\left( {\begin{array}{c}k\\ t\end{array}}\right) }\\ {}&= \prod _{t=1}^{k}w_{t}^{\left( {\begin{array}{c}k\\ t\end{array}}\right) }(x_{1}). \end{aligned} \end{aligned}$$

Furthermore, $\beta _{k}(x_{j}) = \prod _{t=1}^{k}w_{t}^{\left( {\begin{array}{c}k\\ t\end{array}}\right) }(x_{j}) = x_{j}$ for all $j = 2, \ldots , n$. Therefore, $\beta _{k} = \prod _{t=1}^{k}w_{t}^{\left( {\begin{array}{c}k\\ t\end{array}}\right) }$. Hence, $\beta _{k} \in \langle T_{n}, w_{m}: m \ge 0\rangle$ and so, we get the required result. $\square$

4 The Automorphism Group of $G_{n,k}$

4.1 Notation

For positive integers n and k, with $n \ge 4$ and $k \ge 2$, recall that $G_{n,k} = F_{n}/\gamma _{3}(F_{n}^{\prime })[F^{\prime \prime },~_{k}F_{n}]$ is freely generated by the set $\{y_{1}, \ldots , y_{n}\}$. From now on, we use the following notation. For $i, j \in \{1, \ldots , n\}$, with $i \ne j$, let $\sigma _{i,j}$ and $\tau _{i,j}$ be the automorphisms of $G_{n,k}$ satisfying the conditions

$$\begin{aligned} \sigma _{i,j}(y_{i})= & {} y_{j}, \sigma _{i,j}(y_{j}) = y_{i}, \sigma _{i,j}(y_{r}) = y_{r} ~\text {for all }r \in \{1, \ldots , n\} \setminus \{i, j\}, \\ \tau _{i,j}(y_{i})= & {} y_{i}y_{j}, \tau _{i,j}(y_{r}) = y_{r} ~\text {for all }r \in \{1, \ldots , n\} \setminus \{i\}. \end{aligned}$$

Let $\rho _{1}$ be the automorphism of $G_{n,k}$ satisfying the conditions

$$\begin{aligned} \rho _{1}(y_{1}) = y_{2}y_{1}, \rho _{1}(y_{r}) = y_{r}, r = 2, \ldots , n. \end{aligned}$$

It is readily verified that $\sigma _{i,j}$, $\tau _{i,j}$ and $\rho _{1}$ are tame automorphisms of $G_{n,k}$.

We write $\rho _{2}$ for the endomorphism of $G_{n,k}$ satisfying the conditions

$$\begin{aligned} \rho _{2}(y_{1}) = y_{1}^{-1}, \rho _{2}(y_{2}) = y_{2}y_{1}, \rho _{2}(y_{r}) = y_{r}, r = 3, \ldots , n. \end{aligned}$$

We point out that $\rho _{2}$ is a tame automorphism of $G_{n,k}$: it is straightforward to verify that $\rho _{2} = \sigma _{1,2}\tau _{1,2}\sigma _{1,2}\mu$, where $\mu$ is the tame automorphism of $G_{n,k}$ satisfying the conditions $\mu (y_{1}) = y_{1}^{-1}$, $\mu (y_{r}) = y_{r}$, $r = 2, \ldots , n$, and $\rho _{2}^{-1} = \rho _{2}$.

We write ${\overline{w}}$ for the endomorphism of $G_{n,k}$ satisfying the conditions

$$\begin{aligned} {\overline{w}}(y_{1}) = y_{1}[y_{1}, y_{2}, [y_{3}, y_{4}]], {\overline{w}}(y_{j}) = y_{j}, j = 2, \ldots , n. \end{aligned}$$

We point out that, by Corollary 1 (2), ${\overline{w}}$ is an automorphism of $G_{n,k}$.

Let $j \in \{1, \ldots , k-1\}$. For positive integers $m_{1}, \ldots , m_{j}$ and $i_{1}, \ldots , i_{j} \in \{1, \ldots , n\}$, with $i_{1}< i_{2}< \ldots < i_{j}$, we write ${\overline{w}}_{(y_{i_{1}}^{m_{1}}y_{i_{2}}^{m_{2}}\ldots y_{i_{j}}^{m_{j}})}$ for the endomorphism of $G_{n,k}$ satisfying the conditions

$$\begin{aligned} \begin{aligned} {\overline{w}}_{(y_{i_{1}}^{m_{1}}y_{i_{2}}^{m_{2}}\ldots y_{i_{j}}^{m_{j}})}(y_{1})&= y_{1}[[y_{1},y_{2}], [y_{3}, y_{4}], ~_{m_{1}}y_{i_{1}}, \ldots , ~_{m_{j}}y_{i_{j}}],\\ {\overline{w}}_{(y_{i_{1}}^{m_{1}}y_{i_{2}}^{m_{2}}\ldots y_{i_{j}}^{m_{j}})}(y_{r})&= y_{r}, r = 2, \ldots , n. \end{aligned} \end{aligned}$$

In the case that some $m_{r} \in \{m_{1}, \ldots , m_{j}\}$ is 1, we write ${\overline{w}}_{(y_{i_{1}}^{m_{1}}\ldots y_{i_{r-1}}^{m_{r-1}}y^{}_{i_{r}}y_{i_{r+1}}^{m_{r+1}}\ldots y_{i_{j}}^{m_{j}})}$ instead of ${\overline{w}}_{(y_{i_{1}}^{m_{1}}\ldots y_{i_{r-1}}^{m_{r-1}}y_{i_{r}}^{^{1}}y_{i_{r+1}}^{m_{r+1}}\ldots y_{i_{j}}^{m_{j}})}$. We point out that, by Corollary 1 (2), ${\overline{w}}_{(y_{i_{1}}^{m_{1}}y_{i_{2}}^{m_{2}}\ldots y_{i_{j}}^{m_{j}})}$ is an automorphism of $G_{n,k}$. For our convenience, for a positive integer m, we write ${\overline{w}}_{m}$ instead of ${\overline{w}}_{(y_{2}^{m})}$, that is, ${\overline{w}}_{m}$ satisfies the conditions

$$\begin{aligned} {\overline{w}}_{m}(y_{1}) = y_{1}[[y_{1},y_{2}], [y_{3}, y_{4}], ~_{m}y_{2}], {\overline{w}}_{m}(y_{r}) = y_{r}, r = 2, \ldots , n. \end{aligned}$$

We point out that for $m \ge k$, ${\overline{w}}_{m}$ is the identity automorphism of $G_{n,k}$.

4.2 A Generating Set of ${\textrm{Aut}}(G_{n,k})$

By Corollary 1 (3), the natural group epimorphism from $G_{n}$ onto $G_{n,k}$ induces a group epimorphism, say $\phi _{n,k}$, from ${\textrm{Aut}}(G_{n})$ onto ${\textrm{Aut}}(G_{n,k})$. Since for $n \ge 4$, by Proposition 1, ${\textrm{Aut}}(G_{n}) = \langle T_{n}, \alpha _{(m,0)}: m \ge 0\rangle$, it follows that ${\textrm{Aut}}(G_{n,k}) = \langle \phi _{n,k}(T_{n}), \phi _{n,k}(\alpha _{(m,0)}): m \ge 0\rangle$. Hence, we have an infinite generating set for ${\textrm{Aut}}(G_{n,k})$ which is independent upon n. Our first aim is to give a finite generating set for ${\textrm{Aut}}(G_{n,k})$, with $n \ge 4$, which we need in the following.

Proposition 3

For positive integers n and k, with $n \ge 4$ and $k \ge 2$, let $G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})[F^{\prime \prime }_{n},~_{k}F_{n}]$ be freely generated by the set $\{y_{1}, \ldots , y_{n}\}$. Then, ${\textrm{Aut}}(G_{n,k}) = \langle T_{n,k}, {\overline{w}}, {\overline{w}}_{m}: m = 1, \ldots , k-1\rangle$.

Proof

Since $\phi _{n,k}$ is a group epimorphism from ${\textrm{Aut}}(G_{n})$ onto ${\textrm{Aut}}(G_{n,k})$ and, by Proposition 2, ${\textrm{Aut}}(G_{n}) = \langle T_{n}, w_{m}: m \ge 0\rangle$, we get ${\textrm{Aut}}(G_{n,k}) = \langle \phi _{n,k}(T_{n}), \phi _{n,k}(w_{m}): m \ge 0\rangle$. Since, by our notation, $\phi _{n,k}(T_{n}) = T_{n, k}$, $\phi _{n,k}(w_{0}) = {\overline{w}}$, $\phi _{n,k}(w_{m}) = {\overline{w}}_{m}$ for $m = 1, \ldots , k-1$ and $\phi _{n,k}(w_{m})$ is the identity automorphism for $k \ge m$, we have the required result. $\square$

4.3 The Case $k = 2$

Since $\gamma _{3}(F_{n}^{\prime }) \subseteq [F_{n}^{\prime \prime }, F_{n}, F_{n}]$, $G_{n,2} = F_{n}/[F_{n}^{\prime \prime }, F_{n}, F_{n}]$, that is, $G_{n,2}$ is a free centre-by-(centre-by-metabelian) group of rank n.

Proposition 4

For a positive integer n, with $n \ge 4$, let $G_{n,2} = F_{n}/[F_{n}^{\prime \prime }, F_{n}, F_{n}]$ be a free centre-by-(centre-by-metabelian) group of rank n freely generated by the set $\{y_{1}, \ldots , y_{n}\}$. Then, ${\textrm{Aut}}(G_{n,2}) = \langle T_{n,2}, {\overline{w}}\rangle$, where ${\overline{w}}$ is the non-tame automorphism of $G_{n,2}$ satisfying the conditions ${\overline{w}}(y_{1}) = y_{1}[[y_{1}, y_{2}], [y_{3}, y_{4}]]$ and ${\overline{w}}(y_{j}) = y_{j}$, $j = 2, \ldots , n$.

Proof

For simplicity, throughout the proof we write $A = [y_{1}, y_{2}]$ and $B = [y_{3}, y_{4}]$. It follows from Proposition 3 that ${\textrm{Aut}}(G_{n,2}) = \langle T_{n,2}, {\overline{w}}, {\overline{w}}_{1}\rangle$. Hence, to prove the required result, it suffices to show that ${\overline{w}}_{1} \in \langle T_{n,2}, {\overline{w}}\rangle$. We show at first that ${\overline{w}}_{(y_{1})} \in \langle T_{n,2}, {\overline{w}}\rangle$ (recall that, by our notation, ${\overline{w}}_{(y_{1})}$ is the automorphism of $G_{n,2}$ satisfying the conditions ${\overline{w}}_{(y_{1})}(y_{1}) = y_{1}[A, B, y_{1}]$, ${\overline{w}}_{(y_{1})}(y_{r}) = y_{r}$, $r = 2, \ldots , n$). We claim that $\sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1} = {\overline{w}}_{(y_{1})}$. By direct calculations, we get

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{1}) = [[y_{1}^{-1}, y_{2}y_{1}], B]^{-1}y_{1}[A, B]^{-1}. \end{aligned}$$

Using the standard group commutator identities $[a, bc] = [a,c][a,b]^{c}$, $[a^{-1},b] = ([a, b]^{-1})^{a^{-1}}$, $ab = ba[a,b]$ and since $\gamma _{3}(G_{n,k}^{\prime }) = \{1\}$, we get

$$\begin{aligned} \begin{aligned} {[[}y_{1}^{-1}, y_{2}y_{1}], B]^{-1}y_{1}&= [[y_{1}^{-1}, y_{2}]^{y_{1}}, B]^{-1}y_{1} = [A^{-1}, B]^{-1}y_{1}\\&= [A, B]y_{1} = y_{1}[A, B][A, B,y_{1}] = y_{1}[A, B,y_{1}][A, B]. \end{aligned} \end{aligned}$$

and hence, it follows that

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{1}) = y_{1}[A, B,y_{1}] = {\overline{w}}_{(y_{1})}(y_{1}). \end{aligned}$$

Similarly, we have

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{2}) = y_{2}[A^{-1}, B]^{-1}[[y_{1}^{-1},y_{2}y_{1}], B]. \end{aligned}$$

Since, by the above calculations, $[A^{-1}, B]^{-1} = [A,B]$ and $[[y_{1}^{-1},y_{2}y_{1}], B] = [A, B]^{-1}$, we get

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{2}) = y_{2}[A, B][A, B]^{-1} = y_{2} = {\overline{w}}_{(y_{1})}(y_{2}). \end{aligned}$$

Furthermore, it is straightforward to verify that

$$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1}(y_{r}) = {\overline{w}}_{(y_{1})}(y_{r}) = y_{r}, r = 3, \ldots , n. \end{aligned}$$

Hence, our claim is true, that is, $\sigma _{1,2}^{-1}{\overline{w}}^{-1}\sigma _{1,2}{\overline{w}}^{-1}\rho _{2}{\overline{w}}\rho _{2}^{-1} = {\overline{w}}_{(y_{1})}$, and thus, ${\overline{w}}_{(y_{1})} \in \langle T_{n,2}, {\overline{w}}\rangle$.

We show now that ${\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1} = {\overline{w}}_{1}$. By direct calculations, we get

$$\begin{aligned} {\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1}(y_{1}) = y_{1}[[A, B, y_{1}]^{-1}[[y_{2}y_{1},y_{2}], B, y_{2}y_{1}]. \end{aligned}$$

By the group commutator identities $[ab, c] = [a,c]^{b}[b,c]$ and $[a, bc] = [a, c][a,b][a, b, c]$ and since $[G_{n,2}^{\prime \prime }, G_{n,2}, G_{n,2}] = \{1\}$, we get

$$\begin{aligned} \begin{aligned}&{[}A, B, y_{1}]^{-1}[[y_{2}y_{1},y_{2}], B, y_{2}y_{1}] = [A, B, y_{1}]^{-1}[A, B, y_{2}y_{1}] \\&\quad = [A, B, y_{1}]^{-1}[A, B, y_{1}][A, B, y_{2}][A, B, y_{2},y_{1}] = [A, B, y_{2}]. \end{aligned} \end{aligned}$$

Hence,

$$\begin{aligned} {\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1}(y_{1}) = y_{1}[[A, B, y_{2}] = {\overline{w}}_{1}(y_{1}). \end{aligned}$$

Furthermore, it is straightforward to verify that

$$\begin{aligned} {\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1}(y_{r}) = {\overline{w}}_{1}(y_{r}) = y_{r}, r = 2, \ldots , n. \end{aligned}$$

Therefore, ${\overline{w}}_{(y_{1})}^{-1}\rho _{1}{\overline{w}}_{(y_{1})}\rho _{1}^{-1} = {\overline{w}}_{1}$. Since, as shown above, ${\overline{w}}_{(y_{1})} \in \langle T_{n,2}, {\overline{w}}\rangle$, it follows that ${\overline{w}}_{1} \in \langle T_{n,2}, {\overline{w}}\rangle$. We point out that, by [6, proof of Theorem 2], the automorphism ${\overline{w}}$ is non-tame. $\square$

4.4 The Case $k \ge 3$

From now on, let n and k be positive integers, with $n \ge k+2\ge 5$. The next result is a crucial step for the proof of Theorem 1.

Proposition 5

For positive integers n and k, with $n \ge k+2\ge 5$, let $G_{n,k} = F_{n}/\gamma _{3}(F^{\prime }_{n})[F^{\prime \prime }_{n},~_{k}F_{n}]$ be freely generated by the set $\{y_{1}, \ldots , y_{n}\}$ and let $H = \langle T_{n.k}, {\overline{w}}\rangle$, where ${\overline{w}}$ is the automorphism of $G_{n,k}$ satisfying the conditions ${\overline{w}}(y_{1}) = y_{1}[[y_{1}, y_{2}], [y_{3}, y_{4}]]$ and ${\overline{w}}(y_{j}) = y_{j}$, $j = 2, \ldots , n$. Then:

• (1) ${\overline{w}}_{(y_{5})} \in H$.

• (2) ${\overline{w}}_{(y_{5}\ldots y_{j})} \in H$ for all $j = 5, \ldots , k+2$.

• (3) ${\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})} \in H$.

• (4) ${\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})} \in H$ for all $r = 1, \ldots , k-2$.

• (5) ${\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{j})} \in H$ for all $r = 1, \ldots , k-2$ and $j = 5, \ldots , k+2$, with $6 \le r+j \le k+3$.

Proof

Throughout the proof, we write $A = [y_{1}, y_{2}]$, $B = [y_{3}, y_{4}]$ and $W = [A, B]$.

• (1) Let $\nu$ be the tame automorphism of $G_{n,k}$ satisfying the conditions

  $$\begin{aligned} \nu (y_{3}) = y_{5}^{-1}y_{3}y_{5}, \nu (y_{4}) = y_{5}^{-1}y_{4}y_{5}, \nu (y_{r}) = y_{r}, r \in \{1, \ldots , n\} \setminus \{3, 4\}. \end{aligned}$$

  We show that ${\overline{w}}^{-1}(\nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1})( \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1}) = {\overline{w}}_{(y_{5})}$. By direct calculations, we get

  $$\begin{aligned} \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1}(y_{1}) = y_{1}[[y_{1}, y_{2}y_{5}], B^{y_{5}}]. \end{aligned}$$

  By the group commutator identities $[a, bc] = [a, c][a,b]^{c}$ and $[a^{c}, b^{c}] = [a,b][a,b,c]$ and since $\gamma _{3}(G_{n,k}^{\prime }) = \{1\}$, we get

  $$\begin{aligned} \begin{aligned} {[[}y_{1}, y_{2}y_{5}], B^{y_{5}}]&= [[y_{1},y_{5}]A^{y_{5}}, B^{y_{5}}] = [[y_{1},y_{5}], B^{y_{5}}][A^{y_{5}}, B^{y_{5}}]\\ {}&= [[y_{1},y_{5}], B^{y_{5}}]W[W, y_{5}] \end{aligned} \end{aligned}$$

  and hence,

  $$\begin{aligned} \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1}(y_{1}) = y_{1}[[y_{1},y_{5}], B^{y_{5}}]W[W, y_{5}]. \end{aligned}$$

  Furthermore, by direct calculations, we get

  $$\begin{aligned} \nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1}(y_{1}) = y_{1}[[y_{1},y_{5}], B^{y_{5}}]^{-1}. \end{aligned}$$

  Hence,

  $$\begin{aligned} \begin{aligned}&{\overline{w}}^{-1}(\nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1})( \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1})(y_{1}) \\&\quad =y_{1}W^{-1}[[y_{1},y_{5}], B^{y_{5}}]^{-1}[[y_{1},y_{5}], B^{y_{5}}]W[W, y_{5}] = y_{1}[W, y_{5}] = {\overline{w}}_{(y_{5})}(y_{1}). \end{aligned} \end{aligned}$$

  Furthermore, it is straightforward to verify that

  $$\begin{aligned} {\overline{w}}^{-1}(\nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1})( \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1})(y_{r}) = {\overline{w}}_{(y_{5})}(y_{r}) = y_{r}, r = 2, \ldots , n. \end{aligned}$$

  Therefore, ${\overline{w}}^{-1}(\nu \sigma _{2,5}{\overline{w}}^{-1}\sigma _{2,5}^{-1}\nu ^{-1})( \nu \tau _{2,5}{\overline{w}}\tau _{2,5}^{-1}\nu ^{-1}) = {\overline{w}}_{(y_{5})}$ and hence, ${\overline{w}}_{(y_{5})} \in H$.

• (2) We use induction on j. For $j = 5$, the result has been proved in Proposition 5 (1). Assume that the result is true for some $5 \le j < k+2$, that is, assume that ${\overline{w}}_{(y_{5}\ldots y_{j})} \in H$. We prove that

  $$\begin{aligned} \begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1} = \sigma _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\sigma _{j,j+1}^{-1}{\overline{w}}_{(y_{5}\ldots y_{j+1})}. \end{aligned} \end{aligned}$$

  We show the above equation for $j > 5$. The case $j = 5$ may be treated similarly. By direct calculations, we get

  $$\begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1}(y_{1}) \!=\! y_{1}[W, y_{5}, \ldots , y_{j}]^{-1}[W, y_{5}, \ldots , y_{j-1}, y_{j}y_{j+1}]. \end{aligned}$$

  Using the standard group commutator identity $[a, bc] = [a,c][a,b][a,b,c]$ and since $\gamma _{3}(G_{n,k}^{\prime }) = \{1\}$, we get

  $$\begin{aligned} \begin{aligned}&{[}W, y_{5}, \ldots , y_{j}]^{-1}[[W, y_{5}, \ldots , y_{j-1}, y_{j}y_{j+1}] \\&= {[}W, y_{5}, \ldots , y_{j}]^{-1}[W, y_{5}, \ldots , y_{j-1}, y_{j+1}][W, y_{5}, \ldots , y_{j}][W, y_{5}, \ldots , y_{j+1}] \\&= {[}W, y_{5}, \ldots , y_{j-1}, y_{j+1}][W, y_{5}, \ldots , y_{j+1}]. \end{aligned} \end{aligned}$$

  Hence,

  $$\begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1}(y_{1}) = y_{1}[W, y_{5}, \ldots , y_{j-1}, y_{j+1}][W, y_{5}, \ldots , y_{j+1}]. \end{aligned}$$

  It is straightforward to verify that

  $$\begin{aligned} \begin{aligned} \sigma _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\sigma _{j,j+1}^{-1}{\overline{w}}_{(y_{5}\ldots y_{j+1})}(y_{1})&= [W, y_{5}\ldots y_{j-1}, y_{j+1}][W, y_{5}\ldots y_{j+1}] \\&= {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1}(y_{1}) \end{aligned} \end{aligned}$$

  and

  $$\begin{aligned} \begin{aligned} \sigma _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\sigma _{j,j+1}^{-1}{\overline{w}}_{(y_{5}\ldots y_{j+1})}(y_{r})&= {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1}(y_{r}) \\ {}&= y_{r}, r = 2, \ldots , n. \end{aligned} \end{aligned}$$

  Hence,

  $$\begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{j})}^{-1}\tau _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\tau _{j,j+1}^{-1} = \sigma _{j,j+1}{\overline{w}}_{(y_{5}\ldots y_{j})}\sigma _{j,j+1}^{-1}{\overline{w}}_{(y_{5}\ldots y_{j+1})}. \end{aligned}$$

  Since, by our inductive argument, ${\overline{w}}_{(y_{5}\ldots y_{j})} \in H$, it follows from the above equation that ${\overline{w}}_{(y_{5}\ldots y_{j+1})} \in H$ and hence, we get the required result.

• (3) We claim that

  $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1} = {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}. \end{aligned}$$

  By similar arguments as in the proof of Proposition 4, we have

  $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1}(y_{1}) = y_{1}[W, y_{5}, \ldots , y_{k+2}, y_{1}]. \end{aligned}$$

  Since, by Lemma 2 (2), $[W, y_{5}, \ldots , y_{k+2}, y_{1}] = [W, y_{1}, y_{5}, \ldots , y_{k+2}]$, we get

  $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1}(y_{1}) = {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}(y_{1}). \end{aligned}$$

  By similar arguments as in the proof of Proposition 4, we have

  $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1}(y_{2}) = y_{2}. \end{aligned}$$

  Furthermore, it is straightforward to verify that

  $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1}(y_{r}) = y_{r}, r = 3, \ldots , n. \end{aligned}$$

  Therefore, our claim is true, that is,

  $$\begin{aligned} \sigma _{1,2}^{-1}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\sigma _{1,2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}^{-1}\rho _{2}{\overline{w}}_{(y_{5}\ldots y_{k+2})}\rho _{2}^{-1} = {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}. \end{aligned}$$

  Since, by Proposition 5 (2) (for $j = k+2$), ${\overline{w}}_{(y_{5}\ldots y_{k+2})} \in H$, it follows from the above equation that ${\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})} \in H$.

• (4) We use induction on r. At first, we prove the required result for $r = 1$. We show that ${\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}^{-1}\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1} = {\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})}$. By direct calculations, we get

  $$\begin{aligned} \begin{aligned} \rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{1})&= y_{1}[[y_{2}y_{1}, y_{2}], B, y_{2}y_{1}, y_{5}, \ldots , y_{k+2}]\\&= y_{1}[W, y_{2}y_{1}, y_{5}, \ldots , y_{k+2}]. \end{aligned} \end{aligned}$$

  Since $\gamma _{3}(G_{n,k}^{\prime }) = \{1\}$, it follows from Lemma 2 (1) that

  $$\begin{aligned} \begin{aligned}&\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{1}) \\&\quad =y_{1}[W, y_{1}, y_{5}, \ldots , y_{k+2}][W, y_{2}, y_{5}, \ldots , y_{k+2}][W, y_{2}, y_{1}, y_{5}, \ldots , y_{k+2}]. \end{aligned} \end{aligned}$$

  Since $[G_{n,k}^{\prime \prime }, ~_{k}G_{n,k}] = \{1\}$, we have $[W, y_{2}, y_{1}, y_{5}, \ldots , y_{k+2}] = 1$ and hence,

  $$\begin{aligned} \rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{1}) = y_{1}[W, y_{1}, y_{5}, \ldots , y_{k+2}][W, y_{2}, y_{5}, \ldots , y_{k+2}]. \end{aligned}$$

  Therefore,

  $$\begin{aligned} {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}^{-1}\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{1})= & {} y_{1}[W, y_{2}, y_{5}, \ldots , y_{k+2}] \\= & {} {\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})}(y_{1}). \end{aligned}$$

  It is straightforward to verify that

  $$\begin{aligned} {\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}^{-1}\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1}(y_{r}) = {\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})}(y_{r}) = y_{r} , r = 2, \ldots , n. \end{aligned}$$

  Therefore, ${\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}^{-1}\rho _{1}{\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})}\rho _{1}^{-1} = {\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})}$. Since, by Proposition 5 (3), ${\overline{w}}_{(y_{1}y_{5}\ldots y_{k+2})} \in H$, it follows from the above equation that ${\overline{w}}_{(y_{2}y_{5}\ldots y_{k+2})} \in H$, that is, the required result is true for $r = 1$.

  Assume that the result is true for some $r \ge 1$, that is, assume that ${\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})} \in H$ for some $1 \le r < k-2$. We show that

  $$\begin{aligned} {\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1} = {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}. \end{aligned}$$

  Since

  $$\begin{aligned} \begin{aligned}&{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1}(y_{j}) \\&= {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}(y_{j})=y_{j},j = 2, \ldots , n, \end{aligned} \end{aligned}$$

  it suffices to show that

  $$\begin{aligned} {\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1}(y_{1}) \!=\! {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}(y_{1}). \end{aligned}$$

  By similar calculations as in the proof of Proposition 5 (2) and since $[G_{n,k}^{\prime \prime }, ~_{k}G_{n,k}] = \{1\}$, we get

  $$\begin{aligned} \begin{aligned}&{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1}(y_{1}) \\ {}&\quad = y_{1}[W, ~_{r}y_{2}, y_{5}, \ldots , y_{k+2-r}, y_{2}] \end{aligned} \end{aligned}$$

  and hence, by Lemma 2 (2),

  $$\begin{aligned} \begin{aligned}&{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1}(y_{1}) \\ {}&\quad = y_{1}[W, ~_{r+1}y_{2}, y_{5}, \ldots , y_{k+2-r}] = {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}(y_{1}). \end{aligned} \end{aligned}$$

  Thus,

  $$\begin{aligned} {\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}^{-1}\tau _{k+3-r, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})}\tau _{k+3-r,2}^{-1} = {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})}. \end{aligned}$$

  Since, by our inductive argument, ${\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{k+3-r})} \in H$, it follows from the above equation that ${\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{k+3-(r+1)})} \in H$ and hence, we get the required result.

• (5) For $k = 3$ (and hence, $r = 1$ and $j = 5$), it follows from Proposition 5 (4) that ${\overline{w}}_{(y_{2}y_{5})} \in H$ and so, in what follows, we may assume that $k > 3$. We use inverse induction on $t = r+j$. For $t = k+3$ (that is, $j = k+3-r$), the required result has been proved in Proposition 5 (4). Assume that the result is true for some $6 < t \le k+3$, that is, assume that ${\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r})} \in H$ for all $r = 1, \ldots , t-5$. Let $r+j = t-1$. We consider the following cases:

  • (a) $k \ge 4$ and $t = 7$. In this case, we have to show that ${\overline{w}}_{(y_{2}y_{5})} \in H$. By similar calculations as in the proof of Proposition 5 (4), we get

    $$\begin{aligned} {\overline{w}}_{(y_{5}y_{6})}^{-1}\tau _{6, 2}{\overline{w}}_{(y_{5}y_{6})}\tau _{6,2}^{-1} = {\overline{w}}_{(y_{2}y_{5})}{\overline{w}}_{(y_{2}y_{5}y_{6})}. \end{aligned}$$

    Since, by Proposition 5 (2), ${\overline{w}}_{(y_{5}y_{6})} \in H$ and, by our inductive argument, ${\overline{w}}_{(y_{2}y_{5}y_{6})} \in H$, from the above equation we get the required result.

  • (b) $k > 4$ and $t > 7$. In this case, we have to show that ${\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})} \in H$ for all $r = 1, \ldots , t-6$. By similar calculations as in the proof of Proposition 5 (4), we get

    $$\begin{aligned} {\overline{w}}_{(y_{5}\ldots y_{t-1})}^{-1}\tau _{t-1, 2}{\overline{w}}_{(y_{5}\ldots y_{t-1})}\tau _{t-1,2}^{-1} = {\overline{w}}_{(y_{2}y_{5}\ldots y_{t-2})}{\overline{w}}_{(y_{2}y_{5}\ldots y_{t-1})} \end{aligned}$$

    and, for $r = 1, \ldots , t - 7$,

    $$\begin{aligned} \begin{aligned}&{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})}^{-1}\tau _{t-r-1, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})}\tau _{t-r-1,2}^{-1}\\ {}&\quad = {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{t-r-2)})}{\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{t-r-1})}. \end{aligned} \end{aligned}$$

    Since, by our inductive argument, ${\overline{w}}_{(y_{2}y_{5}\ldots y_{t-1})}, {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{t-r-1})} \in H$, it follows from the above equations that

    $$\begin{aligned} {\overline{w}}_{(y_{2}y_{5}\ldots y_{t-2})}^{-1}{\overline{w}}_{(y_{5}\ldots y_{t-1})}^{-1}\tau _{t-1, 2}{\overline{w}}_{(y_{5}\ldots y_{t-1})}\tau _{t-1,2}^{-1} \in H \end{aligned}$$

    (1)

    and

    $$\begin{aligned} {\overline{w}}_{(y_{2}^{r+1}y_{5}\ldots y_{t-r-2})}^{-1}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})}^{-1}\tau _{t-r-1, 2}{\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})}\tau _{t-r-1,2}^{-1} \in H. \end{aligned}$$

    (2)

    Since, by Proposition 5 (2), ${\overline{w}}_{(y_{5}\ldots y_{t-1})} \in H$, by equation (1) we get ${\overline{w}}_{(y_{2}y_{5}\ldots y_{t-2})} \in H$. Hence, by equation (2) and an obvious induction on r, we get ${\overline{w}}_{(y_{2}^{r}y_{5}\ldots y_{t-r-1})} \in H$ for all $r = 1, \ldots , t-6$.

  By the above cases, we get the required result.

$\square$

4.5 Proof of Theorem 1

We are now in position to give a proof of Theorem 1.

Proof of Theorem 1

For $k = 2$, the result has been proved in Proposition 4 and hence, we may assume that $n \ge k+2 \ge 5$. We write $H = \langle T_{n,k}, {\overline{w}}\rangle$. We point out that, by [6, proof of Theorem 2], the automorphism ${\overline{w}}$ is non-tame. Since, by Proposition 3, ${\textrm{Aut}}(G_{n,k}) = \langle T_{n,k}, {\overline{w}}, {\overline{w}}_{m}: m = 1, \ldots , k-1\rangle$, to complete the proof, it suffices to show that ${\overline{w}}_{m} \in H$ for all $m = 1, \ldots , k-1$. Assume at first that $m = 1$. By similar calculations as in the proof of Proposition 5 (2), we get

$$\begin{aligned} {\overline{w}}_{(y_{5})}^{-1}\tau _{5,2}{\overline{w}}_{(y_{5})}\tau _{5,2}^{-1} = {\overline{w}}_{1}{\overline{w}}_{(y_{2}y_{5})}. \end{aligned}$$

Since, by Proposition 5 (1), ${\overline{w}}_{(y_{5})} \in H$ and, by Proposition 5 (5) (for $r = 1$ and $j =5$), ${\overline{w}}_{(y_{2}y_{5})} \in H$, it follows from the above equation that ${\overline{w}}_{1} \in H$. Now, fix some $2 \le m \le k-1$. By similar calculations as in the proof of Proposition 5 (2), we get

$$\begin{aligned} {\overline{w}}_{(y_{2}^{m-1}y_{5})}^{-1}\tau _{5,2}{\overline{w}}_{(y_{2}^{m-1}y_{5})}\tau _{5,2}^{-1} = {\overline{w}}_{m}{\overline{w}}_{(y_{2}^{m}y_{5})}. \end{aligned}$$

Since, by Proposition 5 (5), ${\overline{w}}_{(y_{2}^{m}y_{5})} \in H$ for all $m = 2, \ldots , k-2$ and ${\overline{w}}_{(y_{2}^{k-1}y_{5})}$ is the identity mapping, it follows from the above equation that ${\overline{w}}_{m} \in H$ for all $m = 2, \ldots , k-1$. Hence, we get the required result. $\square$

4.6 Concluding Remarks

• (1) It would be interesting to study the case $4 \le n < k+2$, with $k \ge 3$. By Proposition 3, ${\textrm{Aut}}(G_{n,k}) = \langle T_{n,k}, {\overline{w}}, {\overline{w}}_{m}: m = 1, \ldots , k-1\rangle$. Write $d_{n,k}$ for the minimum number of generators of ${\textrm{Aut}}(G_{n,k})$. It would be interesting to know if there exists a number d such that $d_{n,k} \le d$ for all n and d, with $4 \le n < k+2$.

• (2) It has been conjectured by A. Papistas in [9] that ${\textrm{Aut}}(G_{n})$ is not finitely generated. If such an aforementioned number d for the minimum number of generators of ${\textrm{Aut}}(G_{n,k})$ does not exist, then the conjecture is true. (Assume that such a number d does not exist and suppose, on the contrary, that the conjecture is not true, that is, suppose that ${\textrm{Aut}}(G_{n})$ is finitely generated. Then, by Proposition 2, ${\textrm{Aut}}(G_{n})$ is generated by $T_{n}$ and a fixed finite subset of the set $\{w_{m}: m \ge 0\}$. Hence, by Proposition 3, for sufficiently large k, the number of generators of ${\textrm{Aut}}(G_{n,k})$ is independent upon n and k, which is the required contradiction.)

• (3) For positive integers n and k, with $n \ge 4$ and $k \ge 3$, it is still an open problem to find a generating set for ${\textrm{Aut}}(F_{n})/[F_{n}^{\prime \prime }, ~_{k}F_{n}]$.