Polynomial extension property in the classical Cartan domain ${\mathcal {R}_{II}}$

Abstract

In this work, it is shown that for the classical Cartan domain $\mathcal {R}_{II}$ consisting of symmetric $2\times 2$ matrices, every algebraic subset of $\mathcal {R}_{II}$, which admits the polynomial extension property, is a holomorphic retract.

1 Introduction

Denote by $\mathcal R_{II}$ the classical Cartan domain of type II, i.e.

$$\begin{aligned} \mathcal R_{II}=\left\{ A\in \mathcal M_{2\times 2}(\mathbb C) : A=A^T, \ I-A^*A>0 \right\} . \end{aligned}$$

Here $\mathcal {M}_{2\times 2}(\mathbb {C})$ stands for the space of $2\times 2$ matrices with complex elements and I is the unit matrix.

In this paper, we examine certain varieties V of $\mathcal {R}_{II}$ for which the restrictions of polynomials to V can be extended to holomorphic functions on $\mathcal {R}_{II}$ without increasing their supremum norm. The origin of that sort of studies goes back to Rudin’s book [13] and one of the goals is to determine whether such a set V is a holomorphic retract. The important results were obtained by Agler and McCarthy [2] for the subsets of the bidisc. The authors also provided motivations concerning connections of the extension property to the Nevanlinna-Pick interpolation and to the von Neumann inequality.

Generalizations of the problem on the bidisc have split in particular into studies of sets in the higher dimensional polydisc on the one hand, and on the other, in the symmetrized bidisc, a domain which is an image of the bidisc under the map $(z,w)\mapsto (z+w,zw)$. In case of the polydisc, there are known partial results concerning relatively polynomially convex (or even algebraic) subsets V, see [9, 10], and [11]. The problem in $\mathbb {D}^3$ is completely solved only with the additional assumption that the extension operator may be chosen to be linear (cf. [6]). The description of varieties admitting the extension property in the symmetrized bidisc was obtained in [1] and independently in [4].

The symmetrized bidisc, as well as the other domain, the tetrablock, are related to the $\mu$-synthesis problem (cf. [15]). These domains are also of interest for the theory of invariant distances. Recall that the latter can be described as an image $\pi (\mathcal {R}_{II})$ of a mapping $\pi (z)=(z_{11},z_{22},\det z)$, where $z=\begin{pmatrix} z_{11} &{} z_{12} \\ z_{21} &{} z_{22} \end{pmatrix}$. Therefore the Cartan domain $\mathcal {R}_{II}$ plays a similar role for the tetrablock as the bidisc plays for the symmetrized bidisc.

Cartan domains are the open unit balls of Cartan factors, which appear as important examples of $J^*$-algebras, the class that from the operator theoretic point of view may be considered as a generalization of $C^*$-algebras (see [7]). We remark that the classification of holomorphic retracts in the open unit ball of any commutative $C^*$-algebra with identity was obtained in [12].

2 Main theorem

Let V be an arbitrary subset of $\mathbb {C}^n$. We will say that a function $f:V\rightarrow \mathbb {C}$ is holomorphic on V if, for every point $\zeta \in V$, it can be extended to a holomorphic function $F\in \mathcal {O}(U)$ for U being some neighborhood of $\zeta$ in $\mathbb {C}^n$.

Let $H^\infty (V)$ denote the algebra of bounded holomorphic functions on V with the supremum norm and let P(V) be the subalgebra of polynomials of $H^\infty (V)$.

We will say that a subset V of a bounded domain U has the polynomial extension property with respect to U if, for every polynomial $p\in \mathcal {P}(V)$, there exists a bounded holomorphic function $F\in H^\infty (U)$ such that

$$\begin{aligned} F|_V=p\quad \text { and }\quad ||F||_{H^\infty (U)}=||p||_{H^\infty (V)}. \end{aligned}$$

Let m be the Möbius distance on the unit disc:

$$\begin{aligned} m(z,w):=\left| \frac{z-w}{1-z\overline{w}}\right| , \quad z,w\in \mathbb {D}. \end{aligned}$$

For U a domain in $\mathbb {C}^n$, denote by $k_U$ the Kobayashi pseudodistance:

$$\begin{aligned} k_U(z,w)=\inf \lbrace p(\lambda ,\mu ): \text { there is } \varphi \in \mathcal {O}(\mathbb {D},U) \text { such that } \varphi (\lambda )=z, \varphi (\mu )=w\rbrace . \end{aligned}$$

Here $p(\lambda ,\mu )=\text {tanh}^{-1}(m(\lambda ,\mu ))$ is the Poincaré distance on the unit disc $\mathbb {D}$ in $\mathbb {C}$. Similarly we write $c_U$ for the Carathéodory pseudodistance:

$$\begin{aligned} c_U(z,w)=\sup \lbrace p(f(z),f(w)):f\in \mathcal {O}(U,\mathbb {D})\rbrace , \quad z,w\in U. \end{aligned}$$

The Kobayashi extremal map with respect to two distinct points $z,w\in U$ is defined as a function $f \in \mathcal {O}(\mathbb {D},U)$ for which there are $\lambda , \mu \in \mathbb {D}$ such that $f(\lambda )=z, f(\mu )=w$, and

$$\begin{aligned} k_U(z,w)=p(\lambda , \mu ). \end{aligned}$$

The range of the Kobayashi extremal map is called a complex geodesic.

Similarly we define the Carathéodory extremal map, for a pair of points z and w, as a map $g\in \mathcal {O}(U,\mathbb {D})$ which satisfies

$$\begin{aligned} c_U(z,w)=p(g(z),g(w)). \end{aligned}$$

The infinitesimal version of the above, called the Carathéodory-Reiffen pseudometric, is defined as

$$\begin{aligned} \gamma _U(z;X):=\sup \lbrace |F'(z)X|: F\in H^\infty (U), ||F||\le 1, F(z)=0\rbrace . \end{aligned}$$

Remark 1

In case when a bounded domain U is convex, it follows from the Lempert theorem (see, e.g., [8, Theorem 11.2.1]) that for any pair of points in $\mathbb {D}$ and for every Kobayashi extremal f through these points, there exists a Carathéodory extremal map g being a left-inverse to f, which means that $g\circ f=\text {id}|_{\mathbb {D}}$.

Recall that a subset V of U is called a holomorphic retract if there exists a holomorphic map $r:U\rightarrow V$ such that $r|_V=\text {id}|_V$.

We call the set V algebraic in $U\subset \mathbb {C}^n$ if there are polynomials $p_1,...,p_s$ such that $V=U\cap \bigcap _{j=1}^s p_j^{-1}(0)$.

The main result of this note is:

Theorem 2

Suppose V is an algebraic subset of $\mathcal {R}_{II}$ which admits the polynomial extension property.

Then V is a holomorphic retract.

3 Tools

The group of automorphisms of $\mathcal {R}_{II}$ contains mappings of the form

$$\begin{aligned} A\mapsto UAU^T, \end{aligned}$$

where U is a unitary matrix; and the mappings (cf. [3]):

$$\begin{aligned} \Phi _A(X)=(I-AA^*)^{-\frac{1}{2}}(X-A)(I-A^*X)^{-1}(I-A^*A)^{\frac{1}{2}} \quad \text { for } X,A\in \mathcal {R}_{II}. \end{aligned}$$

Note that $\Phi _A(0)=-A$ and its inverse is given by $\Phi _A^{-1}=\Phi _{-A}$.

Definition 3

We say that two points x, y of $\mathcal {R}_{II}$ form a balanced pair if there are an automorphism $\varphi$ and a complex scalar a such that $x = \varphi (0)$ and $y = \varphi (a\cdot I)$. Obviously, $a \in \mathbb D$.

Similarly, a pair (x, v) composed of a point $x\in \mathcal {R}_{II}$ and a vector $v\in T_x^{\mathbb {C}}V$ is called infinitesimally balanced if there is an automorphism $\varphi$ of $\mathcal {R}_{II}$ such that $x=\varphi (0)$ and there exists a sequence $\lbrace a_n \rbrace$, $a_n\rightarrow 0$, such that $\frac{\phi (a_nI)}{a_n}\rightarrow \alpha v$ for some $\alpha \in \mathbb {C}$.

Before we proceed with a proof of the main theorem, we list some classical tools from complex analytic geometry that will be of use in the sequel.

Proposition 4

([5, Theorem 3.7]). Let $\mathcal {X}$ be an analytic set in $\mathbb {C}^n$ and $a\in \mathcal {X}$ with $\text {dim}_a\mathcal {X}=k$. If there is a connected neighborhood $a\in U=U'\times U''$ such that $\pi _k:U\cap \mathcal {X}\rightarrow U'\subset \mathbb {C}^k$ is proper, then there exists an analytic set $\mathcal {Y}\subset U'$, $\text {dim}\mathcal {Y}<k$, and $p\in \mathbb {N}$ such that

• 1. $\pi _k:U\cap \mathcal {X}\setminus \pi _k^{-1}(\mathcal {Y})\rightarrow U'\setminus \mathcal {Y}$ is a locally biholomorphic p-sheeted cover. In particular, $\# \pi _k^{-1}(z')\cap \mathcal {V}\cap U=p$ for all $z'\in U'\setminus \mathcal {Y}$.

• 2. $\pi _k^{-1}(\mathcal {Y})$ is nowhere dense in $\mathcal {X}_{(k)}\cap U$, where $\mathcal {X}_{(k)}=\lbrace z\in \mathcal {X}:\text {dim}_z \mathcal {X}=k\rbrace$.

Proposition 5

(Identity principle). Let $\mathcal {X},\mathcal {Y}$ be analytic, where $\mathcal X$ is additionally irreducible. If $w\in \mathcal {X}\cap \mathcal {Y}$ and the germs $(\mathcal {X})_w$ and $(\mathcal {Y})_w$ are equal, then $\mathcal {X}\subset \mathcal {Y}$.

Proposition 6

(The analytic graph theorem). Let $f:X\rightarrow Y$ be a locally bounded function, where X, Y are complex manifolds. Then f is holomorphic if and only if the graph $\lbrace (x,f(x)):x\in X\rbrace$ is analytic in $X\times Y$.

Lemma 7

([9]). Let $\Omega$ be a bounded domain, and let V be a relatively polynomially convex subset of $\Omega$ that has the polynomial extension property.

Then $\overline{V}$ is connected.

4 Proof of Theorem 2

Similarly as noted in [14], observe that despite the fact that polynomials p are not invariant under automorphisms $\varphi$ of the Cartan domain $\mathcal {R}_{II}$, the composition $p\circ \varphi$ can be uniformly approximated by polynomials on $\overline{\mathcal {R}_{II}}$. This allows us to simplify problems by moving points to the origin. In particular, considering a balanced pair (x, y), the automorphisms that map $x\mapsto 0$ and $y\mapsto \alpha \cdot I$ will be used frequently.

The following proposition comes from [9]:

Proposition 8

Let $\Omega$ be bounded, and assume that $V\subset \Omega$ has the polynomial extension property. Let $\phi$ be a Caratheodory-Pick extremal for $\Omega$ for some data. If $\phi |_V$ is in $P(\overline{V})$, then $\overline{\phi (V)}$ contains the unit circle $\mathbb {T}$.

Here P(K) denotes the uniform closure of the polynomials in C(K).

Lemma 9

Let V be relatively polynomially convex and have the polynomial extension property. If x and y are two points in V that form a balanced pair, then the whole disc $\{\varphi (\lambda \cdot I):\lambda \in \mathbb D\}$ lies in V for an appropriate automorphism $\varphi$ of the form of the Definition 3.

The same assertion holds provided that $(x,v)\in V\times T^{\mathbb C}_xV$ forms an infinitesimally balanced pair (it has sense even if V is not necessarily smooth at x).

Proof

We can choose an automorphism $\psi$ of $\mathcal {R}_{II}$ such that $\psi (x)=0$ and $\psi (y)=a\cdot I$. Then 0 and $a\cdot I$ are contained in the analytic disc $k:\mathbb {D}\rightarrow \mathcal {R}_{II}$, $k(z)=z\cdot I$. Consider the mapping $\begin{pmatrix} z_{11} &{} w \\ w &{} z_{22}\end{pmatrix} \mapsto \dfrac{z_{11}+z_{22}}{2}$. It is a left inverse to k. It follows from Proposition 8 that $\left\{ \begin{pmatrix} \zeta &{} 0 \\ 0 &{} \zeta \end{pmatrix}:\zeta \in \mathbb {T} \right\} \subset \overline{\psi (V)}$. From this and because of the relative polynomial convexity of $\psi (V)$, we have

$$\begin{aligned} \left\{ \begin{pmatrix} \lambda &{} 0 \\ 0 &{} \lambda \end{pmatrix}:\lambda \in \mathbb {D} \right\} \subset \psi (V). \end{aligned}$$

It is now enough to set $\varphi =\psi ^{-1}$.

For the infinitesimal case, it is enough to note that we can assume $x=0$ and $v\in \mathbb {C} I$. The rest of the proof goes as in the regular case. $\square$

Lemma 10

Suppose that V is algebraic and has the polynomial extension property. Let $W=V\cap \{z_{12}=0\}$. If W is neither discrete nor two-dimensional, then, up to a permutation of $z_{11}$ and $z_{22}$, it is of the form $W_f=\{\mathop {\mathrm {diag}}(x, f(x)):\ x\in \mathbb D\}$, where $f\in \mathcal O(\mathbb D, \mathbb D)$. Here (and further) $\mathop {\mathrm {diag}}$ stands for a diagonal matrix.

In the proof, we will use the following

Lemma 11

([9, Lemma 3.3]). Suppose $X\subset \mathbb {D}^2$ contains the set $\mathcal {S}=\lbrace (x,f(x)):x\in \mathbb {D}\rbrace$, where $f:\mathbb {D}\rightarrow \mathbb {D}$ is a holomorphic function satisfying $f(0)=0$, and $\mathcal {S}$ is not all of X. Then X contains a balanced pair.

Lemma 11 can be reformulated by considering X as a subset of $\mathcal {R}_{II}\cap \lbrace z_{12}=0\rbrace$ and $\mathcal {S}=\lbrace \mathop {\mathrm {diag}}(x,f(x)):x\in \mathbb {D}\rbrace$. The proof is essentially the same as in [9]. Just note that taking any point $\mathop {\mathrm {diag}}(z_1,w_1)\in X\setminus \mathcal {S}$, we can use the automorphism $\Phi _{\mathop {\mathrm {diag}}(z_1,f(z_1))}(\mathop {\mathrm {diag}}(x,y))=\mathop {\mathrm {diag}}\left( \frac{x-z_1}{1-\overline{z_1}x},\frac{y-f(z_1)}{1-\overline{f(z_1)}y}\right)$.

Proof of Lemma 10

Step 1. We are treating W as a subset of $\mathbb C^2 = \mathbb C_{z_{11}}\times \mathbb C_{z_{22}}$. First we shall show that there is an f, a holomorphic selfmapping of the unit disc, such that $W_f\subset W$. Using an automorphism, we can assume that 0 is a regular point of W, which means that W is near 0 a one-dimensional complex submanifold, i.e. it can be written as a graph $(\lambda , g(\lambda ))$ of a holomorphic function g such that $g(0)=0$, $|\lambda |<\epsilon$. Permuting $z_{11}$ and $z_{22}$, we can additionally assume that $|g'(0)|\le 1$.

If $g'(0)$ is a unimodular constant, let us say $\omega$, then according to Lemma 9, the variety $W_{f_\omega }$ is contained in W, where $f_\omega (\lambda ) = \omega \lambda$, $\lambda \in \mathbb D$.

Suppose that $|g'(0)|<1$. Let $W_1$ be an irreducible component of W containing 0. Note also that if $W\setminus \{0\}$ intersects $\{|z_{22}|=|z_{11}|\}$, we are done. So we can assume that $W_1\setminus \{0\}$ is contained in $\{|z_{22}|<|z_{11}|\}$. Let $\pi :\mathbb {C}^2 \rightarrow \mathbb C$ be the projection $(z_{11}, z_{22}) \mapsto z_{11}$, which is proper on $W_1$. From this inclusion, $W_1$ is single-sheeted near 0 (it is enough to consider the preimage $\pi ^{-1}(0)$) and by Proposition 4, it is actually single-sheeted, except for at most a discrete set of singular points. But then the describing functions are locally bounded and consequently, based on the analytic graph theorem, they are holomorphic. This means that the set of singular points in $W_1$ is empty. This gives Step 1.

Step 2. Lemma 11, together with Lemma 9, implies that W is either a graph of a holomorphic function or, up to the application of an automorphism, it contains at least two intersecting balanced geodesics of the form $\mathcal {D}_j:=\lbrace \mathop {\mathrm {diag}}(\lambda ,\omega _j\lambda ):\lambda \in \mathbb {D}\rbrace$, where $\omega _j\in \mathbb {T}$ and $j=1,2$.

Indeed, suppose $W_f\subsetneq W$, so there are points $a\in W_f$ and $b\in W\setminus W_f$ that form a balanced pair. Without loss of generality, assume $a=0$ and $b=\mathop {\mathrm {diag}}(\beta ,\omega _1 \beta )$ for some $\beta \in \mathbb {D}$ and $\omega _1\in \mathbb {T}$. This gives the first balanced disc which by assumption is not the whole W. Choose any point $\mathop {\mathrm {diag}}(x,y)\in W\setminus \mathcal {D}_1$ and an automorphism $\psi =\Phi _{\mathop {\mathrm {diag}}(x,y)}$. Then

$$\begin{aligned} \psi (\mathop {\mathrm {diag}}(x,\omega _1x))=\mathop {\mathrm {diag}}(0,w_1)\in \psi (\mathcal {D}_1),\\ \psi (\mathop {\mathrm {diag}}(\overline{\omega }_1y,y))=\mathop {\mathrm {diag}}(w_2,0) \in \psi (\mathcal {D}_1). \end{aligned}$$

By a continuity argument, there is $w_0\in \mathbb {D}$ and $\zeta \in \mathbb {T}$ such that $\mathop {\mathrm {diag}}(w_0,\zeta w_0) \in \psi (\mathcal {D}_1)$. Moreover $\psi (\mathop {\mathrm {diag}}(x,y))=0\in \psi (W\setminus \mathcal {D}_1)$. Therefore the point $\mathop {\mathrm {diag}}(x,y)$ together with $\psi ^{-1}(\mathop {\mathrm {diag}}(w_0,\zeta \omega _0))$ form a balanced pair which gives the second balanced disc.

Suppose then that W contains at least two such geodesics as above. Since W is algebraic (as an intersection of V with $p^{-1}(0)$ for $p(z)=z_{12}$), we can find a neighbourhood U of the origin in $\mathcal {R}_{II}$ such that 0 is the only singular point of $W\cap U$.

Choose $\epsilon >0$ small enough such that $\mathcal {D}_j(\epsilon ):=\lbrace \mathop {\mathrm {diag}}(\lambda , \omega _j\lambda ):\lambda \in \epsilon \mathbb {D}\rbrace \subset V\cap U$ for $j=1,2$. Fix $\lambda _0 \in \epsilon \mathbb {D}$ and apply the automorphism $\Phi _A:=\Phi _{\mathop {\mathrm {diag}}(\lambda _0,\omega _1 \lambda _0)}$. Then

$$\begin{aligned} \Phi _A(0)&=\mathop {\mathrm {diag}}(-\lambda _0,-\omega _1\lambda _0);\\ \Phi _A(\mathop {\mathrm {diag}}(\lambda _0,\omega _1 \lambda _0))&=0;\\ \Phi _A(\mathcal {D}_2)&=\mathop {\mathrm {diag}}\left( \frac{\lambda -\lambda _0}{1-\overline{\lambda _0}\lambda },\frac{\omega _2\lambda -\omega _1\lambda _0}{1-\overline{\omega _1}\omega _2\overline{\lambda _0}\lambda }\right) . \end{aligned}$$

Taking $\lambda =\lambda _0$ and $\lambda =\frac{\omega _1}{\omega _2}\lambda _0$, we get that both points $\mathop {\mathrm {diag}}\left( 0,\frac{\lambda _0(\omega _2-\omega _1)}{1-\overline{\omega _1}\omega _2|\lambda _0|^2}\right)$, $\mathop {\mathrm {diag}}\left( \frac{\lambda _0\left( \frac{\omega _1}{\omega _2}-1\right) }{1-\frac{\overline{\omega _1}\omega _2}{\overline{\omega _2}}|\lambda _0|^2},0\right)$ belong to $\Phi _A(\mathcal {D}_2)$. Now take a curve $\gamma \subset \epsilon \mathbb {D}\setminus \lbrace 0 \rbrace$ joining $\lambda _0$ and $\frac{\omega _1}{\omega _2}\lambda _0$. By a continuity argument, there is a point in $\Phi _A (\mathcal {D}_2)\setminus \lbrace \Phi _A(0)\rbrace$, for a certain value $\lambda \in \gamma$, that (belongs to $\lbrace |z_{11}|=|z_{22}|\rbrace$ and thus) is balanced with $0=\Phi _A(\mathop {\mathrm {diag}}(\lambda _0,\omega _1\lambda _0))$. Therefore, by Lemma 9, there is a balanced disc through these points, which is a contradiction to the assumption that 0 was the only singular point.

This finishes the proof of Lemma 10. $\square$

Lemma 12

Let $x,y\in V$ be distinct. Then there is a geodesic in V that contains x and y. In particular, V is totally geodesic. Moreover, either $\dim V=2$ or V consists of a single geodesic.

Proof

We can assume that $x=0$ and $y=\mathop {\mathrm {diag}}(\mu _1, \mu _2)$, where $0\le |\mu _2| \le |\mu _1|$ and $\mu _1\ne 0$. If $|\mu _1|=|\mu _2|$, the assertion comes from Lemma 9. Suppose $|\mu _1|>|\mu _2|$. We shall show that $W:=V\cap \{z_{12}=0\}$ is not discrete.

Set $F(z)=z_{11}$, a left inverse to the geodesic $\lambda \mapsto \begin{pmatrix} \lambda &{} 0\\ 0 &{} \frac{\mu _2}{\mu _1}\lambda \end{pmatrix}$ that passes through 0 and $\mathop {\mathrm {diag}}(\mu _1,\mu _2)$. From Proposition 8, it follows that $\overline{F(V)}$ contains the unit circle $\mathbb {T}$. Hence there is a function $\eta$ on $\mathbb {T}$ such that the set $\lbrace \mathop {\mathrm {diag}}(\omega , \eta (\omega )):\omega \in \mathbb {T}\rbrace$ is contained in $\overline{V}$.

Case 1. Suppose that V is one-dimensional. Since V is an intersection of an algebraic variety $\mathcal {V}$ with $\mathcal {R}_{II}$, the set of singular points is discrete, so there is a point $\omega _0 \in \mathbb {T}$ such that $P:=\mathop {\mathrm {diag}}(\omega _0, \eta (\omega _0))$ is a regular point of $\mathcal {V}$. In some neighbourhood of P, we can describe $\mathcal {V}$ as a graph of the holomorphic map $\begin{pmatrix} \lambda &{} g(\lambda )\\ g(\lambda ) &{} f(\lambda ) \end{pmatrix}$. Since $g(\zeta )=0$ for $\zeta$ in some neighbourhood of $\omega _0$ in $\mathbb {T}$, we have $g\equiv 0$. Hence $V\cap \lbrace z_{12}=0\rbrace$ is not discrete.

Case 2. Assume now that $\dim V=2$. If the set of singular points of the form $\mathop {\mathrm {diag}}(\omega , \eta (\omega ))$ is discrete, then we have the same situation as in Case 1. Suppose then that in the set $S:=\lbrace \mathop {\mathrm {diag}}(\omega , \eta (\omega )):\omega \in \mathbb {T}\rbrace$ there is no regular point of $\mathcal {V}$. Therefore $S\subset \text {Sing }\mathcal {V}$, and there exists a point $\mathop {\mathrm {diag}}(\omega _1,\eta (\omega _1))$, which is one-dimensional in $\mathcal {V}$, and belongs to $S\cap \text {Reg}(\text {Sing }\mathcal {V})$. The latest follows because the intersection of S with $\text {Sing}(\text {Sing } \mathcal {V})$ is discrete. Since the set of singular points is analytic, we have that $\text {Sing }\mathcal {V}$ is a graph of a holomorphic map in a neighbourhood of $\mathop {\mathrm {diag}}(\omega _1, \eta (\omega _1))$. This again reduces the problem to Case 1.

If now $W:=V\cap \lbrace z_{12}=0\rbrace$ is one dimensional, the existence of a geodesic follows from the previous lemma.

The remaining case is that W is two-dimensional. We can assume the 0 is a regular point. Then the germ of the set $U:=\left\{ \begin{pmatrix} x &{} 0\\ 0 &{} y \end{pmatrix}: x,y\in \mathbb {D}\right\}$ is contained in W and since U is irreducible in $\mathcal {R}_{II}$, by the identity principle for analytic sets, we get that $U\subset W$. Moreover U contains both 0, $\mathop {\mathrm {diag}}(\mu _1,\mu _2)$, and some geodesics through them.

For the uniqueness, take two arbitrary points x, y in V such that x is regular, and let $\mathcal G$ be a geodesic through them. Suppose there exists $a\in V\setminus \mathcal G$. Then there is a geodesic $\mathcal {G}_1$ containing a and x. Because x is smooth, the germs $(\mathcal {G})_x$ and $(\mathcal {G}_1)_x$ coincide and hence $\mathcal {G}_1=\mathcal {G}$ by the identity principle. $\square$

Remark 13

Every complex geodesic in a convex domain is a retract. The standard argument is as follows:

Choose the Kobayashi extremal $k:\mathbb {D}\rightarrow \mathcal {R}_{II}$ such that $k(\mathbb {D})=V$. Since $\mathcal {R}_{II}$ is convex, it follows from Lempert’s theorem that every Kobayashi extremal has a left inverse $L:\mathcal {R}_{II} \rightarrow \mathbb {D}$. Setting $\rho :=k\circ L$, we get the desired retraction.

Proof of Theorem 2

If V is discrete, then it is a single point. The case $\text {dim}V=1$ is justified by the previous lemma and Remark 13. Hence we shall consider the case when V is two-dimensional.

By the transitivity of automorphisms in $\mathcal {R}_{II}$, we can assume that $0\in V$ is a regular point. Suppose first that there is a point in V (resp. a vector in $T_0^\mathbb {C} V$) that forms with zero a balanced (resp. an infinitesimally balanced) pair. Composing with an automorphism and applying Lemma 9, we can assume that $\mathop {\mathrm {diag}}(\lambda ,-\lambda )$ is a subset of V. Applying again a unitary map $\frac{1}{\sqrt{2}}\left( \begin{array}{cc} 1 &{} 1\\ -1 &{} 1\end{array}\right)$, we can in fact assume that $\lambda J\in V$ for all $\lambda \in \mathbb {D}$, where J stands for the antidiagonal symmetric matrix.

Locally, in some neighbourhood of the origin, we can describe V as $\left( \begin{array}{cc} x &{} y\\ y &{}\alpha x + \beta y+ f(x,y)\end{array}\right)$, where $f(x,y) = O(||(x,y)||^2)$ and $|\alpha |\le 1$. Note that since $\mathbb {D} J\subset V$, we get $\beta =0$.

We now consider three cases.

Case 1. Let $|\alpha |=1$ and after composing with an automorphism, we lose no generality assuming that $\alpha =1$. Define

$$\begin{aligned} \gamma _k(t):=\begin{pmatrix} t &{} ikt\\ ikt &{} t+f(t,ikt) \end{pmatrix}, \end{aligned}$$

where $t\in (-\epsilon ,\epsilon )$ for some $\epsilon >0$ and $k\in \mathbb {R}$. We have $\gamma _k'(0):=\begin{pmatrix} 1 &{} ik\\ ik &{} 1 \end{pmatrix}\in T_0^{\mathbb {C}}V$, which is infinitesimally balanced with zero for any $k\in (-\delta , \delta )$ for some $\delta >0$. For any fixed $t_0$, set $g_{t_0}(y):=f(t_0,yt_0)$, $y\in \mathbb {C}$. Obviously $g_{t_0}(ik)=0$ whenever $k\in (-\delta ,\delta )$. By the identity principle, we have $g_{t_0}(z)$ is a zero function and then $f(t_0,\cdot )\equiv 0$. Similarly we show $f(\cdot , y)\equiv 0$ and hence f is a zero function.

The local description of V near the origin we are left with is $\left( \begin{array}{cc} x &{} y\\ y &{} x\end{array}\right)$. To see that this is indeed a retract, compose it with the automorphism $V\mapsto UAU^T$, where $U:=\left( \begin{array}{cc} \frac{1}{\sqrt{2}} &{} \frac{1}{\sqrt{2}}\\ -\frac{1}{\sqrt{2}} &{} \frac{1}{\sqrt{2}}\end{array}\right) .$

This means that at the point 0, the germ of V coincides with the germ of the set

$$\begin{aligned} \left\{ \left( \begin{array}{cc} z &{} 0\\ 0 &{} w\end{array}\right) : z,w\in \mathbb {D}\right\} . \end{aligned}$$

Clearly, both sets are irreducible (V is irreducible because it is totally geodesic), which means that they are equal.

Case 2. Now let $|\alpha |<1$ and $f=f(x)$ depends only on the first variable. Set $g(x)=\alpha x + f(x)$ and apply the automorphism $\Phi _A$, where $A:=\begin{pmatrix} a &{} 0\\ 0 &{} g(a) \end{pmatrix}$ for a close to 0. Then

$$\begin{aligned} \Phi _A(X)= & {} \frac{1}{(1-\overline{a}x)(1-\overline{g(a)}g(x))-\overline{a g(a)}y^2}\\&\cdot \begin{pmatrix} (x-a)(1-\overline{g(a)}g(x))+\overline{g(a)}y^2 &{} \sqrt{(1-|a|^2)(1-|g(a)|^2)}y\\ \sqrt{(1-|a|^2)(1-|g(a)|^2)}y &{} (g(x)-g(a))(1-\overline{a}x)+\overline{a}y^2 \end{pmatrix} \end{aligned}$$

for X in some neighbourhood of A. Obviously $\Phi _A(A)=0$ and

$$\begin{aligned} \Phi _A \begin{pmatrix} a &{} y\\ y &{} g(a) \end{pmatrix}=\frac{1}{C-\overline{ag(a)}y^2}\begin{pmatrix} \overline{g(a)}y^2 &{} Dy\\ Dy &{} \overline{a}y^2 \end{pmatrix}, \end{aligned}$$

where C and D are appropriate constants. Hence the vector $J=\begin{pmatrix} 0 &{} 1\\ 1 &{} 0 \end{pmatrix}$ belongs to $T_0^\mathbb {C}(\Phi _A(V))$ and it forms an infinitesimally balanced pair with zero. Therefore $\mathbb {D}J\subset \Phi _A(V)$, a contradiction to the local description of $\Phi _A(V)$ near 0.

Case 3. Let $|\alpha |<1$ and $f\not \equiv 0$ depends nontrivially on both variables x and y. We can assume $\alpha \in \mathbb {R}$. Consider an implicit function $F\in C^1(\mathbb {R}^3,\mathbb {R}^2)$ given as $(x_1,x_2,y)\mapsto (x_1+\alpha x_1+\text {Re}(f(x,y)),-x_2+\alpha x_2+\text {Im}(f(x,y)))=(0,0)$, where $x=x_1+ix_2\in \mathbb {C}$ and $y\in \mathbb {R}$ are sufficiently small. The first $2\times 2$ minor of the Jacobian matrix is equal to $\mathop {\mathrm {diag}}(1+\alpha ,-1+\alpha )$ and since $|\alpha |<1$, it follows from the implicit function theorem that there is a curve $\gamma$ such that $\overline{\gamma }(t)+\alpha \gamma (t)+f(\gamma (t),t)=0$. Note that $\frac{1}{\sqrt{|\gamma (t)|^2+t^2}}\begin{pmatrix} \gamma (t) &{} t \\ t &{} \alpha \gamma (t)+f(\gamma (t),t) \end{pmatrix}$ is unitary for all $t\in (-\epsilon ,\epsilon )\setminus \lbrace 0 \rbrace$, where $\epsilon >0$ is sufficiently small.

Since f is $O(||(x,y)||^2)$, we have that $v:=\begin{pmatrix} \gamma '(0) &{} 1\\ 1 &{} -\overline{\gamma }'(0) \end{pmatrix}$ is in $T_0^\mathbb {C}V$ and forms an infinitesimally balanced pair with zero (because v is a unitary matrix multiplied by some scalar). Hence $\mathbb {C}v\cap \mathcal {R}_{II}\subset V$, violating the nonlinearity of f.

We are left to show that in any neighbourhood of zero in V, there is a point, which is balanced or infinitesimally balanced to 0. Consider again the local description of V near the origin $\left( \begin{array}{cc} x &{} y\\ y &{}\alpha x + \beta y+ f(x,y)\end{array}\right)$.

If $|\alpha |=1$, we fix $y=0$ and then the pair $(0,\mathop {\mathrm {diag}}(1,\alpha ))$ is infinitesimally balanced.

Suppose $|\alpha | <1$. Parametrize $y:=t\in \mathbb {R}$ and let $x=x(t)=O(t)$ be such that $\overline{x}(t)+\alpha x(t)+\beta t=0$ for $t\in (-\epsilon ,\epsilon )$ with some $\epsilon >0$. The vector $v=\begin{pmatrix} x'(0) &{} 1\\ 1 &{} -\overline{x}'(0) \end{pmatrix}$ belongs to the tangent space $T_0^{\mathbb {C}}V$ and forms an infinitesimally balanced pair with zero. This finishes the proof of the theorem. $\square$

It seems to us that the main theorem may also be formulated in terms of the property of being a Carathéodory set instead of having the polynomial extension property (with the same general lines of the proofs). The notion of a Carathéodory set was recently introduced by Kosiński and Zwonek in [10]:

Definition 14

Let V be an analytic variety in a subdomain D of $\mathbb {C}^n$.

We say that V is a Carathéodory set if

$$\begin{aligned} c_D(z,w)=c_V(z,w)\quad \text {for all }z,w\in V. \end{aligned}$$

We say that V is an infinitesimal Carathéodory set if

$$\begin{aligned} \gamma _D(w;X)=\gamma _V(w;X) \end{aligned}$$

for any regular point $w\in V_{\text {reg}}$ and $X\in T_wV$.