Analytic torsion of nilmanifolds
with (2,3,5) distributions
November 28, 2023
We consider generic rank two distributions on 5-dimensional nilmanifolds, and show that the analytic torsion of their Rumin complex coincides with the Ray–Singer torsion.
The classical Ray–Singer analytic torsion [1] is a spectral invariant extracted from the de Rham complex of a closed manifold. The celebrated Cheeger–Müller theorem [2]–[5] asserts that this analytic torsion essentially coincides with the Reidemeister torsion, a topological invariant.
Rumin and Seshadri [6] have introduced an analytic torsion of the Rumin complex on contact manifolds [7]–[9] and showed that it coincides with the Ray–Singer torsion for 3-dimensional CR Seifert manifolds. Further computations for contact spheres and lens spaces have been carried out by Kitaoka [10], [11]. Recently, Albin and Quan [12] proved that the Rumin–Seshadri analytic torsion differs from the Ray–Singer torsion by the integral of a local quantity, which yet has to be identified explicitly.
An analytic torsion for the Rumin complex on more general filtered manifolds [13]–[16] has been proposed in [17]. This analytic torsion is only defined if the osculating algebras of the filtered manifold have pure cohomology. The latter assumption appears to be rather restrictive [17]. We are only aware of three types of filtered manifolds with this property: trivially filtered manifolds giving rise to the Ray–Singer torsion, contact manifolds giving rise the Rumin–Seshadri torsion, and generic rank two distributions in dimension five which are also known as (2,3,5) distributions [17].
A generic rank two distributions in dimension five is a rank two subbundle \(\mathcal{D}\) in the tangent bundle of a 5-manifold \(M\) such that Lie brackets of sections of \(\mathcal{D}\) span a subbundle \([\mathcal{D},\mathcal{D}]\) of rank three, and triple brackets of sections of \(\mathcal{D}\) span all of the tangent bundle, \([\mathcal{D},[\mathcal{D},\mathcal{D}]]=TM\). These geometric structures have first been studied by Cartan [18]. The Lie bracket of vector fields induces a fiberwise Lie bracket on the associated graded bundle, \[\mathfrak tM=\tfrac{TM}{[\mathcal{D},\mathcal{D}]}\oplus\tfrac{[\mathcal{D},\mathcal{D}]}{\mathcal{D}}\oplus\mathcal{D}.\] This is a locally trivial bundle of graded Lie algebras over \(M\) called the bundle of osculating algebras. Its fibers are all isomorphic to the 5-dimensional graded nilpotent Lie algebra \[\mathfrak g=\mathfrak g_{-3}\oplus\mathfrak g_{-2}\oplus\mathfrak g_{-1}\] with graded basis \(X_1,X_2\in\mathfrak g_{-1}\), \(X_3\in\mathfrak g_{-2}\), \(X_4,X_5\in\mathfrak g_{-3}\) and brackets \[\label{E:brackets} [X_1,X_2]=X_3,\quad[X_1,X_3]=X_4,\quad[X_2,X_3]=X_5.\tag{1}\]
The simply connected Lie group with Lie algebra \(\mathfrak g\) will be denoted by \(G\). The left invariant 2-plane field spanned by \(X_1\) and \(X_2\) provides a basic example of a (2,3,5) distribution on \(G\). Unlike contact or Engel structures, (2,3,5) distributions do have local geometry. A distribution of this type is locally diffeomorphic to the aforementioned left invariant distribution on \(G\) if and only if Cartan’s [18] harmonic curvature tensor, a section of \(S^4\mathcal{D}^*\), vanishes. This curvature tensor is constructed using an equivalent description of (2,3,5) distributions as regular normal parabolic geometries of type \((G_2,P)\) where \(G_2\) denotes the split real form of the exceptional Lie group and \(P\) denotes the parabolic subgroup corresponding to the longer root, see [18], [19] or [20]. Generic rank two distributions in dimension five have attracted quite some attention recently, cf. [17], [19], [21]–[37].
The Rumin complex associated to a (2,3,5) distribution on a 5-manifold \(M\) is a natural sequence of higher order differential operators \[\cdots\to\Gamma\bigl(\mathcal{H}^q(\mathfrak tM)\bigr)\xrightarrow{D_q}\Gamma\bigl(\mathcal{H}^{q+1}(\mathfrak tM)\bigr)\to\cdots\] where \(\mathcal{H}^q(\mathfrak tM)\) denotes the vector bundle obtained by passing to the fiberwise Lie algebra cohomology of \(\mathfrak tM\) with trivial coefficients. The Betti numbers are \(\mathop{\mathrm{rk}}\mathcal{H}^q(\mathfrak tM))=\dim H^q(\mathfrak g)=1,2,3,2,1\) for \(q=0,\dotsc,4\) and the Heisenberg order of the Rumin differential \(D_q\) is \(k_q=1,3,2,3,1\) for \(q=0,\dotsc,4\), see [38] and [39]. The Rumin differentials form a complex, \(D_{q+1}D_q=0\), that computes the de Rham cohomology of \(M\). Actually, there exist injective differential operators \(L_q\colon\Gamma(\mathcal{H}^q(\mathfrak tM))\to\Omega^q(M)\), embedding the Rumin complex as a subcomplex in the de Rham complex and inducing isomorphisms on cohomology. Twisting with a flat complex vector bundle \(F\) we obtain a complex of differential operators \[\label{E:Rumin46M} \cdots\to\Gamma\bigl(\mathcal{H}^q(\mathfrak tM)\otimes F\bigr)\xrightarrow{D_q}\Gamma\bigl(\mathcal{H}^{q+1}(\mathfrak tM)\otimes F\bigr)\to\cdots\tag{2}\] computing \(H^*(M;F)\), the de Rham cohomology of \(M\) with coefficients in \(F\). Rumin has shown that the sequence 2 becomes exact on the level of the Heisenberg principal symbol, see [13], [14], or [39]. Hence, the Rumin complex is a Rockland [40] complex, the analogue of an elliptic complex in the Heisenberg calculus, see [39] for more details.
A fiberwise graded Euclidean inner product \(g\) on \(\mathfrak tM\) and a fiberwise Hermitian inner product \(h\) on \(F\) give rise to \(L^2\) inner products on \(\Gamma(\mathcal{H}^q(\mathfrak tM)\otimes F)\) which in turn provide formal adjoints \(D_q^*\) of the Rumin differentials in 2 . Assuming \(M\) to be closed, the operator \(D_q^*D_q\) has an infinite dimensional kernel if \(q>0\) but the remaining part of its spectrum consists of isolated positive eigenvalues with finite multiplicities only. Moreover, \((D_q^*D_q)^{-s}\) is trace class for \(\Re s>10/2k_q\). The number ten appears here because this is the homogeneous dimension of the filtered manifold \(M\). Furthermore, the function \(\mathop{\mathrm{tr}}(D_q^*D_q)^{-s}\) admits an analytic continuation to a meromorphic function on the entire complex plane which is holomorphic at \(s=0\), see [17]. This permits to define the zeta regularized determinant \({\det}_*|D_q|\) by \[\label{E:edtDq46def} \log{\det}_*|D_q|:=-\tfrac12\tfrac\partial{\partial s}\big|_{s=0}\mathop{\mathrm{tr}}(D_q^*D_q)^{-s}.\tag{3}\] The notation \({\det}_*\) indicates that the zero eigenspace does not contribute, i.e., we are considering the regularized product of nonzero eigenvalues. Correspondingly, the complex powers are defined to vanish on the kernel of \(D_q^*D_q\). The analytic torsion \(\tau(M,\mathcal{D},F,g,h)\) is the graded determinant of the Rumin complex, i.e., \[\label{E:tau46def} \log\tau(M,\mathcal{D},F,g,h):=\sum_{q=0}^4(-1)^q\log{\det}_*|D_q|.\tag{4}\] By Poincaré duality, see [13], [14], or [17], we have \({\det}_*|D_0|={\det}_*|D_4|\) and \({\det}_*|D_1|={\det}_*|D_3|\), provided \(h\) is parallel. Hence, in this (unitary) case the torsion may be expressed in terms of three determinants, \[\tau(M,\mathcal{D},F,g,h)=\frac{{\det}_*^2|D_0|\cdot{\det}_*|D_2|}{{\det}_*^2|D_1|}.\]
Since \(\mathfrak g\) has pure cohomology, the grading automorphism \(\phi_\tau\in\mathop{\mathrm{Aut}}_\textrm{gr}(\mathfrak g)\) acts as a scalar on each cohomology, namely by \(\tau^{N_q}\) on \(H^q(\mathfrak g)\) where \(N_q=0,1,4,6,9,10\) for \(q=0,\dotsc,5\). These numbers are related to the Heisenberg orders of the Rumin differentials via the equation \(k_q=N_{q+1}-N_q\). We fix natural numbers \(a_q\) such that \(\kappa:=a_qk_q\) is independent of \(q\). The smallest possible choice would be \(a_q=6,2,3,2,6\) with \(\kappa=6\). In general, \(\kappa\) is a multiple of six. Then the Rumin–Seshadri operators \[\label{E:Delta46def} \Delta_q:=\bigl(D_{q-1}D_{q-1}^*\bigr)^{a_{q-1}}+\bigl(D_q^*D_q\bigr)^{a_q}\tag{5}\] are all of Heisenberg order \(2\kappa\). These operators are analytically much better behaved than \(D_q^*D_q\). Indeed, \(\Delta_q\) is a Rockland [40] operator and, thus, admits a parametrix in the Heisenberg calculus [39], [41], [42]. Moreover, \(\Delta_q^{-s}\) is trace class for \(\Re s>10/2\kappa\), and the zeta function \(\mathop{\mathrm{tr}}\Delta_q^{-s}\) admits an analytic continuation to a meromorphic function on the entire complex plane which is holomorphic at \(s=0\), see [43]. The analytic properties of the operator \(D_q^*D_q\) stated in the preceding paragraph can readily be deduced from the corresponding properties of \(\Delta_q\), see [17] for more details. Putting \[\label{E:zeta46M46def} \zeta_{M,\mathcal{D},F,g,h}(s):=\mathop{\mathrm{str}}(N\Delta^{-s}):=\sum_{q=0}^5(-1)^qN_q\mathop{\mathrm{tr}}(\Delta_q^{-s})\tag{6}\] the analytic torsion of the Rumin complex can be expressed in the form \[\label{E:tau46via46zeta} \tau(M,\mathcal{D},F,g,h)=\exp\left(\tfrac1{2\kappa}\zeta'_{M,\mathcal{D},F,g,h}(0)\right)\tag{7}\] which is analogous to the formulas for the Ray–Singer torsion in [1] and the Rumin–Seshadri torsion in [6], see [17] for more details.
It turns out to be convenient to incorporate the zero eigenspaces of \(\Delta_q\) and consider the analytic torsion of the Rumin complex as a norm \(\|-\|^{\mathop{\mathrm{sdet}}H^*(M;F)}_{\mathcal{D},g,h}\) on the graded determinant line \[\mathop{\mathrm{sdet}}H^*(M;F)=\bigotimes_{q=0}^5\bigl(\det H^q(M;F)\bigr)^{(-1)^q}.\] Basic properties of this torsion have been established in [17]. In the acyclic case, it reduces to (the reciprocal of) \(\tau(M,\mathcal{D},F,g,h)\). The Ray–Singer torsion, too, is best regarded as a norm on this graded determinant line [2], [44]. We will denote it by \(\|-\|_\textrm{RS}^{\mathop{\mathrm{sdet}}H^*(M;F)}\). The Ray–Singer torsion does not depend on metric choices since the dimension of \(M\) is odd.
In this paper we will determine the analytic torsion of the Rumin complex on nilmanifolds \(\Gamma\setminus G\) where \(\Gamma\) is a lattice in \(G\), i.e., a cocompact discrete subgroup in \(G\). We consider any left invariant (2,3,5) distribution \(\mathcal{D}_G\) on \(G\) and any left invariant fiberwise graded Euclidean inner product on \(\mathfrak tG\). These descend to a (2,3,5) distribution \(\mathcal{D}_{\Gamma\setminus G}\) on the nilmanifold \(\Gamma\setminus G\) and a fiberwise graded Euclidean inner product \(g_{\Gamma\setminus G}\) on \(\mathfrak t(\Gamma\setminus G)\). For a unitary representation \(\rho\colon\Gamma\to U(k)\) we let \(h_\rho\) denote the canonical (parallel) fiberwise Hermitian inner product on the associated flat complex vector bundle \(F_\rho:=G\times_\rho\mathbb{C}^k\) over \(\Gamma\setminus G\).
We are now in a position to state our main result.
Theorem 1. If \(\chi\colon\Gamma\to U(1)\) is a nontrivial unitary character, then the twisted Rumin complex on the nilmanifold \(\Gamma\setminus G\) associated with \(\mathcal{D}_{\Gamma\setminus G}\) and \(F_\chi\) is acyclic and its analytic torsion is trivial, that is, \[\tau\bigl(\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi\bigr)=1.\]
In the situation of this theorem, the Ray–Singer torsion is known to be trivial too, \(\tau_\textrm{RS}(\Gamma\setminus G;F_\chi)=1\), see Lemma 11 below. Hence, it coincides with the analytic torsion of the Rumin complex. More generally, we have:
Corollary 1. For any unitary representation \(\rho\colon\Gamma\to U(k)\) the analytic torsion of the Rumin complex coincides with the Ray–Singer torsion, that is, \[\|-\|_{\mathcal{D}_{\Gamma\setminus G},g_{\Gamma\setminus G},h_\rho}^{\mathop{\mathrm{sdet}}H^*(\Gamma\setminus G;F_\rho)} = \|-\|_\textrm{RS}^{\mathop{\mathrm{sdet}}H^*(\Gamma\setminus G;F_\rho)}.\]
These are the first (2,3,5) distributions for which the analytic torsion of the Rumin complex has been computed. In [17] a partial result of this type has been obtained by exploiting a large discrete symmetry group of the distribution \(\mathcal{D}_{\Gamma\setminus G}\).
The proof given below is based on the decomposition of the Rumin complex over \(\Gamma\setminus G\) into a countable direct sum \(D_*=\bigoplus_\rho m(\rho)\cdot\rho(D_*)\) of Rumin complexes in irreducible unitary representations \(\rho\) of \(G\), denoted by \(\rho(D_*)\). The multiplicities \(m(\rho)\) of the contributing representations are known explicitly through a formula due to Howe [45] and Richardson [46] and involve counting the number of solutions to certain quadratic congruences, cf. Lemma 5(III), 38 , or 54 below. The zeta function in 6 decomposes accordingly, \[\label{E:zzz} \zeta_{\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi}(s)=\sum_\rho m(\rho)\cdot\zeta_\rho(s)\tag{8}\] where \(\zeta_\rho(s)\) denotes the zeta function associated with \(\rho(D_*)\). The values \(\zeta_\rho(0)\) and \(\zeta'_\rho(0)\) have been determined in [30] for every irreducible unitary representation \(\rho\). Even though the sum on the right hand side in 8 converges only for \(\Re s>10/2\kappa\), we are able to conclude \(\zeta_{\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi}'(0)=0\) via regularization, whence the theorem stated above. This is the subtlest part of the paper at hand and builds on further properties of \(\zeta_\rho(s)\) obtained in [30]. Decomposing \[\sum_\rho m(\rho)\cdot\zeta_\rho(s)=\zeta_{\mathrm{I},\Gamma,\chi}(s)+\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)+\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\] according to the three types of irreducible unitary representations of \(G\), we will use Epstein zeta functions to obtain the analytic continuation of each of the three summands, cf. 57 , 62 , and 77 below.
Alternatively, one could try to extend Albin and Quan’s [12] analysis of the sub-Riemannian limit in order to show that the torsion of the Rumin complex of a (2,3,5) distribution differs from the Ray–Singer torsion by the integral of a local quantity cf. [12]. This, too, would immediately imply the corollary stated above.
The remaining part of this paper is organized as follows. In Section 2 we provide an explicit description of all lattices in \(G\). In Section 3 we use the aforementioned result due to Howe [45] and Richardson [46] to decompose the space of sections of \(F_\chi\) into irreducible \(G\)-representations. In Section 4 we describe the corresponding decomposition of the zeta function \(\zeta_{\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi}(s)\). In Section 5, building on results obtained in [30], we evaluate the derivative of this zeta function at \(s=0\). In Section 6 we derive the theorem and corollary stated above.
In this section we provide explicit descriptions of all lattices in \(G\).
Let \(X_1,\dotsc,X_5\) be a graded basis of \(\mathfrak g\) with brackets as in 1 . For the sake of notational simplicity we will use this basis to identify \(\mathfrak g\) with \(\mathbb{R}^5\), that is, a vector \((x_1,\dotsc,x_5)^t\in\mathbb{R}^5\) will be identified with \(\sum_{i=1}^5x_iX_i\in\mathfrak g\).
The exponential map provides a diffeomorphism \(\exp\colon\mathfrak g\to G\). Using the Baker–Campbell–Hausdorff formula we find \[\exp\begin{pmatrix}x_1\\\vdots\\x_5\end{pmatrix} \exp\begin{pmatrix}y_1\\\vdots\\y_5\end{pmatrix} =\exp\begin{pmatrix}z_1\\\vdots\\z_5\end{pmatrix}\] where \[\label{E:zxy} z=x\cdot y :=\left(\begin{array}{l} x_1+y_1\\ x_2+y_2\\ x_3+y_3+\frac{x_1y_2-x_2y_1}{2}\\x_4+y_4+\frac{x_1y_3-x_3y_1}{2}+\frac{(x_1-y_1)(x_1y_2-x_2y_1)}{12}\\ x_5+y_5+\frac{x_2y_3-x_3y_2}{2}+\frac{(x_2-y_2)(x_1y_2-x_2y_1)}{12} \end{array}\right).\tag{9}\]
The center of \(G\) will be denoted by \(Z\). Clearly, \(Z=\exp(\mathfrak z)\) where \(\mathfrak z\) denotes the center of \(\mathfrak g\) which is spanned by \(X_4,X_5\). For the group of commutators we have \([G,G]=\exp([\mathfrak g,\mathfrak g])\) and the derived subalgebra \([\mathfrak g,\mathfrak g]\) is spanned by \(X_3,X_4,X_5\).
For \(r\in\mathbb{N}\) and \(e,f,g,h,u,v\in\mathbb{Q}\) we consider \(\tilde{\gamma}_i\in\mathfrak g\) defined by \[\label{E:Gamma46tgen} \tilde{\gamma}_1=\begin{pmatrix}1\\0\\0\\0\\0\end{pmatrix},\quad \tilde{\gamma}_2=\begin{pmatrix}0\\1\\0\\0\\0\end{pmatrix},\quad \tilde{\gamma}_3=\begin{pmatrix}0\\0\\1/r\\u/2r\\v/2r\end{pmatrix},\quad \tilde{\gamma}_4=\begin{pmatrix}0\\0\\0\\e\\f\end{pmatrix},\quad \tilde{\gamma}_5=\begin{pmatrix}0\\0\\0\\g\\h\end{pmatrix},\tag{10}\] and let \(\Gamma\) denote the subgroup of \(G\) generated by the exponentials \[\gamma_i:=\exp\tilde{\gamma}_i,\qquad i=1,\dotsc,5.\] Let \(\Gamma''\) denote the lattice in \(\mathbb{R}^2\) generated by \[\label{E:Gamma3939} \begin{pmatrix}1/r\\0\end{pmatrix},\quad \begin{pmatrix}0\\1/r\end{pmatrix},\quad \begin{pmatrix}\frac{u-1}{2}\\\frac{v-1}{2}\end{pmatrix},\quad \begin{pmatrix}e\\f\end{pmatrix},\quad \begin{pmatrix}g\\h\end{pmatrix}.\tag{11}\]
The following generalizes [17].
Lemma 1. The subgroup \(\Gamma\) is a lattice in \(G\). Moreover: \[\begin{align} \tag{12} \log\Gamma &=\left\{\begin{pmatrix}x_1\\\vdots\\ x_5\end{pmatrix}\in\mathfrak g\,\middle| \begin{array}{rl} x_1,x_2&\in\mathbb{Z}\\ x_3-\frac{x_1x_2}{2}&\in\frac{1}{r}\mathbb{Z}\\ \begin{pmatrix}x_4-\tfrac{x_1^2x_2}{12}-\frac{x_1+u}{2}(x_3-\frac{x_1x_2}{2})\\x_5+\tfrac{x_1x_2^2}{12}+\frac{x_2-v}{2}(x_3-\frac{x_1x_2}{2})\end{pmatrix}&\in\Gamma'' \end{array}\right\} \\ \tag{13} \log\bigl(\Gamma\cap[G,G]\bigr) &=\left\{\begin{pmatrix}x_1\\\vdots\\ x_5\end{pmatrix}\in\mathfrak g\,\middle| \begin{array}{rcl} x_1,x_2&=&0\\ x_3&\in&\frac{1}{r}\mathbb{Z}\\ \begin{pmatrix}x_4-\frac{u}{2}x_3\\x_5-\frac{v}{2}x_3\end{pmatrix}&\in&\Gamma'' \end{array}\right\} \\ \tag{14} \log\bigl(\Gamma\cap Z\bigr) &=\left\{\begin{pmatrix}x_1\\\vdots\\ x_5\end{pmatrix}\in\mathfrak g\,\middle| \begin{array}{rcl} x_1,x_2,x_3&=&0\\ \begin{pmatrix}x_4\\x_5\end{pmatrix}&\in&\Gamma'' \end{array}\right\} \\ \tag{15} \log\bigl([\Gamma,\Gamma]\bigr) &=\left\{\begin{pmatrix}x_1\\\vdots\\ x_5\end{pmatrix}\in\mathfrak g\,\middle| \begin{array}{rcl} x_1,x_2&=&0\\ x_3&\in&\mathbb{Z}\\ x_4-\frac{x_3}{2},x_5-\frac{x_3}{2}&\in&\frac{1}{r}\mathbb{Z} \end{array}\right\} \\ \tag{16} \log\bigl([\Gamma,\Gamma]\cap Z\bigr) &=\left\{\begin{pmatrix}x_1\\\vdots\\ x_5\end{pmatrix}\in\mathfrak g\,\middle| \begin{array}{rcl} x_1,x_2,x_3&=&0\\ x_4,x_5&\in&\frac{1}{r}\mathbb{Z} \end{array}\right\} \end{align}\]
Proof. With \(z\) as in 9 we have \[\begin{align} z_3-\tfrac{z_1z_2}2&=x_3-\tfrac{x_1x_2}2+y_3-\tfrac{y_1y_2}2-x_2y_1\\ z_4-\tfrac{z_1^2z_2}{12}-\tfrac{z_1+u}2(z_3-\tfrac{z_1z_2}2) &=x_4-\tfrac{x_1^2x_2}{12}-\tfrac{x_1+u}2(x_3-\tfrac{x_1x_2}2)\\ &\qquad+y_4-\tfrac{y_1^2y_2}{12}-\tfrac{y_1+u}2(y_3-\tfrac{y_1y_2}2)\\ &\qquad-y_1(x_3-\tfrac{x_1x_2}2)+\tfrac{u-1}2x_2y_1+\tfrac{y_1(y_1+1)x_2}2\\ z_5+\tfrac{z_1z_2^2}{12}+\tfrac{z_2-v}2(z_3-\tfrac{z_1z_2}2) &=x_5+\tfrac{x_1x_2^2}{12}+\tfrac{x_2-v}2(x_3-\tfrac{x_1x_2}2)\\ &\qquad+y_5+\tfrac{y_1y_2^2}{12}+\tfrac{y_2-v}2(y_3-\tfrac{y_1y_2}2)\\ &\qquad+x_2(y_3-\tfrac{y_1y_2}2)+\tfrac{v-1}2x_2y_1-\tfrac{x_2(x_2+1)y_1}2 \end{align}\] as well as: \[\begin{align} (-x_3)-\tfrac{(-x_1)(-x_2)}2&=-\bigl(x_3-\tfrac{x_1x_2}2\bigr)-x_1x_2\\ (-x_4)-\tfrac{(-x_1)^2(-x_2)}{12}-\tfrac{(-x_1)+u}2&\bigl((-x_3)-\tfrac{(-x_1)(-x_2)}2\bigr)\\ &=-\Bigl(x_4-\tfrac{x_1^2x_2}{12}-\tfrac{x_1+u}2(x_3-\tfrac{x_1x_2}2)\Bigr)\\ &\qquad-x_1\bigl(x_3-\tfrac{x_1x_2}2\bigr)+\tfrac{u-1}2x_1x_2-\tfrac{x_1(x_1-1)x_2}2\\ (-x_5)+\tfrac{(-x_1)(-x_2)^2}{12}+\tfrac{(-x_2)-v}2&\bigl((-x_3)-\tfrac{(-x_1)(-x_2)}2\bigr)\\ &=-\Bigl(x_5+\tfrac{x_1x_2^2}{12}+\tfrac{x_2-v}2(x_3-\tfrac{x_1x_2}2)\Bigr)\\ &\qquad+x_2\bigl(x_3-\tfrac{x_1x_2}2\bigr)+\tfrac{v-1}2x_1x_2+\tfrac{x_1x_2(x_2+1)}2\\ \end{align}\] Using these relations one readily shows that the right hand side in 12 is a lattice containing \(\Gamma\). Using the computations \[\label{E:g312} \log(\gamma_1^k\gamma_2^l)=\begin{pmatrix}k\\l\\kl/2\\k^2l/12\\-kl^2/12\end{pmatrix},\quad \log\bigl(\gamma_3^r[\gamma_1,\gamma_2]^{-1}\bigr)=\begin{pmatrix}0\\0\\0\\\frac{u-1}{2}\\\frac{v-1}{2}\end{pmatrix},\tag{17}\] \[\label{E:g124613} \log[\gamma_1,\gamma_3]=\begin{pmatrix}0\\0\\0\\1/r\\0\end{pmatrix},\quad \log[\gamma_2,\gamma_3]=\begin{pmatrix}0\\0\\0\\0\\1/r\end{pmatrix},\tag{18}\] it is easy to see that this lattice is generated by \(\gamma_1,\dotsc,\gamma_5\). Taking also into account \[\label{E:Gamma46comm} \log[\gamma_1,\gamma_2]=\begin{pmatrix}0\\0\\1\\1/2\\1/2\end{pmatrix},\quad \log[\gamma_1,[\gamma_1,\gamma_2]]=\begin{pmatrix}0\\0\\0\\1\\0\end{pmatrix},\quad \log[\gamma_2,[\gamma_1,\gamma_2]]=\begin{pmatrix}0\\0\\0\\0\\1\end{pmatrix},\tag{19}\] we obtain the description of \([\Gamma,\Gamma]\) in 15 . Here the formula \[\log[\exp x,\exp y]=\begin{pmatrix}0\\0\\x_1y_2-x_2y_1\\x_1y_3-x_3y_1+\frac{(x_1+y_1)(x_1y_2-x_2y_1)}{2}\\x_2y_3-x_3y_2+\frac{(x_2+y_2)(x_1y_2-x_2y_1)}{2}\end{pmatrix}\] for commutators is helpful. The remaining assertions are now obvious. ◻
Lemma 2. Every lattice in \(G\) is of the form considered above, up to a not necessarily graded automorphism of \(G\).
Proof. Inspecting the proof of [47] we see that every lattice \(\Gamma\) in \(G\) is generated by five elements \(\gamma_1,\dotsc,\gamma_5\) such that \(\gamma_1,\gamma_2\in\Gamma\), \(\gamma_3\in\Gamma\cap[G,G]\), and \(\gamma_4,\gamma_5\in\Gamma\cap Z\). By [17] there exists a not necessarily graded automorphism of \(\mathfrak g\) that maps \(\log\gamma_1\) to \(\tilde{\gamma}_1\) and \(\log\gamma_2\) to \(\tilde{\gamma}_2\), cf. 10 . Up to an automorphism of \(G\) we may thus assume \(\gamma_1=\exp\tilde{\gamma}_1\) and \(\gamma_2=\exp\tilde{\gamma}_2\). As \(\gamma_3\in[G,G]\) and \(\gamma_4,\gamma_5\in Z\), they must be of the form \(\gamma_i=\exp\tilde{\gamma}_i\) where \(\tilde{\gamma}_i\), \(i=3,4,5\), are as indicated in 10 with, a priori, real numbers \(r,u,v,e,f,g,h\). To see that these numbers must all be rational it suffices to observe that \(\log(\Gamma\cap Z)\) is a lattice in \(\mathfrak g_{-3}=\mathbb{R}^2\) which contains the vectors in 11 by 17 and 18 , but also contains the two unit vectors in view of 19 . Write \(1/r=s/t\) where \(s\) and \(t\geq1\) are coprime integers. Hence, there exist integers \(k\) and \(l\) such that \(ks+lt=1\). Note that \(\gamma_1,\gamma_2,\gamma_3^k[\gamma_1,\gamma_2]^l,\gamma_4,\gamma_5\) still generate \(\Gamma\) for we have \(\bigl(\gamma_3^k[\gamma_1,\gamma_2]^l\bigr)^s=\gamma_3\mod Z\) in view of 17 , and we may assume that \(\gamma_4,\gamma_5\) generate \(\Gamma\cap Z\). Replacing \(\gamma_3\) with \(\gamma_3^k[\gamma_1,\gamma_2]^l\), we may thus assume that the image of \(\log\gamma_3\) in \([\mathfrak g,\mathfrak g]/\mathfrak z=\mathfrak g_{-2}=\mathbb{R}\) is \(1/t\). ◻
The natural homomorphism \(p\colon G\to G/[G,G]\cong\mathbb{R}^2\) gives rise to a short exact sequence of abelian groups \[\label{E:abelGamma} 0\to\frac{\Gamma\cap[G,G]}{[\Gamma,\Gamma]}\to\frac{\Gamma{[\Gamma,\Gamma]}}{\to} p(\Gamma)\to0\tag{20}\] where \(\frac{\Gamma\cap[G,G]}{[\Gamma,\Gamma]}\) is a finite abelian group and \(p(\Gamma)\cong\mathbb{Z}^2\). In particular, the sequence splits and we obtain an isomorphism \[\label{E:Gamma46ab} \frac{\Gamma{[\Gamma,\Gamma]}}{\cong}\frac{\Gamma\cap[G,G]}{[\Gamma,\Gamma]}\oplus\mathbb{Z}^2.\tag{21}\] The group of unitary characters of \(\Gamma\), thus, is a finite union of 2-tori, \[\label{E:Gamma42} \hom\bigl(\Gamma,U(1)\bigr)\cong A\times U(1)\times U(1),\tag{22}\] where \(A=\hom\left(\frac{\Gamma\cap[G,G]}{[\Gamma,\Gamma]},U(1)\right)\) is a finite abelian group.
To specify a character \(\chi\colon\Gamma\to U(1)\) it suffices to know its values on the generators, \(\chi(\gamma_i)\) for \(i=1,\dotsc,5\).
Lemma 3. Suppose \(\chi\colon\Gamma\to U(1)\) is a unitary character, and let \(c\) be a real number such that \(\chi(\gamma_3)=e^{2\pi\mathbf{i}c/r}\). Then there exist integers \(\lambda_0,\mu_0\) such that \[\label{E:lm0i} \lambda_0\in r\mathbb{Z},\quad\mu_0\in r\mathbb{Z},\quad\lambda_0\tfrac{u-1}2+\mu_0\tfrac{v-1}2\in c+\mathbb{Z},\tag{23}\] \[\label{E:lm0ii} e^{2\pi\mathbf{i}(\lambda_0e+\mu_0f)}=\chi(\gamma_4),\quad e^{2\pi\mathbf{i}(\lambda_0g+\mu_0h)}=\chi(\gamma_5).\tag{24}\]
Proof. Since \(\Gamma\cap Z\) is a lattice in \(Z\cong\mathbb{R}^2\) it admits a basis consisting of two elements. Clearly, there exists a functional \(\alpha\in\mathfrak z^*\) such that the homomorphism \(Z\to U(1)\), \(z\mapsto e^{2\pi\mathbf{i}\alpha(\log z)}\) coincides with \(\chi\) on the aforementioned basis of \(\Gamma\cap Z\). We conclude that \[\label{E:chiZ} \chi(\gamma)=e^{2\pi\mathbf{i}\alpha(\log\gamma)}\tag{25}\] for all \(\gamma\in\Gamma\cap Z\). Via the identification \(\mathfrak z^*=(\mathbb{R}^2)^*\), we have \(\alpha=(\lambda_0,\mu_0)\) for some real numbers \(\lambda_0\) and \(\mu_0\). Putting \(\gamma=\gamma_4\) and \(\gamma=\gamma_5\) in 25 , we obtain the two equations in 24 , respectively. Putting \(\gamma=\gamma_3^r[\gamma_1,\gamma_2]^{-1}\) in 25 and using 17 we obtain the last equation in 23 , for \(\chi(\gamma_3^r[\gamma_1,\gamma_2]^{-1})=e^{2\pi\mathbf{i}c}\). Putting \(\gamma=[\gamma_1,\gamma_3]\) and \(\gamma=[\gamma_2,\gamma_3]\) in 25 and using 18 we obtain the first and second equation in 23 , respectively, for \(\chi\) vanishes on commutators. ◻
If \(\Gamma\) is a lattice in a simply connected nilpotent Lie group \(N\), then \(L^2(\Gamma\setminus N)\) decomposes into a countable direct sum of irreducible unitary representations of \(N\). The multiplicities of the representations appearing in this decomposition have been studied by Moore [48]. An explicit formula for these multiplicities has been conjectured by Mostow and proved, independently, by Howe [45] and Richardson [46]. More generally, for every unitary character \(\chi\colon\Gamma\to U(1)\), the induced representation \[L^2\bigl(N\times_\chi\mathbb{C}\bigr) =\left\{g\colon N\to\mathbb{C}\, \middle|\begin{array}{c}\text{g(\gamma n)=\chi(\gamma)g(n) for \gamma\in\Gamma and n\in N,}\\|g|\in L^2(\Gamma\setminus N)\end{array} \right\}\] decomposes into a countable direct sum of irreducible unitary representations and the multiplicities are known explicitly, cf. [45] and [46].
Lemma 4 (Howe, Richardson). Let \(N\) be a simply connected nilpotent Lie group, suppose \(\Gamma\) a lattice in \(N\), and let \(\chi\colon\Gamma\to U(1)\) denote a unitary character. Then \(L^2(N\times_\chi\mathbb{C})\) decomposes into a countable direct sum of irreducible unitary representations of \(N\). If an irreducible unitary representation of \(N\) appears in this decomposition, then it is induced from a rational maximal character via Kirillov’s construction. Moreover, if the rational maximal character \((\bar f,M)\) induces \(\pi\), then the multiplicity of \(\pi\) in \(L^2(N\times_\chi\mathbb{C})\) coincides with the (finite) cardinality \(\sharp((M\setminus N)_\chi/\Gamma)\), that is, the number of \(\Gamma\) orbits in \[(M\setminus N)_\chi =\left\{n\in M\setminus N\,\middle| \begin{array}{c}\text{(\bar f^n,M^n) is rational, and}\\\bar f^n|_{\Gamma\cap M^n}=\chi|_{\Gamma\cap M^n}\end{array} \right\}.\] Here \(\bar f^n(m)=\bar f(nmn^{-1})\) and \(M^n=n^{-1}Mn\) for \(n\in N\) and \(m\in M\).
Let us briefly recall some of the terminology used in the preceding lemma. To this end let \(\mathfrak n\) denote the Lie algebra of \(N\), suppose \(f\in\mathfrak n^*\), and let \(\mathfrak m\) denote a maximal subordinate subalgebra, i.e., a subalgebra of maximal dimension in \(\mathfrak n\) such that \(f([\mathfrak m,\mathfrak m])=0\). Then \[\label{E:barf} \bar f(m):=e^{2\pi\mathbf{i}f(\log m)}\tag{26}\] defines a unitary character on the group \(M:=\exp(\mathfrak m)\). According to Kirillov [49]–[51] such a character induces an irreducible unitary representation of \(N\) given by right translation on the Hilbert space \[\left\{h\colon N\to\mathbb{C}\, \middle|\begin{array}{c}\text{h(mn)=\bar f(m)h(n) for m\in M and n\in N,}\\|h|\in L^2(M\setminus N)\end{array} \right\}.\] One may always assume \(\mathfrak m\) to be special [46] and then \((\bar f,M)\) is called a maximal character [46]. The group \(N\) acts by conjugation on the set of maximal characters [46] and the stabilizer of \((\bar f,M)\) is \(M\), see [46]. A maximal character \((\bar f,M)\) is called rational [46] if it can be obtained from a (rational) linear functional \(f\in\mathfrak n^*\) mapping \(\log\Gamma\) into the rational numbers and \(\mathfrak m\) is rational with respect to the rational structure on \(\mathfrak n\) provided by \(\log\Gamma\). Note that \(\mathfrak m\) is rational if and only if \(\Gamma\cap M\) is a lattice in \(M\), i.e., if and only if \((\Gamma\cap M)\setminus M\) is compact. Clearly, the action of \(\Gamma\) preserves the subset of rational maximal characters and the integrality condition \(\bar f|_{\Gamma\cap M}=\chi|_{\Gamma\cap M}\).
In the remaining part of this section we will specialize the preceding lemma to the 5-dimensional Lie group \(G\) considered before. All irreducible unitary representations of this Lie group have been described explicitly by Dixmier in [52]. There are three types of irreducible unitary representations of \(G\) which we now describe in terms of a graded basis \(X_1,\dotsc,X_5\) of \(\mathfrak g\) satisfying 1 . If \(\rho\) is a representation of \(G\), we let \(\rho'\) denote the corresponding infinitesimal representation of \(\mathfrak g\).
(I) Scalar representations: For \((\alpha,\beta)\in\mathbb{R}^2\) there is an irreducible unitary representation \(\rho_{\alpha,\beta}\) of \(G\) on \(\mathbb{C}\) such that \[\label{E:rep46scalar} \rho'_{\alpha,\beta}(X_1)=2\pi\mathbf{i}\alpha,\qquad \rho'_{\alpha,\beta}(X_2)=2\pi\mathbf{i}\beta,\tag{27}\] and \(\rho'_{\alpha,\beta}(X_3)=\rho'_{\alpha,\beta}(X_4)=\rho'_{\alpha,\beta}(X_5)=0\). Via Kirillov’s construction, the functional \(f\in\mathfrak g^*\) with \(f(X_1)=\alpha\), \(f(X_2)=\beta\), and \(f(X_3)=f(X_4)=f(X_5)=0\) induces a representation isomorphic to \(\rho_{\alpha,\beta}\). These are precisely the irreducible representations which factor through the abelianization \(G/[G,G]\cong\mathbb{R}^2\).
(II) Schrödinger representations: For \(0\neq\hbar\in\mathbb{R}\) there is an irreducible unitary representation \(\rho_\hbar\) of \(G\) on \(L^2(\mathbb{R})=L^2(\mathbb{R},d\theta)\) such that \[\label{E:Schroedinger} \rho'_\hbar(X_1)=\partial_\theta,\qquad \rho'_\hbar(X_2)=2\pi\mathbf{i}\hbar\cdot\theta,\qquad \rho'_\hbar(X_3)=2\pi\mathbf{i}\hbar,\tag{28}\] and \(\rho'_\hbar(X_4)=\rho'_\hbar(X_5)=0\). Via Kirillov’s construction, any functional \(f\in\mathfrak g^*\) with \(f(X_3)=\hbar\) and \(f(X_4)=f(X_5)=0\) induces a representation isomorphic to \(\rho_\hbar\). These are precisely the irreducible unitary representations which factor through the 3-dimensional Heisenberg group \(H=G/Z\) but do not factor through the abelianization \(G/[G,G]\).
(III) Generic representations: For real numbers \(\lambda,\mu,\nu\) with \((\lambda,\mu)\neq(0,0)\) there is an irreducible unitary representation \(\rho_{\lambda,\mu,\nu}\) of \(G\) on \(L^2(\mathbb{R})=L^2(\mathbb{R},d\theta)\) such that: \[\begin{align} \notag \rho'_{\lambda,\mu,\nu}(X_1) &=\frac{\lambda{(\lambda^2+\mu^2)^{1/3}}}{\cdot}\partial_\theta-\frac{2\pi\mathbf{i}\mu}{(\lambda^2+\mu^2)^{1/3}}\cdot\frac{\theta^2+\nu(\lambda^2+\mu^2)^{-2/3}}{2} \\\notag \rho'_{\lambda,\mu,\nu}(X_2) &=\frac{\mu{(\lambda^2+\mu^2)^{1/3}}}{\cdot}\partial_\theta+\frac{2\pi\mathbf{i}\lambda}{(\lambda^2+\mu^2)^{1/3}}\cdot\frac{\theta^2+\nu(\lambda^2+\mu^2)^{-2/3}}{2} \\\label{E:rep46gen46X3} \rho'_{\lambda,\mu,\nu}(X_3) &=2\pi\mathbf{i}(\lambda^2+\mu^2)^{1/3}\cdot\theta, \\\notag \rho'_{\lambda,\mu,\nu}(X_4) &=2\pi\mathbf{i}\lambda, \\\notag \rho'_{\lambda,\mu,\nu}(X_5) &=2\pi\mathbf{i}\mu. \end{align}\tag{29}\] This differs from the representation given in [52] by a conjugation with a unitary scaling on \(L^2(\mathbb{R})\) which we have introduced for better compatibility with the grading automorphism. Note that \[\rho'_{\lambda,\mu,\nu}\bigl(X_3X_3+2X_1X_5-2X_2X_4\bigr)=(2\pi)^2\nu.\] Any functional \(f\in\mathfrak g^*\) with \(f(X_4)=\lambda\), \(f(X_5)=\mu\), and \[\label{E:rep46gen46nu} f(X_3)^2+2f(X_1)f(X_5)-2f(X_2)f(X_4)=\nu\tag{30}\] induces a representation isomorphic to \(\rho_{\lambda,\mu,\nu}\).
The representations listed above are mutually nonequivalent, and they comprise all equivalence classes of irreducible unitary representations of \(G\). Given the convention in 26 it turns out to be convenient to incorporate the factors \(2\pi\) in the labeling of the representations, as indicated above, when considering integrality with respect to a lattice \(\Gamma\) generated by the exponentials of \(\tilde{\gamma}_i\) as in 10 .
Lemma 5. Let \(\Gamma\) denote the lattice spanned by \(\gamma_1,\dotsc,\gamma_5\), where \(\gamma_i=\exp\tilde{\gamma}_i\) and \(\tilde{\gamma}_1,\dotsc,\tilde{\gamma}_5\) are as indicated in 10 . Suppose \(\chi\colon\Gamma\to U(1)\) is a unitary character, and let \(a,b,c\) be real numbers such that \(\chi(\gamma_1)=e^{2\pi\mathbf{i}a}\), \(\chi(\gamma_2)=e^{2\pi\mathbf{i}b}\), and \(\chi(\gamma_3)=e^{2\pi\mathbf{i}c/r}\). Then \(L^2(G\times_\chi\mathbb{C})\) decomposes into a countable direct sum of irreducible unitary representations with the following multiplicities:
(I) For \(\alpha,\beta\in\mathbb{R}\) the representation \(\rho_{\alpha,\beta}\) appears with multiplicity \[m\bigl(\rho_{\alpha,\beta}\bigr) =\begin{cases} 1&\text{if \chi|_{\Gamma\cap[G,G]}=1, \alpha\in a+\mathbb{Z}, \beta\in b+\mathbb{Z}, and} \\0&\text{otherwise.} \end{cases}\] The condition \(\chi|_{\Gamma\cap[G,G]}=1\) is equivalent to \(c\in r\mathbb{Z}\) and \(\chi(\gamma_4)=\chi(\gamma_5)=1\).
(II) For \(0\neq\hbar\in\mathbb{R}\) the representation \(\rho_\hbar\) appears with multiplicity \[m\bigl(\rho_\hbar\bigr) =\begin{cases} |\hbar|&\text{if \chi|_{\Gamma\cap Z}=1, \hbar\in c+r\mathbb{Z}, and} \\0&\text{otherwise.} \end{cases}\] The condition \(\chi|_{\Gamma\cap Z}=1\) is equivalent to \(c\in\mathbb{Z}\) and \(\chi(\gamma_4)=\chi(\gamma_5)=1\).
(III) For \(\lambda,\mu,\nu\in\mathbb{R}\) with \((\lambda,\mu)\neq(0,0)\) the multiplicity of the representation \(\rho_{\lambda,\mu,\nu}\) vanishes unless \[\label{E:lm46lat1} \lambda\in r\mathbb{Z},\quad\mu\in r\mathbb{Z},\quad\lambda\tfrac{u-1}2+\mu\tfrac{v-1}2\in c+\mathbb{Z},\tag{31}\] \[\label{E:lm46lat2} e^{2\pi\mathbf{i}(\lambda e+\mu f)}=\chi(\gamma_4),\quad e^{2\pi\mathbf{i}(\lambda g+\mu h)}=\chi(\gamma_5),\tag{32}\] and \[\label{E:lm46lat3} \nu=\nu_0\mod r\mathbb{Z}\tag{33}\] where \[\label{E:nu046def} \nu_0:=2(a\mu-b\lambda)+\frac{\lambda^2\mu^2}{12d^2}+\frac{\bigl(2w-(\lambda+\mu)+\lambda\mu/d\bigr)^2}{4}\tag{34}\] with \(d:=\gcd(\lambda,\mu)\) and \[\label{E:w46def} w:=c-\lambda\tfrac{u-1}2-\mu\tfrac{v-1}2.\tag{35}\] In this case, the multiplicity is \[\label{E:mlmn} m\bigl(\rho_{\lambda,\mu,\nu}\bigr) =\sharp\Bigl\{k\in\mathbb{Z}/\tfrac dr\mathbb{Z} \Bigm|\nu=\nu_0+rk(rk+d)+2rkw\mod 2d\mathbb{Z} \Bigr\}.\tag{36}\]
Proof. We specialize Lemma 4 to \(N=G\). Suppose \(f\in\mathfrak g^*\) and put \(f_i=f(X_i)\).
Let us begin by considering the case \(f|_{[\mathfrak g,\mathfrak g]}=0\), i.e., \(f_3=f_4=f_5=0\). Via Kirillov’s construction, such a functional induces a representation isomorphic to the scalar representation labeled \(\rho_{f_1,f_2}\) in 27 . In this case, \(\mathfrak m=\mathfrak g\) is the unique (rational) maximal subordinated subalgebra. Hence, \(M=G\). The corresponding maximal character \((\bar f,M)\) is rational iff \(f_1\) and \(f_2\) are both rational numbers, cf. 10 and 26 . Moreover, \(\bar f|_{\Gamma\cap M}=\chi|_{\Gamma\cap M}\) if and only if \(\bar f(\gamma_i)=\chi(\gamma_i)\) for \(i=1,\dotsc,5\). As \(\bar f(\gamma_i)=e^{2\pi\mathbf{i}f_i}\) for \(i=1,2\) and \(\bar f(\gamma_i)=1\) for \(i=3,4,5\), this is the case iff \(f_1\in a+\mathbb{Z}\), \(f_2\in b+\mathbb{Z}\), and \(1=\chi(\gamma_3)=\chi(\gamma_4)=\chi(\gamma_5)\). As \((\bar f,M)\) is fixed under the action of \(G\), each of these representations occurs with multiplicity one in view of Lemma 4. According to Lemma 1, the group \(\Gamma\cap[G,G]\) is generated by \(\gamma_3,[\gamma_1,\gamma_3],[\gamma_2,\gamma_3],\gamma_3^r[\gamma_1,\gamma_2]^{-1},\gamma_4,\gamma_5\), see 17 and 18 . Hence, the condition \(\chi|_{\Gamma\cap[G,G]}=1\) is equivalent to \(\chi(\gamma_3)=\chi(\gamma_4)=\chi(\gamma_5)=1\).
Let us next consider the case \(f|_{\mathfrak z}=0\) and \(f_{[\mathfrak g,\mathfrak g]}\neq0\), i.e., \(f_4=f_5=0\) and \(f_3\neq0\). Via Kirillov’s construction, such a functional induces a representation isomorphic to the Schrödinger representation labeled \(\rho_{f_3}\) in 28 . In this case, the subspace \(\mathfrak m\) spanned by \(X_2,X_3,X_4,X_5\) is a rational maximal subordinated subalgebra which is stable under the action of \(G\). Using Lemma 1, we see that the group \(\Gamma\cap M\) is generated by \[\gamma_2,\quad\gamma_3,\quad\text{and}\quad\Gamma\cap Z.\] The corresponding maximal character \((\bar f,M)\) is rational iff \(f_2\) and \(f_3\) are both rational numbers. Moreover, \(\bar f|_{\Gamma\cap M}=\chi|_{\Gamma\cap M}\) if and only if \(\bar f(\gamma_2)=\chi(\gamma_2)\), \(\bar f(\gamma_3)=\chi(\gamma_3)\), and \(\bar f|_{\Gamma\cap Z}=\chi|_{\Gamma\cap Z}\). Equivalently, \(f_2\in b+\mathbb{Z}\), \(f_3\in c+\mathbb{Z}\), and \(1=\chi|_{\Gamma\cap Z}\). Suppose \(g=\exp(\sum_{i=1}^5x_iX_i)\). A straightforward calculation yields \((\mathop{\mathrm{Ad}}_g^*f)(X_2)=f_2+x_1f_3\) and \((\mathop{\mathrm{Ad}}_g^*f)(X_i)=f_i\) for \(i=3,4,5\). Hence, \((\bar f^g,M^g=M)\) is rational and \(\bar f^g|_{\Gamma\cap M^g}=\chi|_{\Gamma\cap M^g}\) iff \(f_2+x_1f_3\in b+\mathbb{Z}\), \(f_3\in c+\mathbb{Z}\), and \(\chi|_{\Gamma\cap Z}=1\). As integral \(x_1\) correspond to \(g\in\Gamma\), we conclude form Lemma 4, that the multiplicity is \(|f_3|\), provided \(f_3\in c+\mathbb{Z}\), and \(\chi|_{\Gamma\cap Z}=1\). According to Lemma 1, the group \(\Gamma\cap Z\) is generated by \([\gamma_1,\gamma_3],[\gamma_2,\gamma_3],\gamma_3^r[\gamma_1,\gamma_2]^{-1},\gamma_4,\gamma_5\). Hence, the condition \(\chi|_{\Gamma\cap Z}=1\) is equivalent to \(c\in\mathbb{Z}\) and \(1=\chi(\gamma_4)=\chi(\gamma_5)\).
Let us finally turn to the case \(f|_{\mathfrak z}\neq0\), i.e., \((f_4,f_5)\neq(0,0)\). Via Kirillov’s construction, such a functional induces a representation isomorphic to the generic representation labeled \(\rho_{\lambda,\mu,\nu}\) in 29 where, cf. 30 , \[\label{E:lmn46v46fi} \lambda=f_4,\qquad\mu=f_5,\qquad\nu=f_3^2+2(f_1f_5-f_2f_4).\tag{37}\] In this case, the subspace \(\mathfrak m\) spanned by \(f_5X_1-f_4X_2,X_3,X_4,X_5\) is the unique maximal subordinated subalgebra. We assume that the corresponding maximal character \((\bar f, M)\) is rational, i.e., \(f_1f_5-f_2f_4,f_3,f_4,f_5\) are all rational. As the group \(\Gamma\cap Z\) is generated by \([\gamma_1,\gamma_3],[\gamma_2,\gamma_3],\gamma_3^r[\gamma_1,\gamma_2]^{-1},\gamma_4,\gamma_5\), we have \(\bar f|_{\Gamma\cap Z}=\chi|_{\Gamma\cap Z}\) if and only if 31 and 32 hold true, cf. 17 , 18 , and 10 . We assume from now on that this is the case. In particular, \(f_4,f_5\) are integral and we write \(d=\gcd(f_4,f_5)\in r\mathbb{Z}\). Using Lemma 1 and 9 we see that the group \(\Gamma\cap M\) is generated by \[\gamma_1^{f_5/d}\gamma_2^{-f_4/d}=\exp\begin{pmatrix}f_5/d\\-f_4/d\\-f_4f_5/2d^2\\-f_4f_5^2/12d^3\\-f_4^2f_5/12d^3\end{pmatrix},\quad\gamma_3=\exp\begin{pmatrix}0\\0\\1/r\\u/2r\\v/2r\end{pmatrix},\quad\text{and}\quad\Gamma\cap Z.\] Hence, \(\bar f|_{\Gamma\cap M}=\chi|_{\Gamma\cap M}\) if and only if (furthermore) \[\frac{f_1f_5-f_2f_4}{d}-\frac{f_3f_4f_5}{2d^2}-\frac{f_4^2f_5^2}{6d^3}=\frac{af_5-bf_4}{d}\mod\mathbb{Z}\] and \[f_3/r+(f_4u+f_5v)/2r=c/r\mod\mathbb{Z}.\] Using 37 , 35 and 34 , this is readily seen to be equivalent to \[\nu=\nu_0+\bigl(f_3+\lambda\mu/2d\bigr)^2-\bigl(w-(\lambda+\mu)/2+\lambda\mu/2d\bigr)^2\mod 2d\mathbb{Z}\] and \[f_3=w-(\lambda+\mu)/2\mod r\mathbb{Z}.\] This is the case if and only if there exists an integer \(k\) such that \[f_3=rk+w-(\lambda+\mu)/2\] and \[\nu=\nu_0+rk\bigl(rk+2w-(\lambda+\mu)+\lambda\mu/d\bigr)\mod2d\mathbb{Z}.\] The latter can be replaced with the equivalent condition \[\nu=\nu_0+rk(rk+d)+2rkw\mod2d\mathbb{Z},\] for we have \[d-(\lambda+\mu)+\lambda\mu/d=d(1-\lambda/d)(1-\mu/d)\in2d\mathbb{Z},\] as \(\lambda/d\) and \(\mu/d\) can not both be even. In particular, 33 must hold in this case. Recall that \(\lambda\), \(\mu\) and \(\nu\) are invariant under coadjoint action by \(G\). Moreover, for \(g=\exp(x_1X_1+\cdots+x_5X_5)\), we have \((\mathop{\mathrm{Ad}}_g^*f)(X_3)=f_3+x_1f_4+x_2f_5\). Hence, \((\bar f,M)\) and \((\bar f^g,M^g)\) lie in the same \(\Gamma\)-orbit iff \((\mathop{\mathrm{Ad}}_g^*f)(X_3)-f_3\) is divisible by \(d\). The formula for the multiplicity now follows from Lemma 4. ◻
For integers \(l,r,w,n\in\mathbb{Z}\) with \(l,r\geq1\) we define \[\label{E:mult} m(l,r,w,n):=\sharp\Bigl\{k\in\mathbb{Z}/l\mathbb{Z}\Bigm|rk(k+l)+2wk\equiv n\mod2l\mathbb{Z}\Bigr\}.\tag{38}\] Clearly, \[\label{E:M46mult} m(l,r,w,n+2l)=m(l,r,w,n)\tag{39}\] and \[\label{E:m46total} \sum_{n=1}^{2l}m(l,r,w,n)=l.\tag{40}\]
With this notation, the statement in Lemma 5 may be expressed in the form: \[\begin{gather} \label{E:rep46deco} L^2(G\times_\chi\mathbb{C}) =\underbrace{\bigoplus_{(\alpha,\beta)\in(a,b)+\mathbb{Z}^2}\rho_{\alpha,\beta}}_{\substack{\textrm{only appears if}\\\textrm{\chi|_{\Gamma\cap[G,G]}=1}}} \,\,\,\oplus\,\,\,\underbrace{\bigoplus_{\hbar\in c+r\mathbb{Z}}|\hbar|\rho_\hbar}_{\substack{\textrm{only appears if}\\\textrm{\chi|_{\Gamma\cap Z}=1}}} \\ \oplus\bigoplus_{\substack{(0,0)\neq(\lambda,\mu)\in\\(\lambda_0,\mu_0)+(\Gamma'')^*}}\quad\bigoplus_{n\in\mathbb{Z}}\,\,\,m\bigl(\tfrac dr,r,w,n\bigr)\rho_{\lambda,\mu,\nu_0+rn}. \end{gather}\tag{41}\] Here \(\lambda_0,\mu_0\in r\mathbb{Z}\) are as in Lemma 3, and \[\label{E:Gamma393942} (\Gamma'')^*=\left\{(l,m)\in\mathbb{R}^2\middle|\begin{array}{c}\frac{l}{r}\in\mathbb{Z},\frac{m}{r}\in\mathbb{Z},l\frac{u-1}{2}+m\frac{v-1}{2}\in\mathbb{Z}\\le+mf\in\mathbb{Z},lg+mh\in\mathbb{Z}\end{array} \right\} \subseteq(r\mathbb{Z})\times(r\mathbb{Z})\tag{42}\] denotes the lattice dual to the lattice \(\Gamma''\) spanned by the vectors in 11 . Moreover, \(d=\gcd(\lambda,\mu)\), and \(w\), \(\nu_0\) are defined 34 and 35 .
In this section we use the decomposition in 41 to decompose the twisted Rumin complex associated with a standard (2,3,5) distribution on the nilmanifold \(\Gamma\setminus G\), a standard fiberwise graded Euclidean inner product on \(\mathfrak t(\Gamma\setminus G)\), and a unitary character \(\chi\colon\Gamma\to U(1)\). This yields a decomposition of the corresponding zeta function, cf. 8 .
We continue to use a graded basis \(X_1,\dotsc,X_5\) of \(\mathfrak g\) satisfying the relations in 1 . Let \(\mathcal{D}_G\) denote the left invariant distribution on \(G\) spanned by \(X_1\) and \(X_2\). Left translation provides a trivialization of the tangent bundle that induces a trivialization of the bundle of osculating algebras, \(\mathfrak tG=G\times\mathfrak g\). Passing to the fiberwise Lie algebra cohomology, we obtain a trivialization of the vector bundle \(\mathcal{H}^q(\mathfrak tG)=G\times H^q(\mathfrak g)\). Via this identification, the untwisted Rumin differential is a left invariant differential operator \(D_q\colon C^\infty(G)\otimes H^q(\mathfrak g)\to C^\infty(G)\otimes H^{q+1}(\mathfrak g)\) which may be considered as \[\label{E:Dq46Ug} D_q\in\mathcal{U}(\mathfrak g)\otimes L\bigl(H^q(\mathfrak g),H^{q+1}(\mathfrak g)\bigr)\tag{43}\] where \(\mathcal{U}(\mathfrak g)\) denotes the universal enveloping algebra of \(\mathfrak g\). With respect to a particular basis of \(H^q(\mathfrak g)\), the Rumin differentials take the form: \[\begin{align} D_0&=\begin{pmatrix}X_1\\ X_2\end{pmatrix} \\ D_1&=\begin{pmatrix}-X_{112}-X_{13}-X_4&X_{111}\\-\sqrt2X_{122}-\sqrt2X_5&\sqrt2X_{211}-\sqrt2X_4\\-X_{222}&X_{221}-X_{23}-X_5\end{pmatrix} \\ D_2&=\begin{pmatrix}-X_{12}-X_3&X_{11}/\sqrt2&0\\-X_{22}/\sqrt2&-\tfrac32X_3&X_{11}/\sqrt2\\0&-X_{22}/\sqrt2&X_{21}-X_3\end{pmatrix} \\ D_3&=\begin{pmatrix}X_{122}+X_{32}-X_5&-\sqrt2X_{112}+\sqrt2X_4&X_{111}\\X_{222}&-\sqrt2X_{221}-\sqrt2X_5&X_{211}-X_{31}+X_4\end{pmatrix} \\ D_4&=\begin{pmatrix}-X_2,X_1\end{pmatrix} \end{align}\] Here we are using the notation \(X_{j_1\cdots j_k}=X_{j_1}\cdots X_{j_k}\). These formulas for \(D_q\) have been derived in [53], where they are expressed with respect to a slightly different basis, see also [39] or [30].
Suppose \(\Gamma\) is a lattice in \(G\) that is of the form considered in Section 2, i.e., generated by the exponentials of \(\tilde{\gamma}_i\) as in 10 . Furthermore, let \(\chi\colon\Gamma\to U(1)\) be a unitary character. For the sections of the associated flat line bundle \(F_\chi\) over the nilmanifold \(\Gamma\setminus G\) we have a canonical identification \[\label{E:F46chi46sec} \Gamma^\infty(F_\chi) =C^\infty(G\times_\chi\mathbb{C}) :=\left\{f\in C^\infty(G,\mathbb{C})\middle|\begin{array}{c}f(\gamma g)=\chi(\gamma)f(g)\\\text{for \gamma\in\Gamma and g\in G}\end{array}\right\}.\tag{44}\] The left invariant 2-plane field \(\mathcal{D}_G\) descends to a generic distribution of rank two on the nilmanifold \(\Gamma\setminus G\) which we denote by \(\mathcal{D}_{\Gamma\setminus G}\). The trivialization \(\mathcal{H}^q(\mathfrak tG)=G\times H^q(\mathfrak g)\) mentioned above gives rise to a trivialization of the vector bundle \(\mathcal{H}^q(\mathfrak t(\Gamma\setminus G))=(\Gamma\setminus G)\times H^q(\mathfrak g)\). Combining this with 44 we obtain a canonical identification \[\label{E:HqF46G} \Gamma^\infty\bigl(\mathcal{H}^q(\mathfrak t(\Gamma\setminus G))\otimes F_\chi\bigr)=C^\infty(G\times_\chi\mathbb{C})\otimes H^q(\mathfrak g).\tag{45}\] Via this identification the twisted Rumin differentials become operators \[\label{E:DqG} D_q\colon C^\infty(G\times_\chi\mathbb{C})\otimes H^q(\mathfrak g)\to C^\infty(G\times_\chi\mathbb{C})\otimes H^{q+1}(\mathfrak g)\tag{46}\] which are given by the matrices in 43 , with the universal algebra now acting in the (induced) representation \(C^\infty(G\times_\chi\mathbb{C})\).
Let \(g\) denote the graded Euclidean inner product on \(\mathfrak g\) that turns \(X_1,\dotsc,X_5\) into an orthonormal basis. The corresponding left invariant fiberwise graded Euclidean inner product on the bundle of osculating algebras \(\mathfrak tG=G\times\mathfrak g\) descends to a fiberwise graded Euclidean inner product \(g_{\Gamma\setminus G}\) on the bundle of osculating algebras \(\mathfrak t(\Gamma\setminus G)\) over \(\Gamma\setminus G\). The flat line bundle \(F_\chi\) comes with a canonical fiberwise Hermitian inner product denoted by \(h_\chi\). The metrics \(g_{\Gamma\setminus G}\) and \(h_\chi\) provide an \(L^2\) inner product on the space \(\Gamma^\infty\bigl(\mathcal{H}^q(\mathfrak t(\Gamma\setminus G))\otimes F_\chi\bigr)\). Via the identification in 45 this inner product corresponds to the tensor product of the standard \(L^2\) inner product on the induced representation \(C^\infty(G\times_\chi\mathbb{C})\) and the Hermitian inner product on \(H^q(\mathfrak g)\) induced by \(g\).
In view of 45 , the decomposition of the induced representation described in 41 provides a decomposition of the twisted Rumin complex over \(\Gamma\setminus G\) into a countable orthogonal direct sum of Rumin complexes, each associated with an irreducible unitary representations of \(G\). For every irreducible unitary representation \(\rho\) of \(G\), we let \(\rho(D_q)\) denote the operator induced by 43 in this representation, and consider the corresponding Rumin–Seshadri operator, \[\label{E:Delta46rho} \Delta_{\rho,q}:=\bigl(\rho(D_{q-1})\rho(D_{q-1})^*\bigr)^{a_{q-1}}+\bigl(\rho(D_q)^*\rho(D_q)\bigr)^{a_q}.\tag{47}\] Moreover, we put \[\label{E:zeta46rho46def} \zeta_\rho(s) :=\mathop{\mathrm{str}}\bigl(N\Delta_\rho^{-s}\bigr) :=\sum_{q=0}^5(-1)^qN_q\mathop{\mathrm{tr}}\Delta_{\rho,q}^{-s}.\tag{48}\] These zeta functions are known to converge for \(\Re s\) sufficiently large, depending on \(\rho\), and they admit analytic continuation to meromorphic functions on the entire complex plane which are holomorphic at \(s=0\), see [30]. As the Rumin complex is a Rockland complex, the operators \(\Delta_{\rho,q}\) have trivial kernel for nontrivial \(\rho\). Clearly, \(\Delta_{\rho,q}\) vanishes if \(\rho\) is the trivial representation, and so does \(\zeta_\rho(s)\).
Using the notation from Lemma 5 we define \[\label{E:zeta46I} \zeta_{\mathrm{I},\Gamma,\chi}(s) :=\sideset{}{'}\sum_{(\alpha,\beta)\in(a,b)+\mathbb{Z}^2}\zeta_{\rho_{\alpha,\beta}}(s)\tag{49}\] if \(\chi|_{\Gamma\cap[G,G]}=1\), and \(\zeta_{\mathrm{I},\Gamma,\chi}(s):=0\) otherwise. Moreover, \[\label{E:zeta46II} \zeta_{\mathrm{I\!I},\Gamma,\chi}(s) :=\sideset{}{'}\sum_{\hbar\in c+r\mathbb{Z}}|\hbar|\zeta_{\rho_\hbar}(s)\tag{50}\] if \(\chi|_{\Gamma\cap Z}=1\), and \(\zeta_{\mathrm{I\!I},\Gamma,\chi}(s):=0\) otherwise. Finally, \[\label{E:zeta46III} \zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s) :=\sideset{}{'}\sum_{\substack{(\lambda,\mu)\in\\(\lambda_0,\mu_0)+(\Gamma'')^*}}\,\,\,\sum_{n\in\mathbb{Z}}\,\,\,m(\tfrac dr,r,w,n)\zeta_{\rho_{\lambda,\mu,\nu_0+nr}}(s).\tag{51}\] Here \(\lambda_0,\mu_0\) are as in Lemma 3, \((\Gamma'')^*\) is the dual lattice in 42 , \(d=\gcd(\lambda,\mu)\), \(w\) is given in 35 , \(\nu_0\) is given in 34 , and the multiplicity is defined in 38 . Moreover, we are using the common convention to decorate the summation symbol with a prime to indicate that the summand with index zero (if any) is omitted.
Lemma 6. For all unitary characters \(\chi\colon\Gamma\to U(1)\) and \(\Re s>10/2\kappa\) we have \[\label{E:zeta46deco} \zeta_{\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi}(s) =\zeta_{\mathrm{I},\Gamma,\chi}(s)+\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)+\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\tag{52}\] where the left hand side has been defined in 6 .
Proof. Via the identification in 45 the decomposition in 41 provides a decomposition of the twisted Rumin complex \(D_*\) in 46 into a countable orthogonal direct sum of complexes, \[\label{E:comp46deco} D_* =\bigoplus_\rho m(\rho)\cdot\rho(D_*),\tag{53}\] where the sum is over all irreducible unitary representations \(\rho\) of \(G\) and the multiplicities are given in Lemma 5. By 5 and 47 , and since \(\rho(\Delta_q)=\Delta_{\rho,q}\), this yields \[\Delta_q =\bigoplus_\rho m(\rho)\cdot\Delta_{\rho,q}.\] Using 6 and 48 , we get \[\zeta_{\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi}(s) =\sideset{}{'}\sum_\rho m(\rho)\cdot\zeta_\rho(s).\] Both sides converge for \(\Re s>10/2\kappa\) in view of [43]. We omit the trivial representation \(\rho\) on the right hand side because \(\zeta_\rho(s)\) vanishes identically for this \(\rho\). Combining this with the description of the multiplicities in Lemma 5 and the definitions in 49 – 51 , we obtain the lemma. ◻
Subsequently, we will analyze each of the three summands in 52 individually.
Remark 1. Let us specialize to the simple lattice \(\Gamma=\Gamma_0\) generated by \(X_1\) and \(X_2\). In this case \(r=1\), \(u=v=1\), \(e=f=g=h=0\), \(c=0\), and \(w=0\). Hence, \[\zeta_{\mathrm{I},\Gamma_0,\chi}(s) =\sideset{}{'}\sum_{(\alpha,\beta)\in(a,b)+\mathbb{Z}^2}\zeta_{\rho_{\alpha,\beta}}(s),\qquad \zeta_{\mathrm{I\!I},\Gamma_0,\chi}(s) =\sideset{}{'}\sum_{\hbar\in\mathbb{Z}}|\hbar|\zeta_{\rho_\hbar}(s)\] and \[\zeta_{\mathrm{I\!I\!I},\Gamma_0,\chi}(s) :=\sideset{}{'}\sum_{(\lambda,\mu)\in\mathbb{Z}^2}\,\,\,\sum_{n\in\mathbb{Z}}\,\,\,m(d,n)\zeta_{\rho_{\lambda,\mu,\nu_0+n}}(s)\] where \(d=\gcd(\lambda,\mu)\), \[\label{E:mdn} m(d,n) :=m(d,1,0,n) =\sharp\Bigl\{k\in\mathbb{Z}/d\mathbb{Z}\Bigm|k(k+d)\equiv n\mod2d\mathbb{Z}\Bigr\},\tag{54}\] and \(\nu_0\) as in 34 with \(w=0\). For this lattice \(\Gamma_0\cap[G,G]=[\Gamma_0,\Gamma_0]\) by Lemma 1, \(\Gamma_0/[\Gamma_0,\Gamma_0]\cong\mathbb{Z}^2\) by 21 , and \(\hom(\Gamma_0,U(1))\cong U(1)\times U(1)\) according to 22 .
We continue to use the notation set up in the preceding section and Lemma 5. In particular, \(\Gamma\) denotes the lattice in \(G\) spanned by the exponentials of \(\tilde{\gamma}_i\) as in 10 , \(\chi\colon\Gamma\to U(1)\) is a unitary character, and \(g\) is a graded Euclidean inner product on \(\mathfrak g\) such that \(X_1,\dotsc,X_5\) is an orthonormal basis of \(\mathfrak g\). This implies, that every graded automorphism \(\phi\in\mathop{\mathrm{Aut}}_\textrm{gr}(\mathfrak g)=\mathop{\mathrm{GL}}(\mathfrak g_{-1})\cong\mathop{\mathrm{GL}}_2(\mathbb{R})\) that preserves \(g|_{\mathfrak g_{-1}}\) also preserves \(g\). By naturality of the Rumin differentials, \(\phi\cdot D_q=D_q\) where the dot denotes the natural left action of \(\mathop{\mathrm{Aut}}_\textrm{gr}(\mathfrak g)\) on \(\mathcal{U}(\mathfrak g)\otimes L\bigl(H^q(\mathfrak g),H^{q+1}(\mathfrak g)\bigr)\), cf. 43 . Hence, if \(\rho\) is an irreducible unitary representation of \(G\), then \[\label{E:zeta46phi} \zeta_{\rho\circ\phi}(s)=\zeta_\rho(s)\tag{55}\] for every graded automorphism \(\phi\in\mathop{\mathrm{Aut}}_\textrm{gr}(\mathfrak g)\) preserving \(g|_{\mathfrak g_{-1}}\). Moreover, for \(\tau>0\) we have \[\label{E:zeta46phit} \zeta_{\rho\circ\phi_\tau}(s)=\tau^{2\kappa s}\zeta_{\rho}(s)\tag{56}\] where \(\phi_\tau\in\mathop{\mathrm{Aut}}_\textrm{gr}(\mathfrak g)\) denotes the grading automorphism acting by \(\tau^j\) on \(\mathfrak g_j\). For more details we refer to [30].
Lemma 7. If \(\chi|_{\Gamma\cap[G,G]}=1\), then the sum in 49 converges for \(\Re s>1/\kappa\), and \[\label{E:factor46I} \zeta_{\mathrm{I},\Gamma,\chi}(s) =Z_\mathrm{Epst}(2\kappa s;a,b)\cdot\zeta_{\rho_{1,0}}(s)\tag{57}\] where \[Z_\mathrm{Epst}(s;a,b):=\sideset{}{'}\sum_{(\alpha,\beta)\in(a,b)+\mathbb{Z}^2}\bigl(\alpha^2+\beta^2\bigr)^{-s/2}\] denotes an Epstein zeta function. In particular, \(\zeta_{\mathrm{I},\Gamma,\chi}(s)\) extends to a meromorphic function on the entire complex plane which has a single (simple) pole at \(s=1/\kappa\) with residue \(\frac{\pi}{\kappa}\cdot\zeta_{\rho_{1,0}}(1/\kappa)\). Moreover, for all unitary characters \(\chi\), \[\label{E:zeta46I460} \zeta_{\mathrm{I},\Gamma,\chi}(0)=0, \qquad \zeta_{\mathrm{I},\Gamma,\chi}'(0) =\begin{cases}0&\text{if \chi is nontrivial, and}\\2\kappa\log2&\text{if \chi is trivial.}\end{cases}\tag{58}\]
Proof. W.l.o.g.\(\chi|_{\Gamma\cap[G,G]}=1\). The homogeneity in 56 gives \[\zeta_{\rho_{\tau\alpha,\tau\beta}}(s)=\tau^{-2\kappa s}\zeta_{\rho_{\alpha,\beta}}(s),\qquad\tau>0.\] For \((\alpha,\beta)\neq(0,0)\) we may apply 55 with a graded automorphism \(\phi\in\mathop{\mathrm{Aut}}_\textrm{gr}(\mathfrak g)\) that acts as a rotation on \(\mathfrak g_{-1}\), and then use the latter homogeneity to obtain \[\zeta_{\rho_{\alpha,\beta}}(s)=(\alpha^2+\beta^2)^{-\kappa s}\cdot\zeta_{\rho_{1,0}}(s).\] Summing over all \((\alpha,\beta)\) in the shifted lattice, we obtain 57 , cf. 49 .
As the scalar representations \(\rho_{\alpha,\beta}\) are one dimensional, \(\zeta_{\rho_{\alpha,\beta}}(s)\) is an entire function, and 48 yields \[\label{E:qwerty97} \zeta_{\rho_{\alpha,\beta}}(0)=\mathop{\mathrm{str}}(N)=\sum_{q=0}^5(-1)^qN_q\dim H^q(\mathfrak g)=0.\tag{59}\] By [30], \[\label{E:qwerty98} \exp\left(\tfrac1{2\kappa}\zeta_{\rho_{\alpha,\beta}}'(0)\right) =\frac{1}{2}.\tag{60}\]
The Epstein [54] zeta function \(Z_\mathrm{Epst}(s;a,b)\) converges for \(s>2\) and extends to a meromorphic on the entire complex plane which has a single (simple) pole at \(s=2\) with residue \(2\pi\), and \[\label{E:qwerty99} Z_\mathrm{Epst}(0;a,b)= \begin{cases} -1&\text{if a,b\in\mathbb{Z}, and}\\ 0&\text{otherwise.} \end{cases}\tag{61}\] Note that \(a,b\in\mathbb{Z}\) if and only if \(\chi\) is trivial, cf. Lemma 5. Combining this with 57 we see that the sum in 49 converges for \(\Re s>1/\kappa\), and that \(\zeta_{\mathrm{I},\Gamma,\chi}(s)\) extends to a meromorphic function on the entire complex plane which has a single (simple) pole at \(s=1/\kappa\) of residue \(\frac{2\pi}{2\kappa}\cdot\zeta_{\rho_{1,0}}(1/\kappa)\). Furthermore, combining 59 –61 with 57 we obtain 58 . ◻
Lemma 8. If \(\chi|_{\Gamma\cap Z}=1\), then the sum in 50 converges for \(\Re s>2/\kappa\) and \[\label{E:factor46II} \zeta_{\mathrm{I\!I},\Gamma,\chi}(s) =r^{-\kappa s+1}\cdot Z_\mathrm{Epst}(\kappa s-1;\tfrac cr)\cdot\zeta_{\rho_1}(s)\tag{62}\] where \[Z_\mathrm{Epst}(s;a):=\sideset{}{'}\sum_{n\in\mathbb{Z}}|a+n|^{-s}\] denotes an Epstein zeta function. In particular, \(\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)\) extends to a meromorphic function on the entire complex plane which has a simple pole at \(s=2/\kappa\) with residue \(\frac{2}{r\kappa}\cdot\zeta_{\rho_1}(2/\kappa)\). If \(c\notin r\mathbb{Z}\), then this is the only pole of \(\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)\). If \(c\in r\mathbb{Z}\), then \(\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)\) has one further (simple) pole at \(s=1/\kappa\) with residue \(-\mathop{\mathrm{res}}_{s=1/\kappa}\zeta_{\rho_1}(1/\kappa)\). Moreover, for all unitary characters \(\chi\), \[\label{E:zeta46II460} \zeta_{\mathrm{I\!I},\Gamma,\chi}(0)=0 \qquad\text{and}\qquad \zeta_{\mathrm{I\!I},\Gamma,\chi}'(0)=0.\tag{63}\]
Proof. W.l.o.g.\(\chi|_{\Gamma\cap Z}=1\). The homogeneity in 56 gives \[\zeta_{\rho_{\tau^2\hbar}}(s)=\tau^{-2\kappa s}\zeta_{\rho_\hbar}(s),\qquad\tau>0.\] Applying 55 to a graded automorphism \(\phi\in\mathop{\mathrm{Aut}}_\textrm{gr}(\mathfrak g)\) that acts as an isometric reflection on \(\mathfrak g_{-1}\), we obtain \(\zeta_{\rho_{-\hbar}}(s)=\zeta_{\rho_\hbar}(s)\). Hence, \[\zeta_{\rho_\hbar}(s) =|\hbar|^{-\kappa s}\zeta_{\rho_1}(s)\] for all \(0\neq\hbar\in\mathbb{R}\). This immediately yields 62 , cf. 50 . In view of [30] the function \(\zeta_{\rho_1}(s)\) is meromorphic on the entire complex plane and its poles can only be located at \(s=\frac{1-2j}{\kappa}\), \(j\in\mathbb{N}_0\).
The Epstein [54] zeta function \(Z_\mathrm{Epst}(s;a)\) converges for \(\Re s>1\), it extends to a meromorphic function on the entire complex plane which has a single (simple) pole at \(s=1\) of residue \(2\), it vanishes at \(s\in-2\mathbb{N}\), and it satisfies \[\label{E:qwerty999} Z_\mathrm{Epst}(0;a)= \begin{cases} -1&\text{if a\in\mathbb{Z}, and}\\ 0&\text{otherwise.} \end{cases}\tag{64}\] Combining this with 62 we see that the sum in 50 converges for \(\Re s>2/\kappa\) and that \(\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)\) extends to a meromorphic function on the entire complex plane which has a simple pole at \(s=2/\kappa\) with residue \(\frac{1}{r}\cdot\frac{2}{\kappa}\cdot\zeta_{\rho_1}(2/\kappa)\). If \(c\notin r\mathbb{Z}\) then the zeros of \(Z_\mathrm{Epst}(\kappa s-1;\frac{c}{r})\) cancel all the poles of \(\zeta_{\rho_1}(s)\) and \(\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)\) is holomorphic for \(s\neq2/\kappa\). If \(c\in r\mathbb{Z}\) then all but one pole get canceled and \(\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)\) has one further pole at \(s=1/\kappa\) with residue \(-\mathop{\mathrm{res}}_{s=1/\kappa}\zeta_{\rho_1}(s)\).
From [30] we obtain \[\label{E:qwerty0} \zeta_{\rho_\hbar}(0)=0 \qquad\text{and}\qquad \zeta_{\rho_\hbar}'(0)=0.\tag{65}\] Combining this with 62 and the fact that \(\zeta_\mathrm{Epst}(s;a)\) is holomorphic at \(s=-1\) we obtain 63 . ◻
Remark 2. Recall that \[Z_\mathrm{Epst}(s;a) =\begin{cases} \zeta_\mathrm{Hurw}(s,a)+\zeta_\mathrm{Hurw}(s,1-a)&\text{if 0<a<1, and}\\ 2\zeta_\mathrm{Riem}(s)&\text{if a=0.} \end{cases}\] Here \(\zeta_\mathrm{Hurw}(s,a)=\sum_{n=0}^\infty(n+a)^{-s}\) denotes the Hurwitz zeta function, \(a>0\), and \(\zeta_\mathrm{Riem}(s)=\zeta_\mathrm{Hurw}(s,1)=\sum_{n=1}^\infty n^{-s}\) denotes the Riemann zeta function. In particular, for \(0\leq a<1\), \[\zeta_\mathrm{Epst}(-1;a)=-B_2(a),\] where \(B_2(a)=a^2-a+\frac{1}{6}\) denotes the second Bernoulli polynomial. Indeed, this follows from the classical identities \(\zeta_\mathrm{Hurw}(-1,a)=-\tfrac12B_2(a)\), \(B_2(1-a)=B_2(a)\), and \(\zeta_\mathrm{Riem}(-1)=-\frac{1}{12}=-\tfrac12B_2(0)\).
Lemma 9. Suppose \((0,0)\neq(\lambda,\mu)\in\mathbb{R}^2\), \(\nu_0\in\mathbb{R}\), and \(0\neq d\in\mathbb{R}\). Then \[\sum_{n\in\mathbb{Z}}\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}(s)\] converges for \(\Re s>2/\kappa\), and this function admits a meromorphic continuation to the half plane1 \(\Re s>-1/8\kappa\) which is holomorphic at \(s=0\). Moreover, \[\Bigl(\sum_{n\in\mathbb{Z}}\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}\Bigr)(0)=0 \qquad\text{and}\qquad \Bigl(\sum_{n\in\mathbb{Z}}\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}\Bigr)'(0)=0.\]
Proof. The homogeneity in 56 gives \[\label{E:zeta46homog} \zeta_{\rho_{\tau^3\lambda,\tau^3\mu,\tau^4\nu}}(s) =\tau^{-2\kappa s}\zeta_{\rho_{\lambda,\mu,\nu}}(s),\qquad\tau>0.\tag{66}\] W.l.o.g.we may thus assume \(\lambda^2+\mu^2=1\). Furthermore, we may assume \(d>0\) and \(-d\leq\nu_0\leq d\). From [30] we know \[\label{E:zetap46lmn460} \zeta_{\rho_{\lambda,\mu,\nu}}(0)=0 \qquad\text{and}\qquad \zeta_{\rho_{\lambda,\mu,\nu}}'(0)=0\tag{67}\] for all \(\nu\in\mathbb{R}\).
For \(\nu\leq-d\) we have \[\label{E:zeta-} \zeta_{\rho_{\lambda,\mu,\nu}}(s)=|\nu|^{-\kappa s/2}E(s)+|\nu|^{-2\kappa s+3/2}C_-(s)+R_{\nu,-}(s)\tag{68}\] where \(E(s)\), \(C_-(s)\) and \(R_{\nu,-}(s)\) are meromorphic functions on the entire complex plane whose poles are all contained in the set \[P:=\left(\left\{\frac{3-j}{4\kappa}:j\in\mathbb{N}_0\right\}\cup\left\{\frac{l/2+1-k}{\kappa}:k,l\in\mathbb{N}_0\right\}\right)\setminus(-\mathbb{N}_0),\] and the estimate \[\label{E:R-46esti} R_{\nu,-}(s)=O\left(|\nu|^{-5/4}\right)\tag{69}\] holds uniformly on compact subsets of \(\{s\in\mathbb{C}\setminus P:\Re s\geq-1/8\kappa\}\) and for \(\nu\leq-d\). This follows from the first estimate in [30] by taking, with the notation there, \(N=7/4\), \(\sigma=-1/8\kappa\), and \(\varepsilon=d\). Combining 67 , 68 , and 69 , we see that \[\label{E:ECR-0} E(0)=C_-(0)=R_{\nu,-}(0)=0 \qquad\text{and}\qquad E'(0)=C'_-(0)=R'_{\nu,-}(0)=0.\tag{70}\] From 68 we obtain \[\begin{gather} \label{E:zeta46cont-} \sum_{n=1}^\infty\zeta_{\rho_{\lambda,\mu,\nu_0-2dn}}(s) =(2d)^{-\kappa s/2}\zeta_\mathrm{Hurw}\bigl(\tfrac{\kappa s}2;1-\tfrac{\nu_0}{2d}\bigr)E(s) \\ +(2d)^{-2\kappa s+3/2}\zeta_\mathrm{Hurw}\bigl(2\kappa s-\tfrac32;1-\tfrac{\nu_0}{2d}\bigr)C_-(s) +\sum_{n=1}^\infty R_{\nu_0-2dn,-}(s) \end{gather}\tag{71}\] where the last sum on the right hand side converges uniformly on compact subsets of \(\{s\in\mathbb{C}\setminus P:\Re s\geq-1/8\kappa\}\) by the estimate in 69 . The sums making up the Hurwitz zeta functions converge for \(\Re s>2/\kappa\) and \(\Re s>5/4\kappa\), respectively. Hence, \(\sum_{n=1}^\infty\zeta_{\rho_{\lambda,\mu,\nu_0-2dn}}(s)\) converges for \(\Re s>2/\kappa\), and the right hand side in 71 provides the meromorphic continuation to \(\Re s>-1/8\kappa\). Combining this with 70 and the fact that the Hurwitz zeta function \(\zeta_\mathrm{Hurw}(s;a)\) is holomorphic at \(s=0\) and at \(s=-3/2\), we conclude \[\label{E:zeta-sum} \Bigl(\sum_{n=1}^\infty\zeta_{\rho_{\lambda,\mu,\nu_0-2dn}}\Bigr)(0)=0 \qquad\text{and}\qquad \Bigl(\sum_{n=1}^\infty\zeta_{\rho_{\lambda,\mu,\nu_0-2dn}}\Bigr)'(0)=0.\tag{72}\]
For \(\nu\geq d\) we have \[\label{E:zeta43} \zeta_{\rho_{\lambda,\mu,\nu}}(s) =\nu^{-2\kappa s+3/2}C_+(s)+R_{\nu,+}(s)\tag{73}\] where \(C_+(s)\) and \(R_{\nu,+}(s)\) are meromorphic functions on the entire complex plane whose poles are all contained in the set \(P\), and the estimate \[\label{E:R4346esti} R_{\nu,+}(s)=O\left(\nu^{-5/4}\right)\tag{74}\] holds uniformly on compact subsets of \(\{s\in\mathbb{C}\setminus P:\Re s\geq-1/8\kappa\}\) and for \(\nu\geq d\). This follows from the second estimate in [30] by taking again, with the notation there, \(N=7/4\), \(\sigma=-1/8\kappa\), and \(\varepsilon=d\). Combining 67 , 73 , and 74 , we see that \[\label{E:ECR430} C_+(0)=R_{\nu,+}(0)=0 \qquad\text{and}\qquad C'_+(0)=R'_{\nu,+}(0)=0.\tag{75}\] From 73 we obtain \[\begin{gather} \label{E:zeta46cont43} \sum_{n=1}^\infty\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}(s) =(2d)^{-2\kappa s+3/2}\zeta_\mathrm{Hurw}\bigl(2\kappa s-\tfrac32;1+\tfrac{\nu_0}{2d}\bigr)C_+(s) \\ +\sum_{n=1}^\infty R_{\nu_0+2dn,+}(s) \end{gather}\tag{76}\] where the last sum on the right hand side converges on compact subsets of \(\{s\in\mathbb{C}\setminus P:\Re s\geq-1/8\kappa\}\) by the estimate in 74 . As before we conclude that \(\sum_{n=1}^\infty\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}(s)\) converges for \(\Re s>2/\kappa\), and the right hand side in 76 provides the meromorphic continuation to \(\Re s>-1/8\kappa\). Using 75 this yields \[\Bigl(\sum_{n=1}^\infty\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}\Bigr)(0)=0 \qquad\text{and}\qquad \Bigl(\sum_{n=1}^\infty\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}\Bigr)'(0)=0.\] Combining this with 72 and 67 we obtain the lemma. ◻
Lemma 10. The sum in 51 converges for \(\Re s>10/2\kappa\), and \[\label{E:factor46III} \zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s) =\tfrac1r\cdot Z^{(\Gamma'')^*}_\mathrm{Epst}\bigl(\tfrac{2\kappa s-4}3;\lambda_0,\mu_0\bigr)\cdot f(s)+\hat{R}(s),\tag{77}\] where \[\label{E:Z46def} Z_\mathrm{Epst}^{(\Gamma'')^*}\bigl(s;\lambda_0,\mu_0\bigr) :=\sideset{}{'}\sum_{\substack{(\lambda,\mu)\in\\(\lambda_0,\mu_0)+(\Gamma'')^*}}(\lambda^2+\mu^2)^{-s/2}\tag{78}\] denotes an Epstein zeta function, \(f(s)\) denotes a meromorphic function defined in 86 below, and \(\hat{R}(s)\) is an entire function. In particular, \(\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\) extends to a meromorphic function on the entire complex plane which has at most a simple pole at \(s=10/2\kappa\) with residue \(\frac{3\pi}{\kappa r}\cdot\mathop{\mathrm{Area}}(\mathbb{R}^2/\Gamma'')\cdot f(10/2\kappa)\). 2 If \(\chi|_{\Gamma\cap Z}\neq1\), then this is the only pole of \(\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\). If \(\chi|_{\Gamma\cap Z}=1\), then \(\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\) has one further (simple) pole at \(s=2/\kappa\) with residue \(-\frac{1}{r}\cdot\mathop{\mathrm{res}}_{s=2/\kappa}f(s)\). 3 Moreover, \[\label{E:zeta46III460} \zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(0)=0 \qquad\text{and}\qquad \zeta_{\mathrm{I\!I\!I},\Gamma,\chi}'(0)=0.\tag{79}\]
Proof. There exists a graded automorphism \(\phi\in\mathop{\mathrm{Aut}}_\textrm{gr}(\mathfrak g)\) preserving \(g\) and acting as a rotation on \(\mathfrak g_{-3}\) such that \(\rho_{\lambda,\mu,\nu}\circ\phi\) is unitarily equivalent to \(\rho_{\sqrt{\lambda^2+\mu^2},0,\nu}\), cf. [30]. Hence, [30] \[\label{E:rot46III} \mathop{\mathrm{tr}}\left(e^{-t\rho_{\lambda,\mu,\nu}(\Delta_q)}\right) =\mathop{\mathrm{tr}}\left(e^{-t\rho_{\sqrt{\lambda^2+\mu^2},0,\nu}(\Delta_q)}\right).\tag{80}\] For \(t>0\) put \[\label{E:theta46def} \vartheta_{\lambda,\mu,\nu}(t) :=\mathop{\mathrm{str}}\left(Ne^{-t\rho_{\lambda,\mu,\nu}(\Delta)}\right) :=\sum_{q=0}^5(-1)^qN_q\mathop{\mathrm{tr}}\left(e^{-t\rho_{\lambda,\mu,\nu}(\Delta_q)}\right).\tag{81}\] Using 80 and [30] we conclude that there exist a Schwartz function \(k\in\mathcal{S}(\mathbb{R}^3)\) such that \[\label{E:theta46k} \vartheta_{\lambda,\mu,\nu}(t) =\frac{t^{-3/4\kappa}}{\sqrt[4]{\lambda^2+\mu^2}}\int_{-\infty}^\infty k\begin{pmatrix}\frac{t^{2/4\kappa}\nu}{2\sqrt{\lambda^2+\mu^2}}+\frac{x^2}{2}\\t^{3/4\kappa}\sqrt[4]{\lambda^2+\mu^2}x\\t^{6/4\kappa}\sqrt{\lambda^2+\mu^2}\end{pmatrix}dx.\tag{82}\] According to [30] the function \(k\) enjoys the symmetry \[\label{E:k46sym} k\begin{pmatrix}-x_1\\x_2\\-x_3\end{pmatrix} =k\begin{pmatrix}x_1\\x_2\\x_3\end{pmatrix}.\tag{83}\] Note that \(\vartheta_{\lambda,\mu,\nu}(t)\) is a Schwartz function in \(\nu\), for fixed \((\lambda,\mu)\neq(0,0)\) and \(t>0\).
By the Euler–Maclaurin summation formula, for \(\nu_0\in\mathbb{R}\) and \(0\neq d\in\mathbb{R}\), \[\begin{gather} \sum_{n\in\mathbb{Z}}\vartheta_{\lambda,\mu,\nu_0+2dn}(t) =\int_\mathbb{R}\vartheta_{\lambda,\mu,\nu_0+2d\nu}(t)d\nu \\+\frac{(-1)^{N+1}}{N!}\int_\mathbb{R}\left(\tfrac{\partial^N}{\partial\nu^N}\vartheta_{\lambda,\mu,\nu_0+2d\nu}(t)\right)P_N(\nu)d\nu \end{gather}\] where \(P_N(x)=B_N(x-\lfloor x\rfloor)\) denotes the periodic extension of the \(N\)-th Bernoulli polynomial restricted to the unit interval. Substituting \(\nu_0+2d\nu\leftrightarrow\nu\) in the integrals, this yields \[\begin{gather} \sum_{n\in\mathbb{Z}}\vartheta_{\lambda,\mu,\nu_0+2dn}(t) =\frac{1}{2d}\int_\mathbb{R}\vartheta_{\lambda,\mu,\nu}(t)d\nu \\+\frac{(-1)^{N+1}(2d)^{N-1}}{N!}\int_\mathbb{R}\left(\tfrac{\partial^N}{\partial\nu^N}\vartheta_{\lambda,\mu,\nu}(t)\right)P_N\left(\frac{\nu-\nu_0}{2d}\right)d\nu. \end{gather}\] Plugging in 82 , we obtain \[\begin{gather} \sum_{n\in\mathbb{Z}}\vartheta_{\lambda,\mu,\nu_0+2dn}(t) =\frac{t^{-3/4\kappa}}{2d\sqrt[4]{\lambda^2+\mu^2}}\int_{\mathbb{R}^2}k\begin{pmatrix}\frac{t^{2/4\kappa}\nu}{2\sqrt{\lambda^2+\mu^2}}+\frac{x^2}{2}\\t^{3/4\kappa}\sqrt[4]{\lambda^2+\mu^2}x\\t^{6/4\kappa}\sqrt{\lambda^2+\mu^2}\end{pmatrix}dxd\nu \\+\frac{(-1)^{N+1}d^{N-1}t^{-3/4\kappa}t^{N/2\kappa}}{2N!\sqrt[4]{\lambda^2+\mu^2}(\lambda^2+\mu^2)^{N/2}} \int_{\mathbb{R}^2}\frac{\partial^Nk}{\partial x_1^N}\begin{pmatrix}\frac{t^{2/4\kappa}\nu}{2\sqrt{\lambda^2+\mu^2}}+\frac{x^2}{2}\\t^{3/4\kappa}\sqrt[4]{\lambda^2+\mu^2}x\\t^{6/4\kappa}\sqrt{\lambda^2+\mu^2}\end{pmatrix}P_N\left(\frac{\nu-\nu_0}{2d}\right)dxd\nu. \end{gather}\] Substituting \(\Bigl(\frac{t^{2/4\kappa}\nu}{2\sqrt{\lambda^2+\mu^2}}+\frac{x^2}{2},t^{3/4\kappa}\sqrt[4]{\lambda^2+\mu^2}x\Bigr)\leftrightarrow(\nu,x)\) in the integrals, this yields \[\begin{gather} \label{E:qwerty} \sum_{n\in\mathbb{Z}}\vartheta_{\lambda,\mu,\nu_0+2dn}(t) =\frac{t^{-4/2\kappa}}{d}\int_{\mathbb{R}^2}k\begin{pmatrix}\nu\\ x\\t^{3/2\kappa}\sqrt{\lambda^2+\mu^2}\end{pmatrix}dxd\nu \\+\frac{(-1)^{N+1}d^{N-1}t^{(N-4)/2\kappa}}{N!(\lambda^2+\mu^2)^{N/2}} \int_{\mathbb{R}^2}\frac{\partial^Nk}{\partial x_1^N}\begin{pmatrix}\nu\\x\\t^{3/2\kappa}\sqrt{\lambda^2+\mu^2}\end{pmatrix} \\\cdot P_N\left(\frac{2t^{-1/2\kappa}\sqrt{\lambda^2+\mu^2}\nu-t^{-2/\kappa}x^2-\nu_0}{2d}\right)dxd\nu. \end{gather}\tag{84}\] From 48 and 81 we have \[\zeta_{\rho_{\lambda,\mu,\nu}}(s) =\frac{1}{\Gamma(s)}\int_0^\infty t^{s-1}\vartheta_{\lambda,\mu,\nu}(t)dt.\] Plugging in 84 and substituting \(t^{3/2\kappa}\sqrt{\lambda^2+\mu^2}\leftrightarrow t^{3/2\kappa}\) in the integrals, we get \[\label{E:zeta46k} \sum_{n\in\mathbb{Z}}\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}(s) =\frac{(\lambda^2+\mu^2)^{(2-\kappa s)/3}}{d}\cdot f(s)+R_{\lambda,\mu,\nu_0,d,N}(s)\tag{85}\] where \[\label{E:f} f(s):=\frac{1}{\Gamma(s)}\int_0^\infty t^{s-1}t^{-4/2\kappa}\int_{\mathbb{R}^2}k\begin{pmatrix}\nu\\ x\\t^{3/2\kappa}\end{pmatrix}dxd\nu dt\tag{86}\] and \[\begin{gather} \label{E:R} R_{\lambda,\mu,\nu_0,d,N}(s) :=\frac{(-1)^{N+1}d^{N-1}(\lambda^2+\mu^2)^{(2-2N-\kappa s)/3}}{N!\Gamma(s)} \int_0^\infty t^{s-1}t^{(N-4)/2\kappa} \\\cdot\int_{\mathbb{R}^2}\frac{\partial^Nk}{\partial x_1^N}\begin{pmatrix}\nu\\x\\t^{3/2\kappa}\end{pmatrix} P_N\left(\frac{(\lambda^2+\mu^2)^{2/3}(2t^{-1/2\kappa}\nu-t^{-2/\kappa}x^2)-\nu_0}{2d}\right)dxd\nu dt. \end{gather}\tag{87}\] The integral in 86 converges for \(\Re s>4/2\kappa\) and extends to a meromorphic function on the entire complex plane which has only simple poles and these can only be located at \(s=(2-3j)/\kappa\), \(j\in\mathbb{N}_0\). Note here that the inner integral over \(\mathbb{R}^2\) in 86 results in a Schwartz function in the variable \(t^{3/2\kappa}\) which is even according to the symmetry in 83 . The integral in 87 converges for \(\Re s>(4-N)/2\kappa\) and the estimate \[\label{E:R46esti} R_{\lambda,\mu,\nu_0,d,N}(s)=O\left(d^{N-1}(\lambda^2+\mu^2)^{(2-2N-\kappa\Re s)/3}\right),\tag{88}\] holds uniformly for \((\lambda,\mu)\neq(0,0)\), \(\nu_0\), \(d\neq0\) and uniformly for \(s\) in compact subsets contained in \(\Re s>(4-N)/2\kappa\). Hence, \(\sum_{n\in\mathbb{Z}}\zeta_{\rho_{\lambda,\mu,\nu_0+2dn}}(s)\) converges for \(\Re s>4/2\kappa\) and 85 provides the meromorphic extension to the entire complex plane whose poles can only occur at \(s=(2-3j)/\kappa\), \(j\in\mathbb{N}_0\). Assuming \(N>4\) and using Lemma 9 we obtain from 85 and 88 \[\label{E:fR460} f(0)=R_{\lambda,\mu,\nu_0,d,N}(0)=0 \qquad\text{and}\qquad f'(0)=R_{\lambda,\mu,\nu_0,d,N}'(0)=0.\tag{89}\]
Now suppose \((0,0)\neq(\lambda,\mu)\in(\lambda_0,\mu_0)+(\Gamma'')^*\) as in 51 . In particular, \(\lambda\) and \(\mu\) are integers divisible by \(r\), and so is \(d=\gcd(\lambda,\mu)\). Also recall the integer \(w\) defined in 35 , and the real number \(\nu_0\) is given by 34 . Using 39 and 40 we get from 85 \[\label{E:zeta46k2} \sum_{n\in\mathbb{Z}}m\bigl(\tfrac dr,r,w,n\bigr)\zeta_{\rho_{\lambda,\mu,\nu_0+rn}}(s) =\frac{(\lambda^2+\mu^2)^{(2-\kappa s)/3}}{r}\cdot f(s) +\tilde{R}_{\lambda,\mu,N}(s)\tag{90}\] where the sum on the left hand side converges for \(\Re s>4/2\kappa\) and \[\tilde{R}_{\lambda,\mu,N}(s) :=\sum_{n=1}^{2d/r}m\bigl(\tfrac dr,r,w,n\bigr)R_{\lambda,\mu,\nu_0+rn,d,N}(s).\] Assuming \(N>4\) we obtain from 89 \[\label{E:tR460} \tilde{R}_{\lambda,\mu,N}(0)=0 \qquad\text{and}\qquad \tilde{R}_{\lambda,\mu,N}'(0)=0.\tag{91}\] From 88 and 40 , using the obvious estimate \(|d|\leq\sqrt{\lambda^2+\mu^2}\), we get \[\label{E:tR46esti} \tilde{R}_{\lambda,\mu,N}(s)=O\left((\lambda^2+\mu^2)^{(4-N-2\kappa\Re s)/6}\right),\tag{92}\] uniformly for \((0,0)\neq(\lambda,\mu)\in(\lambda_0,\mu_0)+(\Gamma'')^*\), and uniformly for \(s\) in compact subsets contained in \(\Re s>(4-N)/2\kappa\). Summing over all \((\lambda,\mu)\neq(0,0)\) in the shifted lattice, we obtain from 51 and 90 , \[\label{E:zeta46III46fR2} \zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s) =\tfrac1r\cdot Z^{(\Gamma'')^*}_\mathrm{Epst}\bigl(\tfrac{2\kappa s-4}3;\lambda_0,\mu_0\bigr)\cdot f(s)+\hat{R}_N(s),\tag{93}\] where the Epstein zeta function is defined in 78 , and \[\label{E:hatR46def} \hat{R}_N :=\sideset{}{'}\sum_{\substack{(\lambda,\mu)\in\\(\lambda_0,\mu_0)+(\Gamma'')^*}}\tilde{R}_{\lambda,\mu,N}(s).\tag{94}\] The sum in 94 converges uniformly on compact subsets of \(\Re s>(10-N)/2\kappa\) by the estimate in 92 . Assuming \(N>10\) we obtain from 91 and 94 , \[\label{E:hatR460} \hat{R}_N(0)=0 \qquad\text{and}\qquad \hat{R}_N'(0)=0.\tag{95}\] As \(\hat{R}_N(s)\) is an entire function which is independent of \(N\), we obtain 77 .
Using a basis of the lattice \((\Gamma'')^*\), we may write \[Z_\mathrm{Epst}^{(\Gamma'')^*}\bigl(s;\lambda_0,\mu_0\bigr) =\sideset{}{'}\sum_{(\alpha,\beta)\in(\alpha_0,\beta_0)+\mathbb{Z}^2}\bigl(\varphi(\alpha,\beta)\bigr)^{-s/2}\] where \((\alpha_0,\beta_0)\in\mathbb{R}^2\) corresponds to \((\lambda_0,\mu_0)\), and \(\varphi(\alpha,\beta)\) is a positive quadratic form with \(\sqrt{\det\varphi}=\mathop{\mathrm{Area}}((\mathbb{R}^2)^*/(\Gamma'')^*)=1/\mathop{\mathrm{Area}}(\mathbb{R}^2/\Gamma'')\). Hence, this Epstein [54] zeta function converges for \(\Re s>2\), it extends to a meromorphic function on the entire complex plane which has a single (simple) pole at \(s=2\) with residue \(2\pi/\sqrt{\det\varphi}=2\pi\cdot\mathop{\mathrm{Area}}(\mathbb{R}^2/\Gamma'')\), it vanishes at \(s\in-2\mathbb{N}\), and it satisfies \[\label{E:qwerty9999} Z^{(\Gamma'')^*}_\mathrm{Epst}(0;\lambda_0,\mu_0)= \begin{cases} -1&\text{if (\lambda_0,\mu_0)\in\Gamma'', and}\\ 0&\text{otherwise.} \end{cases}\tag{96}\] Note that \((\lambda_0,\mu_0)\in\Gamma''\) if and only if \(\chi|_{\Gamma\cap Z}=1\) by Lemma 3 and Lemma 5(II). Combining this with 77 we see that the sum in 51 converges for \(\Re s>10/2\kappa\) and that \(\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\) extends to a meromorphic function on the entire complex plane which has a simple pole at \(s=10/2\kappa\) with residue \(\frac{1}{r}\cdot\frac{3}{2\kappa}\cdot2\pi\cdot\mathop{\mathrm{Area}}(\mathbb{R}^2/\Gamma'')\cdot f(10/2\kappa)\). If \(\chi|_{\Gamma\cap Z}\neq1\), then the zeros of \(Z_\mathrm{Epst}^{(\Gamma'')^*}(\frac{2\kappa s-4}{3};\lambda_0,\mu_0)\) cancel all the poles of \(f(s)\) and \(\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\) is holomorphic for \(s\neq10/2\kappa\). If \(\chi|_{\Gamma\cap Z}=1\), then all but one pole get canceled and \(\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\) has one further pole at \(s=2/\kappa\) with residue \(-\frac{1}{r}\cdot\mathop{\mathrm{res}}_{s=2/\kappa}f(s)\). As \(Z_\mathrm{Epst}^{(\Gamma'')^*}\bigl(s;\lambda_0,\mu_0\bigr)\) is holomorphic at \(s=-4/3\) we obtain 79 by combining 93 with 89 and 95 . ◻
Remark 3. It is known that \(\zeta_{\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi}(s)\) has a single pole at \(s=10/2\kappa\), see [17] or [55]. Hence, in view of 52 the poles at \(s=1/\kappa\) and \(s=2/\kappa\) of \(\zeta_{\mathrm{I},\Gamma,\chi}(s)\) and \(\zeta_{\mathrm{I\!I},\Gamma,\chi}(s)\) must cancel. By Lemmas 7, 8, and 10 this is the case if and only if \[\label{E:cons1} \mathop{\mathrm{res}}_{s=1/\kappa}\zeta_{\rho_1}(s)=\frac{\pi}{\kappa}\cdot\zeta_{\rho_{1,0}}(1/\kappa)\tag{97}\] and \[\label{E:cons2} \mathop{\mathrm{res}}_{s=2/\kappa}f(s)=\frac{2}{\kappa}\cdot\zeta_{\rho_1}(2/\kappa).\tag{98}\] Moreover, we must have \[\label{E:cons3} \mathop{\mathrm{res}}_{s=10/2\kappa}\zeta_{\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi}(s) =\frac{3\pi}{\kappa r}\cdot\mathop{\mathrm{Area}}(\mathbb{R}^2/\Gamma'')\cdot f(10/2\kappa).\tag{99}\]
Proceeding as in [30] one readily obtains the spectra of \(\rho_{1,0}(\Delta_q)\). For \(q=0,5\) we only have the eigenvalue \((2\pi)^{2\kappa}\) with multiplicity one; for \(q=1,4\) we only have the eigenvalue \((2\pi)^{2\kappa}\) with multiplicity two; and for \(q=2,3\) we have the eigenvalue \((2\pi)^{2\kappa}\) with multiplicity one and the eigenvalue \((2\pi)^{2\kappa}2^{-\kappa/2}\) with multiplicity two. Plugging this into 48 we obtain \[\label{E:cons4} \zeta_{\rho_{1,0}}(s) =(2\pi)^{-2\kappa s}\cdot 4\left(1-2^{\kappa s/2}\right).\tag{100}\]
In [30] the spectra of \(\rho_\hbar(D_q^*D_q)\) have been determined explicitly. Using the computations in [30] we see that \(\mathop{\mathrm{tr}}\rho_\hbar(\Delta_q)^{-s}\) has a simple pole at \(s=1/\kappa\). For \(q=0,5\) the residue is \(1/4\pi\hbar\kappa\), for \(q=1,4\) the residue is \(1/2\pi\hbar\kappa\), and for \(q=2,3\) the residue is \(1/4\pi\hbar\kappa+\sqrt2/2\pi\kappa\). Using 48 this yields \[\label{E:cons5} \mathop{\mathrm{res}}_{s=1/\kappa}\zeta_{\rho_\hbar}(s)=\frac{1-\sqrt2}{\pi\kappa\hbar}\tag{101}\]
Combining 100 with 101 we do indeed get 97 . We will not attempt to verify 98 and 99 independently here.
In order to prove the theorem formulated in the introduction, note first that we may w.l.o.g.assume \(\Gamma\), \(\mathcal{D}_{\Gamma\setminus G}\), and \(g_{\Gamma\setminus G}\) to be of standard form. Indeed, by Lemma 2, we may assume that the lattice \(\Gamma\) is of the form considered in Section 2, i.e., generated by the exponentials of \(\tilde{\gamma}_i\) as in 10 . Moreover, according to [17] the torsion of the Rumin complex does not depend on the choice of \(\mathcal{D}_{\Gamma\setminus G}\) and \(g_{\Gamma\setminus G}\), as long as they are induced from a left invariant (2,3,5) distribution on \(G\) and a left invariant fiberwise graded Euclidean inner product on \(\mathfrak tG\), respectively. Hence we may also assume that \(\mathcal{D}_{\Gamma\setminus G}\) and \(g_{\Gamma\setminus G}\) are of the form considered in Section 4, i.e., \(\mathcal{D}_{\Gamma\setminus G}\) is induced by the left invariant 2-plane field \(\mathcal{D}_G\) on \(G\) spanned by \(X_1\) and \(X_2\), and \(g_{\Gamma\setminus G}\) is induced from a graded Euclidean inner product \(g\) on \(\mathfrak g\) such that \(X_1,\dotsc,X_5\) are orthonormal. Strictly speaking, [17] only covers graded Euclidean inner products induced from left invariant sub-Riemannian metrics on \(\mathcal{D}_G\), but the proof there can readily be extended to the generality required here.
As the Rumin complex computes the de Rham cohomology \(H^*(\Gamma\setminus G;F_\chi)\), one can use the Leray–Serre spectral sequence to prove the acyclicity of the Rumin complex, cf. Lemma 11 below. However, the acyclicity can also be read off the decomposition provided in Lemma 5. Indeed, the irreducible unitary representations appearing in this decomposition are all nontrivial, as \(\chi\) is assumed to be nontrivial. Since the Rumin complex is a Rockland complex, it becomes exact in every nontrivial unitary representation of \(G\). The acyclicity of the Rumin complex thus follows from the decomposition in 53 .
Combining Lemma 6 with Equations 58 , 63 , and 79 yields \[\zeta_{\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi}'(0) =\zeta_{\mathrm{I},\Gamma,\chi}'(0)+\zeta_{\mathrm{I\!I},\Gamma,\chi}'(0)+\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}'(0)=0.\] Using 7 we obtain \[\tau(\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\chi,g_{\Gamma\setminus G},h_\chi)=1.\] This completes the proof of the theorem.
To derive the corollary we need to know the Ray–Singer torsion of \(\Gamma\setminus G\).
Lemma 11. If \(\chi\colon\Gamma\to U(1)\) is a nontrivial character, then \(H^*(\Gamma\setminus G;F_\chi)=0\) and the Ray–Singer torsion is trivial, \(\tau_\textrm{RS}(\Gamma\setminus G;F_\chi)=1\).
Proof. Suppose for now that \(\chi\) does not vanish on \(\Gamma\cap[G,G]\). Recall from [47] or Lemma 1 that \(\Gamma\cap[G,G]\) is a lattice in \([G,G]\) and so is the image of \(\Gamma\) under the canonical homomorphism \(G\to G/[G,G]\cong\mathbb{R}^2\) with (abelian) fiber \([G,G]\). Hence, modding out the lattice, we obtain a fibration \(p\colon\Gamma\setminus G\to B\) with a 2-torus \(B\cong T^2\) as a base and with typical fiber a 3-torus, \(V:=\frac{[G,G]}{\Gamma\cap[G,G]}\cong T^3\). Using the Künneth theorem one readily shows that the cohomology of a torus \(T^n\cong\mathbb{R}^n/\mathbb{Z}^n\) with coefficients in any nontrivial flat complex line bundle is acyclic. As \(\chi\) is nontrivial on \(\Gamma\cap[G,G]\), we therefore have \(H^*(V;F_\chi)=0\). Hence, \(H^*(\Gamma\setminus G;F_\chi)=0\) by the Leray–Serre spectral sequence, and \(\tau_\textrm{RS}(\Gamma\setminus G;F_\chi)=1\) according to [56].
If \(\chi\) vanishes on \(\Gamma\cap[G,G]\), then there exists a flat complex line bundle \(\tilde{F}\) over \(B\) such that \(F_\chi=p^*\tilde{F}\). As \(\chi\) was assumed to be nontrivial, \(\tilde{F}\) must be nontrivial as well. Hence, \(H^*(B;\tilde{F})=0\). Let \(Z'\) denote a 1-parameter subgroup in the center \(Z\) that intersects \(\Gamma\) nontrivially. The canonical homomorphism \(G\to G/[G,G]\) factors into a sequence of homomorphisms \(G\to G/Z'\to G/Z\to G/[G,G]\). Modding out the lattice, this yields a tower of circle bundles \[\label{E:tower} \Gamma\setminus G\xrightarrow{p_3}E_2\xrightarrow{p_2}E_1\xrightarrow{p_1}B,\tag{102}\] such that \(p=p_1\circ p_2\circ p_3\). Using the corresponding Gysin sequences we obtain, successively, \(H^*(E_1;p_1^*\tilde{F})=0\), \(H^*(E_1;p_2^*p_1^*\tilde{F})=0\), and \(H^*(\Gamma\setminus G;F_\chi)=0\) for we have \(p_3^*p_2^*p_1^*\tilde{F}=p^*\tilde{F}=F_\chi\). Moreover, \(\tau_\textrm{RS}(\Gamma\setminus G;F_\chi)=1\) according to [56]. ◻
For a unitary representation \(\rho\colon\Gamma\to U(k)\) we consider the positive real number \[R(\Gamma,\rho):=\frac{\|-\|_{\mathcal{D}_{\Gamma\setminus G},g_{\Gamma\setminus G},h_\rho}^{\mathop{\mathrm{sdet}}H^*(\Gamma\setminus G;F_\rho)}}{\|-\|_\textrm{RS}^{\mathop{\mathrm{sdet}}H^*(\Gamma\setminus G;F_\rho)}}.\] If \(\rho\) is acyclic, that is, if \(H^*(\Gamma\setminus G;F_\rho)=0\), then \(R(\Gamma,\rho)=\frac{\tau_\textrm{RS}(\Gamma\setminus G;F_\rho)}{\tau(\Gamma\setminus G,\mathcal{D}_{\Gamma\setminus G},F_\rho,g_{\Gamma\setminus G},h_\rho)}\), by the very definition. Hence, combining the theorem with Lemma 11 we obtain \[\label{E:R46G46chi} R(\Gamma,\chi)=1\tag{103}\] for every nontrivial character \(\chi\colon\Gamma\to U(1)\). According to [17] the quantity \(R(\Gamma,\chi)\) depends continuously on \(\chi\). Hence, 103 remains true for all unitary characters \(\chi\).
If \(\rho\colon\Gamma\to U(k)\) is irreducible then there exists a sublattice \(\Gamma'\) in \(\Gamma\) and a unitary character \(\chi'\colon\Gamma'\to U(1)\) such that \(\rho\) is isomorphic to the representation of \(\Gamma\) induced by \(\chi'\), see [57]. Hence, \(R(\Gamma,\rho)=R(\Gamma',\chi')\) by [17] and 103 yields \[\label{E:R46G46rho} R(\Gamma,\rho)=1\tag{104}\] for irreducible \(\rho\). Clearly, \(R(\Gamma,\rho_1\oplus\rho_2)=R(\Gamma,\rho_1)\cdot R(\Gamma,\rho_2)\) for any two unitary representations \(\rho_1\) and \(\rho_2\) of \(\Gamma\), see [17]. Hence, 104 remains true for all finite dimensional unitary representation \(\rho\) of \(\Gamma\). This completes the proof of the corollary.
This research was funded in whole or in part by the Austrian Science Fund (FWF) Grant DOI 10.55776/P31663.
Below we will see that this function in fact extends to a meromorphic function on the entire complex plane whose poles can only be located at \(s=(2-3j)/\kappa\), \(j\in\mathbb{N}_0\), cf. 85 .↩︎
If \(f(10/2\kappa)\) vanishes then \(\zeta_{\mathrm{I\!I\!I},\Gamma,\chi}(s)\) is regular at \(s=10/2\kappa\).↩︎
We do not rule out that this residue vanishes either.↩︎