1 Introduction↩︎

1.1 Multivariate Time Series Modelling↩︎

Neural controlled differential equations (NCDEs) offer a number of advantages for modelling real-world multivariate time series. These include being robust to irregular sampling rates and decoupling the number of forward passes through their neural network from the number of observations in the time series. However, there exists a gap in performance between NCDEs and current state-of-the-art approaches for time series modelling, such as S5, the linear recurrent unit (LRU), and MAMBA [1]–[3].

This paper demonstrates that, on a range of multivariate time series benchmarks, NCDEs can outperform current state-of-the-art approaches by utilising a tool from the study of rough paths, the Log-ODE method. We refer to this new approach as Log-NCDEs¹.

1.2 Neural Controlled Differential Equations↩︎

Let \(\{(t_i,x_i)\}_{i=0}^n\) denote a set of observations from a multivariate time series, where \(x_i\in\mathbb{R}^{v-1}\). Let \(X:[t_0,t_n]\rightarrow \mathbb{R}^{v}\) be a continuous interpolation, such that \(X_{t_i}=(t_i, x_i)\), where subscript on time-dependent variables denotes evaluation. Let \(h_t\in\mathbb{R}^u\) and \(z_t\in\mathbb{R}^w\) be the NCDE’s hidden state and output at time \(t\) respectively. Let \(\xi_{\phi}:\mathbb{R}^{v}\rightarrow\mathbb{R}^u\) and \(f_{\theta}:\mathbb{R}^u\rightarrow\mathbb{R}^{u \times v}\) be neural networks, and \(l_{\psi}:\mathbb{R}^u\rightarrow\mathbb{R}^{w}\) be a linear map, where \(\phi\), \(\theta\), and \(\psi\) represent the learnable parameters. A NCDE is defined by \[\begin{align} \label{eq:ncde} h_{t_0} &= \xi_{\phi}(t_0, x_0), \\ h_t &= h_{t_0} + \int_{t_0}^t f_{\theta}(h_s)\text{d} X_s, \\ z_t&=l_{\psi}(h_t), \end{align}\tag{1}\] for \(t\in[t_0,t_n]\), where \(f_{\theta}(h_s)\text{d} X_s\) is matrix-vector multiplication [4]. Details on the regularity required for existence and uniqueness of the solution to 1 can be found in Appendix 8.1. By an extension of the Picard-Lindelöf Theorem, a sufficient condition is \(X\) being of bounded variation and \(f_{\theta}\) being Lipschitz continuous [5].

NCDEs are an attractive option for modelling multivariate time series. They are universal approximators of continuous real-valued functions on time series data [6]. Additionally, since they interact with time series data through \(X\), NCDEs are unaware of when the time series was observed. This makes them robust to irregular sampling rates. Furthermore, the number of forward passes through \(f_{\theta}\) when evaluating 1 is controlled by the differential equation solver. This is opposed to recurrent models, where it is controlled by the number of observations \(n\). By decoupling the number of forward passes through their neural network from the number of observations in the time series, NCDEs can mitigate exploding or vanishing gradients on highly-sampled time series.

To make use of the techniques developed for training neural ordinary differential equations (ODEs) [7], NCDEs are typically rewritten as an ODE, \[\label{eq:ncde95ode} \tilde{h}_t = \tilde{h}_{t_0} + \int_{t_0}^t g_{\theta, X}(\tilde{h}_s)\text{d}s,\tag{2}\] where \(\tilde{h}_{t_0}=h_{t_0}\). [4] proposed taking \(X\) to be a differentiable interpolation and \[\label{eq:cde95diff} g_{\theta, X}(\tilde{h}_s) = f_{\theta}(\tilde{h}_s)\frac{\text{d} X}{\text{d} s}.\tag{3}\]

1.3 Neural Rough Differential Equations↩︎

Neural rough differential equations (NRDEs) are based on the Log-ODE method, which is discussed briefly here, and in full detail in Section 2.

Given a set of intervals \(\{[r_i, r_{i+1}]\}_{i=0}^{m-1}\) satisfying \(t_0=r_0<\ldots< r_m=t_n\), the Log-ODE method replaces a CDE with an ODE on each interval. For a depth\(-N\) method, the ODE on \([r_i,r_{i+1}]\) is defined by \[\label{eq:logode} g_{\theta,X}(\tilde{h}_s) = \bar{f}_{\theta}\left(\tilde{h}_s\right)\frac{\log(S^{N}(X)_{[r_i,r_{i+1}]})}{r_{i+1}-r_i},\tag{4}\] where \(\bar{f}_{\theta}\) is constructed using the iterated Lie brackets of \(f_{\theta}\) and \(\log(S^{N}(X)_{[r_i,r_{i+1}]})\) is the depth-\(N\) truncated log-signature of X over \([r_i,r_{i+1}]\) [8]. Informally, \(\bar{f}_{\theta}\) is a high order description of the vector field \(f_{\theta}\) and \(\log(S^{N}(X)_{[r_i,r_{i+1}]})\) is a high order description of the control path \(X\) over \([r_i,r_{i+1}]\). Note that when using 3 , \(\tilde{h}_t\) is exactly \(h_t\) for all \(t\in[t_0,t_n]\), whereas when using 4 , \(\tilde{h}_t\) is an approximation of \(h_t\) when \(t\in\{r_i\}_{i=1}^m\).

NRDEs replace 3 with 4 , but instead of calculating \(\bar{f}_{\theta}\) using the iterated Lie brackets of \(f_{\theta}\), it is treated as a neural network \(\bar{f}_{\theta}:\mathbb{R}^u\rightarrow\mathbb{R}^{u\times \beta(v,N)}\), where \(\beta(v,N)\) is the dimension of a depth\(-N\) truncated log-signature of a \(v\) dimensional path. By neglecting the structure of \(\bar{f}_{\theta}\), NRDEs are able to reduce the computational burden of evaluating the vector field, at the cost of increasing the output dimension of the neural network.

Compared to NCDEs, NRDEs can reduce the number of forward passes through the network while evaluating the model, as the vector field is autonomous on each interval \([r_i,r_{i+1}]\). This has been shown to lead to improved classification accuracy, alongside reduced time and memory-usage, on time series with up to 17,000 observations [9]. Furthermore, as it is no longer necessary to apply a differentiable interpolation to the time series data, NRDEs are applicable to a wider range of input signals.

1.4 Contributions↩︎

This paper introduces Log-NCDEs, which build on NRDEs by constructing \(\bar{f}_{\theta}\) using the iterated Lie brackets of a NCDE’s vector field, \(f_{\theta}\). For depth’s \(N\geq2\), this change drastically reduces the output dimension of the model’s neural network, without impacting the model’s expressivity. Furthermore, it improves model performance on every dataset considered in this paper. Calculating the Lie brackets requires that \(f_{\theta}\) satisfies a regularity constraint, specifically being \(\mathrm{Lip}(\gamma)\) for \(\gamma\in(N-1,N]\). Section 3.2 presents a novel theoretical result bounding the \(\mathrm{Lip}(\gamma)-\)norm for a class of fully connected neural networks when \(1<\gamma\leq 2\). Following this, Section 3.3 details how to efficiently calculate the Lie brackets of a \(\mathrm{Lip}(\gamma)\) neural network using standard machine learning tools. The paper concludes by showing that, over a range of multivariate time series benchmarks, Log-NCDEs outperform NCDEs, NRDEs, S5, LRU, and MAMBA.

2 Mathematical Background↩︎

The following section provides a thorough mathematical description of the Log-ODE method. It will introduce \(\mathrm{Lip}(\gamma)\) regularity, the Lie bracket of two vector fields, and the signature and log-signature of a path, alongside their respective spaces, the tensor algebra and the free Lie algebra.

2.1 The Tensor Algebra↩︎

As an example, let \(V=\mathbb{R}^2\) and \(W=\mathbb{R}^3\). In this case, the tensor product is the outer product of the two vectors, and the resulting tensor product space is the space of \(2\times 3\) matrices, \(\mathbb{R}^2\otimes\mathbb{R}^3 = \mathbb{R}^{2\times 3}\). The tensor product of \(v\in \mathbb{R}^2\) and \(w\in \mathbb{R}^3\) is defined by \[v\otimes w = \begin{bmatrix} a \\ b \end{bmatrix} \otimes \begin{bmatrix} c \\ d \\ e \end{bmatrix} = \begin{bmatrix} ac & ad & ae \\ bc & bd & be \end{bmatrix},\] where any bilinear function \(\kappa(v,w)\) can be written as a linear function \(\tau(v\otimes w)\).

Details on the choice of norm for \(V^{\otimes i}\) when \(V\) is a complete normed vector space, i.e. a Banach space, can be found in Appendix 8.2.

2.2 \(\mathrm{Lip}(\gamma)\) Functions↩︎

Let \(V\) and \(W\) be Banach spaces and \(\mathbf{L}(V, W)\) denote the space of linear mappings from \(V\) to \(W\) equipped with the operator norm.

If \(f=(f^0, f^1, \ldots, f^k)\in\text{Lip}(\gamma)\), then the \(\text{Lip}(\gamma)-\)norm, denoted \(||f||_{\text{Lip}(\gamma)}\), is the smallest \(M\) for which 5 and 6 hold. When \(0<\gamma \leq 1\), then \(k=0\) and \(f^0\in\text{Lip}(\gamma)\) implies \(f^0\) is bounded and \(\gamma-\)Hölder continuous. When \(\gamma=1\), then \(f^0\) is bounded and Lipschitz. In this paper, we are concerned with the regularity of vector fields on \(\mathbb{R}^u\). In this case, \(f\in\mathrm{Lip}(\gamma)\) for \(\gamma \in (k, k+1]\) implies that \(f\) is bounded, has \(k\) bounded derivatives, and the \(k^{\text{th}}\) derivative satisfies \[||D^kf(y)-D^kf(x)|| \leq M|x-y|^{\gamma-k},\] for all \(x,y\in\mathbb{R}^u\).

2.3 The Free Lie Algebra↩︎

Any associative algebra, \((V,\times)\), has a Lie bracket structure with Lie bracket defined by \[[x, y] = x\times y - y\times x,\] for all \(x,y \in V\). For example, \(V=\mathbb{R}^{n\times n}\) with the matrix product. For another example, consider the vector space of infinitely differentiable functions from \(\mathbb{R}^u\) to \(\mathbb{R}^u\) with pointwise addition, denoted \(C^{\infty}(\mathbb{R}^u,\mathbb{R}^u)\). This space is a Lie algebra when equipped with the Lie bracket \[[a, b](x) = J_b(x)a(x) - J_a(x)b(x),\] for \(a,b\in C^{\infty}(\mathbb{R}^u,\mathbb{R}^u)\) and \(x\in\mathbb{R}^u\), where \(J_a(x)\in\mathbb{R}^{u\times u}\) is the Jacobian of \(a\) with entries given by \(J_{a}^{ij}(x)=\partial_ja^i(x)\) for \(i,j\in\{1,\ldots,u\}\) [13].

The free Lie algebra generated by \(V\) is the space \[\mathfrak{L}((V)) = \{(l_0,l_1,\ldots):l_i\in L_i\},\] where \(L_0=0\), \(L_1=V\), and \(L_{i+1}\) is the span of \([v,l]\) for \(v\in V\) and \(l\in L_i\).

2.4 The Log-Signature↩︎

Let \(X:[t_0,t_n]\rightarrow V\) have bounded variation and define \[X^n_{[t_0,t_n]} = \underbrace{\int\cdots\int}_{\substack{u_1<\cdots<u_n \\ u_i\in [t_0,t_n]}}\text{d}X_{u_1}\otimes \cdots \otimes \text{d}X_{u_n} \in V^{\otimes n}.\] The signature of the path \(X\) is \[S(X)_{[t_0,t_n]}=(1,X^1_{[t_0,t_n]},\ldots,X^n_{[t_0,t_n]},\ldots) \in \tilde{T}((V)),\] where \(\tilde{T}((V))=\{x\in T((V)) | x=(1,\ldots)\}\) [5]. The depth-\(N\) truncated signature is defined as \(S^N(X)_{[t_0,t_n]}=(1,X^1_{[t_0,t_n]},\ldots,X^N_{[t_0,t_n]}) \in \tilde{T}^N(V)\). The signature is an infinite dimensional vector which describes the path \(X\) over the interval \([t_0,t_n]\). In fact, assuming \(X\) contains time as a channel, linear maps on \(S(X)_{[t_0,t_n]}\) are universal approximators for continuous, real-valued functions of \(X\) [14], [15]. This property of the signature relies on the shuffle-product identity, which states that polynomials of the elements in a truncated signature can be rewritten as linear maps on the signature truncated at greater depth [16], [17]. A consequence of the shuffle-product identity is that not every term in the signature provides new information about the path \(X\). The transformation which removes the information redundancy is the logarithm.

It was shown by [18] that \[\log(S(X)_{[t_0,t_n]}) \in \mathfrak{L}((V)).\] This result plays a crucial role in the Log-ODE method.

2.5 The Log-ODE Method↩︎

The vector field of a CDE is typically thought of as a matrix-valued function \(f:\mathbb{R}^u\rightarrow\mathbb{R}^{u\times v}\). An equivalent formulation is \(f\) being a linear map acting on \(\text{d}X\in\mathbb{R}^{v}\) and returning a vector field on \(\mathbb{R}^u\). In other words, \(f(\cdot)\text{d}X:\mathbb{R}^u\rightarrow\mathbb{R}^u\). This formulation will prove more useful in the following section.

Let \(X:[t_0,t_n]\rightarrow\mathbb{R}^v\) be a continuous path. The (truncated) log-signature is a map which takes \(X\) to the (truncated) free Lie algebra \(\mathfrak{L}^N(\mathbb{R}^v)\). If \(f(\cdot)\text{d}X\) is restricted to smooth vector fields on \(\mathbb{R}^u\), then \(f(\cdot)\text{d}X\) is a linear map from \(\mathbb{R}^v\) to the Lie algebra \(C^{\infty}(\mathbb{R}^u, \mathbb{R}^u)\). By definition 6, there exists a unique linear map \(\bar{f}\) from \(\mathfrak{L}^N(\mathbb{R}^v)\) to the smooth vector fields on \(\mathbb{R}^u\). Figure 1 is a schematic diagram of this relationship. Since \(\bar{f}\) is a Lie algebra homomorphism, it can be defined recursively by \[\label{eq:logoderecurs1} \bar{f}(\cdot)a = f(\cdot)a, \;\; a\in \mathbb{R}^v\tag{7}\] and \[\label{eq:logoderecurs2} \bar{f}(\cdot)[a,b] = [\bar{f}(\cdot)a,\bar{f}(\cdot)b]\tag{8}\] for \([a,b]\in\mathfrak{L}^N(\mathbb{R}^v)\).

Over an interval, the Log-ODE method approximates a CDE using an autonomous ODE constructed by applying the linear map \(\bar{f}\) to the truncated log-signature of the control, as seen in 9. There exist theoretical results bounding the error in the Log-ODE method’s approximation, including when the control and solution paths live in infinite dimensional Banach spaces [8]. However, for a given set of intervals, the series of vector fields \(\{g_{X}(\cdot)\}_{N=1}^{\infty}\) is not guaranteed to converge. In practice, \(N\) is typically chosen as the smallest \(N\) such that a reasonably sized set of intervals \(\{r_i\}_{i=0}^m\) gives an approximation error of the desired level. A recent development has been the introduction of an algorithm which adaptively updates \(N\) and \(\{r_i\}_{i=0}^m\) [19].

3 Method↩︎

3.1 Log Neural Controlled Differential Equations↩︎

Log-NCDEs use the same underlying model as NRDEs \[\label{eq:Log-NCDE} \begin{align} h_t &= h_{t_0} + \int_{t_0}^t g_{\theta, X}(h_s)\text{d}s, \\ g_{\theta, X}(h_s) &= \bar{f}_{\theta}\left(h_s\right)\frac{\log(S^{N}(X)_{[r_i,r_{i+1}]})}{r_{i+1}-r_i}, \end{align}\tag{9}\] but with two major changes. First, instead of parameterising \(\bar{f}_{\theta}\) using a neural network, it is constructed using the iterated Lie brackets of a NCDE’s neural network, \(f_{\theta}\), via 7 and 8 . Second, \(f_{\theta}\) is ensured to be a \(\mathrm{Lip}(\gamma)\) function for \(\gamma\in(N-1,N]\). Figure 2 is a schematic diagram of a Log-NCDE.

These changes have a major benefit. For \(N>1\), Log-NCDEs are exploring a drastically smaller output space during training than NRDEs, while maintaining the same expressivity, as NCDEs are universal approximators. This is because the output dimension of \(f_{\theta}\) is \(u \times v\), whereas the output dimension of \(\bar{f}_{\theta}\) is \(u\times \beta(v,N)\). Figure 3 compares these values for paths of dimension \(v\) from \(1\) to \(15\) and truncation depths of \(N=1\) and \(N=2\). This benefit comes at the cost of needing to calculate the iterated Lie brackets when evaluating Log-NCDEs, which will be quantified in Section 3.4 and explored empirically in Section 5.2.

When \(N=1\), 9 simplifies to \[\label{eq:Log-NCDE95N1} g_{\theta, X}(h_s) = f_{\theta}\left(h_s\right)\frac{X_{r_{i+1}}-X_{r_i}}{r_{i+1}-r_i}.\tag{10}\] Hence, in this case the only difference between Log-NCDEs and NRDEs is the regularisation of \(f_{\theta}\). Furthermore, 10 and 3 are equivalent when \(X\) is a linear interpolation. Therefore, the approach of NCDEs, NRDEs, and Log-NCDEs coincide when using a depth\(-1\) Log-ODE approximation [9].

3.2 \(\mathrm{Lip}(\gamma)\) Neural Networks↩︎

The composition of two \(\mathrm{Lip}(\gamma)\) functions is \(\mathrm{Lip}(\gamma)\) [20]. Hence, a simple approach to ensuring a fully connected neural network (FCNN) is \(\mathrm{Lip}(\gamma)\) is to make each layer \(\mathrm{Lip}(\gamma)\). This can be achieved by choosing an infinitely differentiable activation function, such as \(\text{SiLU}\) [21]. However, in practice this may not ensure sufficient regularity, as demonstrated by Theorem 1.

Assuming that each layer \(\{L^i\}_{i=1}^m\) of \(f_{\theta}\) satisfies \(||L^i||_{\text{Lip}(2)}=1\), an explicit evaluation of 11 gives \[||f_{\theta}||_{\text{Lip}(2)} \leq 5^{2^{m-1}-1}.\] For a depth \(7\) FCNN, this bound is greater than the maximum value of a single precision floating point number. Hence, it may be necessary to control \(||f_{\theta}||_{\text{Lip}(2)}\) explicitly during training. This is achieved by modifying the neural network’s loss function \(L\) to \[\label{eq:weightpenalty} L \mapsto L + \lambda\left(\sum_{i=1}^m||W^i||_2 + ||\mathbf{b}^i||_2\right),\tag{12}\] where \(\lambda\) is a hyperparameter controlling the weight of the penalty. This is an example of weight regularisation, which has long been understood to improve generalisation in NNs [22], [23]. Equation 12 is specifically a variation of spectral norm regularisation [24].

3.3 Constructing the Log-ODE Vector Field↩︎

The linear map \(\bar{f}_{\theta}\) in 9 is defined recursively by 7 and 8 . Assuming \(f_{\theta}(\cdot)a\) is infinitely differentiable, then \(f_{\theta}(\cdot)a\) is an element of the Lie algebra \(C^{\infty}(\mathbb{R}^u, \mathbb{R}^u)\) and \[[f_{\theta}(\cdot)a,f_{\theta}(\cdot)b] = J_{f_{\theta}(\cdot)a}f_{\theta}(\cdot)b - J_{f_{\theta}(\cdot)b}f_{\theta}(\cdot)a,\] as discussed in Section 2.3. Let \(\{e_j\}_{j=1}^{v}\) be the usual basis of \(\mathbb{R}^{v}\). A choice of basis for \(\mathfrak{L}^N(\mathbb{R}^{v})\) is a Hall basis, denoted \(\{\hat{e}_k\}_{k=1}^{\beta(v, N)}\), which is a specific subset of up to the \((N-1)^{\text{th}}\) iterated Lie brackets of \(\{e_j\}_{j=1}^{v}\) [25]. Rewriting 9 using a Hall basis, \[\label{eq:hall} \bar{f}_{\theta}\left(h_s\right)\frac{\log(S^{N}(X)_{[r_i,r_{i+1}]})}{r_{i+1}-r_i} = \sum_{k=1}^{\beta(v, N)}\lambda_k \bar{f}_{\theta}(h_s)\hat{e}_k,\tag{13}\] where \(\lambda_k\) is the term in the scaled log-signature corresponding to the basis element \(\hat{e}_k\). Since each \(\hat{e}_k\) can be written as iterated Lie brackets of \(\{e_j\}_{j=1}^{v}\), it is possible to replace \(\bar{f}_{\theta}(\cdot)\hat{e}_k\) with the iterated Lie brackets of \(f_{\theta}(\cdot)e_i\) using 7 and 8 . Each \(f_{\theta}(\cdot)e_i:\mathbb{R}^u \rightarrow \mathbb{R}^u\) is a vector field defined by the \(i^{\text{th}}\) column of the neural network’s output. Hence, \(g_{\theta, X}\) can be evaluated at a point using iterated Jacobian-vector products (JVPs) of \(f_{\theta}\).

3.4 Computational Cost↩︎

When the signature truncation depth \(N\) is greater than \(1\), NRDEs and Log-NCDEs incur an additional computational cost for each evaluation of the vector field, which we quantify here for \(N=2\). Assume that a NCDE, NRDE, and Log-NCDE are all using an identical FCNN as their vector field, except for the dimension of the final layer in the NRDE. Let \(m\) and \(n_h\) be the depth and dimension of the FCNN’s hidden layers respectively, and \(u\) and \(v\) be the dimension of \(h_t\) and \(X\) from 1 respectively. Letting \(F_{\text{x}}\) be the number of FLOPs to evaluate model x’s vector field, \[\begin{align} F_{\text{NCDE}}&= 2un_h + 2(m-1)n_h^2 + 2uvn_h,\\ F_{\text{NRDE}}&= 2un_h + 2(m-1)n_h^2 + u(v^2-v)n_h,\\ F_{\text{Log-NCDE}}&= 3vF_{\text{NCDE}}, \end{align}\] where the number of FLOPs to calculate a JVP is \(3\) times that of evaluating the FCNN and \(v\) JVPs of \(f_{\theta}\) are needed to evaluate 13 when \(N=2\) [26]. Log-NCDEs and NRDEs have the same asymptotic computational complexity in each variable. However, each JVP is evaluated at the same point \(h_s\). This allows the computational burden of Log-NCDEs on high-dimensional time series to be reduced by constructing a batched function using Jax’s vmap [27]. The computational cost is evaluated empirically in Section 5.2.

3.5 Limitations↩︎

In this paper, we only consider Log-NCDEs which use a depth\(-1\) or depth\(-2\) Log-ODE approximation. This is due to the following two limitations. First, there are no theoretical results explicitly bounding the \(\mathrm{Lip}(\gamma)\) norm of a neural network for \(\gamma>2\). Second, the computational cost required to evaluate \(g_{\theta, X}\) grows rapidly with the depth \(N\). This can make \(N>2\) computationally infeasible, especially for high-dimensional time series. Another general limitation of NCDEs is the need to solve the differential equation recursively, preventing parallelisation. This is in contrast to non-selective structured state-space models, whose underlying model is a differential equation that can be solved parallel in time [28].

4 Experiments↩︎

4.1 Baseline Methods↩︎

Log-NCDEs are compared against six models, which represent the state-of-the-art for a range of deep learning approaches to time series modelling. Four of these models are stacked recurrent models, which use the same general architecture, but with different recurrent layers. The architecture used in this paper is based on the one introduced by [1] and the four different recurrent layers considered are LRU, S5, MAMBA, and S6, the selective state-space recurrence introduced as a component of MAMBA [1]–[3]. The other two baseline models are continuous models; a NCDE using a Hermite cubic spline with backward differences as the interpolation and a NRDE [4], [9]. Further details on all model architectures can be found in Appendices 10.1 and 10.2.

4.2 Toy Dataset↩︎

We construct a toy dataset of \(100{,}000\) time series with \(6\) channels and \(100\) regularly spaced samples each. For every time step, the change in each channel is sampled independently from the discrete probability distribution with density \[p(n) = \int_{n-0.5}^{n+0.5}\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}x^2}\text{d}x,\] where \(n\in\mathbb{Z}\). In other words, the change in a channel at each time step is a sample from a standard normal distribution rounded to the nearest integer. Figure 4 is a plot of a sample path from the toy dataset.

We consider four different binary classifications on the toy dataset. Each classification is a specific term in the signature of the path which depends on a different number of channels.

1. Was the change in the third channel, \(\int_0^1\text{d}X^3_s\), greater than zero?

2. Was the area integral of the third and sixth channels, \(\int_0^1\int_0^u\text{d}X^3_s\text{d}X^6_u\), greater than zero?

3. Was the volume integral of the third, sixth, and first channels, \(\int_0^1\int_0^v\int_0^u\text{d}X^3_s\text{d}X^6_u\text{d}X^1_v\), greater than zero?

4. Was the \(4\)D volume integral of the third, sixth, first, and fourth channels, \(\int_0^1\int_0^w\int_0^v\int_0^u\text{d}X^3_s\text{d}X^6_u\text{d}X^1_v\text{d}X^4_w\), greater than zero?

On this dataset, all models use a hidden state of dimension \(64\) and Adam with a learning rate of \(0.0003\) [29]. LRU, S5, S6, and MAMBA use six blocks. NRDE and Log-NCDE take \(r_{i+1}-r_i\) to be \(4\) observations and the signature truncation depth \(N\) to be \(2\). Full hyperparameter choices are given in Appendix 10.3.

4.3 UEA Multivariate Time Series Classification Archive↩︎

The models considered in this paper are evaluated on six datasets from the UEA multivariate time series classification archive (UEA-MTSCA)². These six datasets were chosen via the following two criteria. First, only datasets with more than \(200\) total cases were considered. Second, the six datasets with the most observations were chosen, as datasets with many observations have previously proved challenging for deep learning approaches to time series modelling. Further details on the chosen datasets can be found in Appendix 10.4. Following [9], the original train and test cases are combined and resplit into new random train, validation, and test cases using a \(70:15:15\) split.

Hyperparameters for all models are found using a grid search over the validation accuracy on a fixed random split of the data. Full details on the hyperparameter grid search are in Appendix 10.4. Having fixed their hyperparameters, models are compared on their average test set accuracy over five different random splits of the data. In order to compare models on their average GPU memory and run time, \(1000\) steps of training are run on an NVIDIA RTX 4090. The average run time is estimated by combining the time for \(1000\) training steps with the average total number of training steps from the five runs over the random data splits.

4.4 PPG-DaLiA↩︎

PPG-DaLiA is a multivariate time series regression dataset, where the aim is to predict a person’s heart rate using data collected from a wrist-worn device [30]. The dataset consists of fifteen individuals with around \(150\) minutes of recording each at a maximum sampling rate of \(128 \,\mathrm{Hz}\). There are six channels; blood volume pulse, electrodermal activity, body temperature, and three-axis acceleration. The data is split into a train, validation, and test set following a \(70:15:15\) split for each individual. After splitting the data, a sliding window of length \(49920\) and step size \(4992\) is applied.

Hyperparameters are found using the same method as for the UEA-MTSCA, but with validation mean squared error and slightly different hyperparameter choices given the high number of observations. Full details can be found in Appendix 10.4. Having fixed their hyperparameters, models are compared on their average mean squared error over five different runs on the same fixed data split.

5 Results↩︎

5.1 Toy Dataset↩︎

Figure 5 compares the performance of the models considered in this paper on the four different toy dataset classifications. As expected, given that the classifications considered are solutions to CDEs, NCDE’s are the best performing model. Since NRDEs and Log-NCDEs are fixed to \(r_{i+1}-r_i\) being \(4\) observations and \(N=2\), they are both approximations of a CDE. Notably, Log-NCDEs consistently outperform NRDEs, providing empirical evidence that NRDEs do not always accurately learn the Lie bracket structure of \(\bar{f}_{\theta}\). All of the stacked recurrent models perform well when the label depends on one or two channels. However, their performance begins to decrease for three channels, and only MAMBA performs well when the label depends on four channels.

5.2 UEA-MTSCA↩︎

Table 1 reports the average and standard deviation of each model’s test set accuracy over five data splits. MAMBA achieves the lowest average accuracy. NCDEs and NRDEs have similar accuracies except on EigenWorms, the dataset with the most observations, where NRDEs significantly improve over NCDEs. However, NRDEs are still outperformed on average accuracy by three stacked recurrent models: LRU, S5, and S6. Log-NCDEs are the best performing model on average accuracy and rank. Compared to NRDEs, they achieve an equal or higher average accuracy on all six datasets and a lower standard deviation on four datasets. These results highlight the improvement in performance which can be achieved by calculating the Lie brackets.

Table 2 details the average GPU memory and run time for each model, with the results for individual datasets given in Appendix 10.5. The major contributors to NCDEs high average run time are time series with many observations. Using a depth\(-2\) Log-ODE method decreases the computational burden on datasets with many observations for both NRDEs and Log-NCDEs. Although NRDEs and Log-NCDEs have the same asymptotic computational complexity, using a batched function to calculate the Lie brackets leads to Log-NCDEs having a lower computational burden on high-dimensional time series than NRDEs, as discussed in Section 3.4. Hence, Log-NCDEs have a lower average run time than both NCDEs and NRDEs. Despite these improvements, all four stacked recurrent models have lower average run times than Log-NCDEs.

5.3 PPG-DaLiA↩︎

Table 3 contains the average and standard deviation of each model’s test set mean squared error on the PPG-DaLiA dataset. In contrast to the UEA-MTSCA experiments, MAMBA is the best performing stacked recurrent model. However, Log-NCDEs still achieve the best performance, obtaining the lowest average test set mean squared error and the second lowest standard deviation.

6 Discussion↩︎

Recent theoretical work on the expressive power of structured state space models may provide an explanation for their performance on the toy dataset. It has been shown that the recurrent layer of non-selective state space models, such as S5, are unable to capture terms in the signature that depend on more than one channel. Instead, the computational burden is placed on the non-linear mixing in-between recurrent blocks. Furthermore, it has been shown that stacked selective state-space models, such as MAMBA, can capture higher order terms in the signature with only linear mixing layers [31]. Although the toy dataset highlights a potential limitation of the stacked recurrent models, this did not translate into poor performance on the real-world datasets considered in this paper.

Log-NCDEs achieve the highest average accuracy on the UEA-MTSCA datasets and the lowest mean squared error on the PPG-DaLiA dataset. In particular, they improve upon NRDEs on every dataset. These results highlight the effectiveness of the Log-ODE method for improving the performance of NCDEs and the importance of calculating the Lie brackets when applying the Log-ODE method. Furthermore, despite increasing the computational cost of each vector field evaluation, Log-NCDEs have a lower average run time than both NCDEs and NRDEs on the UEA-MTSCA datasets. Empirical evidence suggests this is due to the Log-ODE method improving efficiency on time series with many observations, and calculating the Lie brackets lowering the computational burden of the Log-ODE method on high-dimensional time series. In addition to achieving state-of-the-art performance on the regularly sampled datasets considered in this paper, Log-NCDEs maintain the ability of NCDEs to naturally handle irregular sampling, making them an attractive option for real-world applications.

7 Conclusion↩︎

Building on NRDEs, this paper introduced Log-NCDEs, which utilise the Log-ODE method to train NCDEs in an effective and efficient manner. This required proving a novel theoretical result bounding the \(\mathrm{Lip}(\gamma)\) norm of fully connected neural networks for \(1<\gamma\leq 2\), as well as developing an efficient method for calculating the iterated Lie brackets of a neural network. A thorough empirical evaluation demonstrated the benefits of calculating the Lie brackets when applying the Log-ODE method. Furthermore, it showed that Log-NCDEs can achieve state-of-the-art performance on a range of multivariate time series datasets.

A reasonable direction of future work is extending Log-NCDE’s to depth\(-N\) Log-ODE methods for \(N>2\). This would require proving an equivalent result to Theorem 1 for \(\gamma>2\). Furthermore, it would be necessary to address the computational cost of the iterated Lie brackets. This could be achieved by using a structured neural network with cheap Jacobian-vector products as the CDE vector field. Another avenue of future work could be incorporating the recently developed adaptive version of the Log-ODE method [19].

Impact Statement↩︎

This paper presents Log Neural Controlled Differential Equations, a novel approach aimed at advancing the field of time series modelling. Potential applications of the method include healthcare, finance, and biology, where accurate time series modeling plays a crucial role. Despite the clear potential for positive impact, care must be taken to further understand the capabilities and limitations of the model before real-world deployment. Additionally, structured state-space models, an alternative approach to time series modelling, have recently been integrated into large language models (LLMs). The advancement of LLMs has many potential societal consequences, both positive and negative.

Acknowledgements↩︎

Benjamin Walker was funded by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA). Andrew McLeod was funded in part by the EPSRC [grant number EP/S026347/1] and in part by The Alan Turing Institute under the EPSRC grant EP/N510129/1. Terry Lyons was funded in part by the EPSRC [grant number EP/S026347/1], in part by The Alan Turing Institute under the EPSRC grant EP/N510129/1, the Data Centric Engineering Programme (under the Lloyd’s Register Foundation grant G0095), the Defence and Security Programme (funded by the UK Government) and the Office for National Statistics & The Alan Turing Institute (strategic partnership) and in part by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA). The authors would like to acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work. http://dx.doi.org/10.5281/zenodo.22558

8 Additional Mathematical Details↩︎

8.1 Existence and Uniqueness↩︎

Let \(V\) and \(W\) be Banach spaces, \(X:[0,T]\rightarrow V\) and \(y:[0,T]\rightarrow W\) be continuous paths, and \(f(\cdot)\nu\) be a linear map from \(\nu\in V\) to vector fields on \(W\). Assume that \(X\), \(Y\), and \(f\) are regular enough for the integral \[\int_{0}^tf(Y_s)\text{d}X_s\] to be defined for all \(t\in [0,T]\) in the Young sense [32]. The path \(Y\) is said to obey a controlled differential equation (CDE) if \[\label{sgxmoqnt} Y_t = Y_{0} + \int_{0}^tf(Y_s)\text{d}X_s,\tag{14}\] for \(t\in [0,T]\), where \(Y_{0}\in W\) is the initial condition and \(X\) is the control [5]. The existence and uniqueness of the solution to a CDE depends on the smoothness of the control path \(X\) and the vector field \(f\). We will measure the smoothness of a path by the smallest \(p\geq1\) for which the \(p-\)variation is finite and the smoothness of a vector field by the largest \(\gamma>0\) such that the function is \(\text{Lip}(\gamma)\) (defined in Section 2.2).

These theorems extend the classic differential equation existence and uniqueness results to controls with unbounded variation but finite \(p-\)variation for \(p<2\). A proof of these theorems is can be found in [5]. These theorems are sufficient for the differential equations considered in this paper. However, there are many settings where the control has infinite \(p-\)variation for all \(p<2\), such as Brownian motion. The theory of rough paths was developed in order to give meaning to ([eq:cde]) when the control’s \(p-\)variation is finite only for \(p\geq2\) [34]. An introduction to rough path theory can be found in [5].

8.2 The Tensor Algebra↩︎

Let \(V\) be a Banach space and \(V^{\otimes n}\) denote the tensor powers of \(V\), \[V^{\otimes n} = V \underbrace{\otimes \cdots \otimes}_{\text{n-1\;times}} V.\] There is choice in the norm of \(V^{\otimes n}\). In this paper, we follow the setting of [35] and [36]. It is assumed that each \(V^{\otimes n}\) is endowed with a norm such that the following conditions hold for all \(v\in V^{\otimes n}\) and \(w\in V^{\otimes m}\):

1. \(||v|| = ||v_1\otimes \cdots \otimes v_n|| = ||v_{p(1)} \otimes \cdots \otimes v_{p(n)}||\) for all all bijective functions \(p:\{1, \ldots, n\}\rightarrow \{1, \ldots, n\}\),

2. \(|| v \otimes w || \leq ||v||\;||w||\),

3. for any bounded linear functional \(f\) on \(V^{\otimes n}\) and \(g\) on \(V^{\otimes m}\), there exists a unique bounded linear functional \(f \otimes g\) on \(V^{\otimes (m+n)}\) such that \((f \otimes g)(v \otimes w) = f(v)g(w).\)

The tensor algebra’s product is associative and has unit \(\mathbf{1} = (1,0,0,\ldots)\). As \(T((V))\) is an associative algebra, it has a Lie algebra structure, with Lie bracket \[[\mathbf{x}, \mathbf{y}] = \mathbf{x} \otimes \mathbf{y} - \mathbf{y} \otimes \mathbf{x}\] for \(\mathbf{x},\mathbf{y} \in T((V))\) [12].

9 Proof of Theorem 1↩︎

The proof of Theorem 1 relies on two lemmas. The first is a bound on the \(\text{Lip}(\gamma)-\)norm of the composition of two \(\mathrm{Lip}(\gamma)\) functions. The second is a bound on the \(\mathrm{Lip}(2)-\)norm of each layer of a fully connected neural network (FCNN).

9.1 Composition of \(\text{Lip}(\gamma)\) Functions↩︎

The original statement of lemma 1 in [20] gives 15 as \[\label{eq:compboundincorrect} ||g\circ f||_{\text{Lip}(\gamma)} \leq C_{\gamma}||g||_{\text{Lip}(\gamma)}\max\left\{||f||^{\textcolor{red}{k}}_{\text{Lip}(\gamma)},1\right\}.\tag{16}\] We believe this is a small erratum, as for \(g:[0,1]\rightarrow[0,1]\) defined as \(g(x)=x\), 16 implies there exists \(C_1>0\) such that \[||g\circ f||_{\text{Lip}(1)} = ||f||_{\text{Lip}(1)} \leq C_1||g||_{\text{Lip}(1)}=C_1\] for all bounded and Lipschitz \(f:[0,1]\rightarrow[0,1]\). As a counterexample, for any \(C_1>0\), take \(f(x)=x^{n}\) with \(n>\max\{C_1, 1\}\). The following proof of lemma 1 is given in [20].

Bounding the \(\text{Lip}(\gamma)-\)norm of a neural network (NN) requires an explicit form for \(C_{\gamma}\) in 15 . This can be obtained via the explicit calculations mentioned in the proof of lemma 1. Here, we present the case \(\gamma \in (1,2]\).

Note that (18 ) is a stricter bound than (15 ), as for \(\gamma\in(k, k+1]\), \[\max \left\{|| f ||_{\text{Lip}(\gamma,\Sigma,\Omega)}^{\gamma}, 1 \right\} \leq \max\left\{||f||^{k+1}_{\text{Lip}(\gamma)},1\right\}.\] There is equality when \(||f||_{\text{Lip}(\gamma)}\leq 1\) or \(\gamma=2\).

9.2 \(\text{Lip}(2)-\)norm of a Neural Network Layer↩︎

Lemma 2 allows us to bound the \(\text{Lip}(2)-\)norm of a neural network (NN) given a bound on the \(\text{Lip}(2)-\)norm of each layer of a NN. We demonstrate this here for a simple NN.

9.3 Proof of Theorem 1↩︎

10 Experimental Details↩︎

10.1 Stacked Recurrent Models↩︎

The stacked recurrent models considered in this paper are based on the official implementation of S5 located at https://github.com/lindermanlab/S5 [1]. A recurrent block consists of a batch or layer normalisation, a recurrent layer, a GLU layer [37], dropout with rate \(0.1\), and a skip connection. A full model consists of a linear encoder, a number of stacked recurrent blocks, and a final linear layer. The four different recurrent layers considered are the linear recurrent unit, S5, S6, and MAMBA, where S6 refers to the selective state-space recurrence introduced by [3] and MAMBA refers to the combination of a gated MLP, convolution, and S6 recurrence that was also introduced by [3]. S5 and LRU use batch normalisation, whereas S6 and MAMBA use layer normalisation.

10.2 CDE Models↩︎

NCDEs, NRDEs, and Log-NCDEs use a single linear layer as \(\xi_{\phi}\). NCDEs and NRDEs use FCNNs as their vector fields configured in the same way as their original papers [4], [9]. NCDEs use \(\text{ReLU}\) activation functions for the hidden layers and a final activation function of \(\tanh\). NRDEs use the same, but move the \(\tanh\) activation function to be before the final linear layer in the FCNN. Log-NCDEs use a FCNN with SiLU activation functions for the hidden layers and a final activation function of \(\tanh\). NRDEs and Log-NCDEs take their intervals \(r_{i+1} - r_i\) to be a fixed number of observations, referred to as the Log-ODE step.

10.3 Toy Dataset Details↩︎

On the toy dataset, all models use a hidden state of dimension \(64\) and Adam with a learning rate of \(0.0003\) [29]. LRU, S5, S6, and MAMBA use \(6\) blocks and S5, S6, and MAMBA use a state dimension of \(64\). S5 uses \(2\) initialisation blocks and MAMBA uses a convolution dimension of \(4\) and an expansion factor of \(2\). NCDEs, NRDEs, and Log-NCDEs use a FCNN with width \(128\) and depth \(3\) as their vector field. Furthermore, they all use Heun as their differential equation solver with a fixed stepsize of \(0.01\). NRDEs and Log-NCDEs use a Log-ODE step of \(4\) and a signature truncation depth of \(2\). Log-NCDEs do not use any \(\mathrm{Lip}(\gamma)\) regularisation, i.e. \(\lambda=0\).

10.4 UEA-MTSCA and PPG-DaLiA Details↩︎

Table 4 provides details on the dimension, number of observations, and number of classes for the datasets chosen from the UEA-MTSCA for the experiments conducted in this paper. Care is necessary when using the EigenWorms dataset. As of June \(1^{\text{st}}\) 2024, the train and test splits obtained from https://timeseriesclassification.com/description.php?Dataset=EigenWorms contain repeated time series. The repeated data was removed for the experiments in this paper.

Tables 5 and 6 give an overview of the hyperparameters optimised over during the UEA-MTSCA and PPG-DaLiA experiments for the stacked recurrent models and the NCDE models respectively. The optimisation was performed using a grid search of the validation accuracy for the UEA-MTSCA datasets and the mean squared error for the PPG-DaLiA dataset. All models and experiments used Adam as their optimiser and a batch size of \(32\), except the stacked recurrent models on PPG-DaLiA, which used a batch size of \(4\) due to memory constraints. NCDEs, NRDEs, and Log-NCDEs use Heun as their differential equation solver with a fixed stepsize of \(1 / \max\{500, 1 + (\text{Time series length} / \text{Log-ODE step})\}\), with \(\text{Log-ODE step}=1\) for NCDEs. Additionally, Log-NCDEs scale down their initial FCNN parameters by a factor of \(1000\) to reduce the starting \(\mathrm{Lip}(2)\) norm of the vector field.

10.5 Additional Memory and run time Results↩︎

Models are compared on their average GPU memory usage and run time for the UEA-MTSCA datasets. In order to compare the models, \(1000\) steps of training were run on an NVIDIA RTX 4090 with each model using the hyperparameters obtained from the hyperparameter optimisation. The specific choices of the hyperparameters are listed in Tables 5 and 6. In addition to the time for \(1000\) steps and GPU memory usage, shown in Figures 6 (a) and 6 (b) respectively, the average number of total training steps taken to produce the results in Table 1 is recorded in Figure 6 (c). Combining the results for time for \(1000\) training steps and total number of training steps gives an approximation of the total run time on the same hardware, and these results are shown in Figure 6 (d).

Although the time per training step is lower for stacked recurrent models than NCDEs, NRDEs, or Log-NCDEs, they also require more training steps to converge. Additionally, NCDEs, NRDEs, and Log-NCDEs require less GPU memory. The largest contributors to the average run time of NCDEs are the datasets with the most observations, EigenWorms and MotorImagery. The positive impact of the Log-ODE method on computational burden is demonstrated empirically by the decrease in run time achieved by NRDEs and Log-NCDEs on EigenWorms when using a depth\(-2\) Log-ODE method. When a depth\(-1\) Log-ODE method is used, such as NRDEs on MotorImagery, the same decrease is not observed. Section 3.4 demonstrated that Log-NCDEs and NRDEs have the same asymptotic computational complexity. However, when using a depth\(-2\) Log-ODE approximation and the same stepsize, NRDEs and Log-NCDEs exhibit drastically different run times on Heartbeat, a high-dimensional dataset. This difference is partly explained by the model’s having different optimal hyperparameter choices, but even when using identical hyperparameters to the NRDE, Log-NCDE’s run time for \(1000\) steps of training is \(1673\) seconds, whereas NRDE’s run time is \(9539\) seconds. The remaining difference in run time is due to calculating the JVPs of \(f_{\theta}\) using a batched function, as discussed in Section 3.4. If instead the JVPs are calculated using a for-loop, then Log-NCDEs run time increases to \(17045\) seconds.

References↩︎