1 Introduction↩︎

The increasing use of autonomous algorithmic trading systems has transformed modern financial markets. Reinforcement Learning (RL) methods, in particular, have attracted growing attention because of their ability to learn adaptive trading policies directly from market interaction. While these techniques offer significant advantages in high-dimensional and partially observed environments, they also raise important concerns for regulators. Since RL agents optimize rewards without explicit economic understanding or regulatory awareness, they may discover trading behaviours that are profitable but potentially manipulative. As emphasized by [1] and [2] autonomous trading agents may create new forms of market abuse and challenge traditional regulatory frameworks designed around human intent and accountability. Recent research in market manipulation and market microstructure has highlighted the vulnerability of electronic markets to algorithmic strategies capable of distorting price formation and exploiting structural weaknesses of limit order book environments [3]. In modern electronic markets, even relatively simple algorithmic trading rules can artificially influence short-term prices and induce predictable reactions from other market participants. Such behaviours may undermine market integrity and generate misleading signals regarding supply and demand. Moreover, recent studies suggest that reinforcement learning agents may autonomously discover manipulative trading strategies purely through reward maximization and repeated interaction with the environment [4], [5]. Specifically in [4] they predefine honest and manipulative actions, such as spoofing and pinging and use RL to determine when those already-defined actions are optimal under different transaction costs, penalties, and liquidity conditions. The actual price-manipulation mechanism is assumed in the model rather than learned by the agent. On the other hand, in [5] a genetic algorithm discovers a pump-and-dump pattern through feedback with trend-following agents in an artificial market. This differentiates us since we explicitly model non-linear permanent market impact and show that a model-free DDPG agent can recover such strategies in a continuous-action environment.

One potential source of manipulation arises from the presence of non-linear market impact, which is the reaction of prices to trades. A large empirical literature such as [6]–[8] and [9] documents that instantaneous market impact is typically a non-linear function of volume for individual trades, typically with an odd and concave shape. While these effects are well established empirically, they may also create environments in which trading activity can influence prices in exploitable ways. In particular, non-linear impact can reintroduce the possibility of dynamic arbitrage¹, allowing round-trip trading strategies, those with zero initial and terminal inventory, to generate positive expected cash flow purely through their impact on prices. This phenomenon, commonly referred to as price manipulation, was formalized by [10] and further explored in continuous-time settings by [11] and [12]. Understanding when and how such manipulation arises remains a central question in market microstructure.

Although one can often find mathematically price manipulations for a given impact model, their identification from market data poses several challenges. First, the true Data Generating Process (DGP) of the price and trade dynamics is unknown, and models represent at most a stylized description of reality. Second, even when the DGP is known, its parameters must be estimated from a finite sample of data. The error in their estimation can severely undermine the manipulation opportunities. Thus, the mere existence of manipulations does not imply a systematically profitable strategy.

RL offers an intriguing alternative since it is learning based and model-free.Rather than postulating a model and estimating the parameters on a finite sample, it uses this sample to directly identify the policy that maximizes the reward. It is therefore natural to ask whether RL based manipulation identification outperforms in finite sample the model based approach, even under the correct DGP specification. This is the main question we answer in this paper.

The use of RL methods for optimal execution and trading, where actions correspond to trading rates, is widespread, both in academia and in industry. The Deterministic Policy Gradient framework introduced by [13] and its deep learning extension, Deep Deterministic Policy Gradient (DDPG) by [14], enable efficient learning in continuous action spaces. These methods have been successfully applied to optimal execution and portfolio management tasks (see, e.g., [15]–[17]). Early applications of RL to trading include [18], who applied Q-learning to execution problems, and more recent work such as [19] and [20], which demonstrate the effectiveness of deep RL methods in complex market environments. Extensions incorporating realistic features such as stochastic liquidity and partial observability have been studied in [21] and [22], respectively.

The use of RL in financial markets presents several criticalities, since algorithmic trading systems must operate within strict regulatory frameworks designed to prevent manipulation and ensure market integrity. In fact, as pointed out in a recent work by [23], the risk of unintentional market manipulation exists, especially when learning algorithms converge to strategies that artificially influence prices in order to increase profits. This phenomenon is closely related to the broader problem of reward misspecification in machine learning, where the objective function does not fully capture the desired behaviour of the agent. In the context of optimal execution, an RL agent trained to maximize trading profits may inadvertently learn to exploit non-linear impact effects, effectively engaging in dynamic arbitrage even when such behaviour is undesirable or unrealistic, simply following a reward maximisation logic while learning from the environment.

Motivated by these considerations, this paper examines whether an RL agent is more effective than a traditional model-based strategy, which assumes correct model specification but faces parameter uncertainty arising from finite-sample estimation, in generating positive expected cash flows through price manipulation in a market characterized by non-linear price impact. We consider a single-asset setting, where the price evolves according to an Almgren-Chriss (AC) type model [24] with non-linear permanent impact and linear temporary impact (i.e. quadratic temporary impact costs). The agent is constrained to follow a round-trip strategy, meaning that it starts and ends with zero inventory over a finite trading horizon. This constraint ensures that any profit must arise purely from trading dynamics rather than directional exposure. Leveraging on this framework, and extending [12], by considering discrete time and non-vanishing temporary impact costs, we first prove the existence of manipulative strategies from a theoretical perspective, showing that non-linear impact functions can indeed admit solutions with positive expected cash flow even when only two trading velocities are allowed. Due to the non-convexity of the optimization problem, we numerically find the general optimal strategy by using Sequential Least Squares Quadratic Programming (SLSQP), assuming full knowledge of the model parameters. The SLSQP algorithm has been used previously for the solution of optimal execution problem for the non linear transient impact model in [25]. This provides a benchmark solution against which learning-based methods can be compared.

We then consider the problem of learning to manipulate the price from a finite sample of data and we compare two approaches. The first is the model based one: we use a finite sample of simulated TWAP execution to estimate the temporary and permanent impact parameters following the approach proposed by [26]. Notice that in this approach the correct DGP is known and only the parameters are estimated. We then use them to implement the optimal manipulation strategies obtained with the SLSQP optimization on a test set of simulations. The second approach is model free and based on RL: we train a DDPG algorithm to learn directly to manipulate the price on a finite sample of the same size of the one of TWAP executions. Then, as before, we test the learned strategy on a test set. For both approaches, the critical variable is volatility since, depending on its level, it preventsan accurate estimation of impact parameters or an effective learning of the strategy, respectively.

Our first finding is that the RL agent is capable of discovering trading strategies that yield positive expected cash flow, effectively identifying price-manipulative behaviors without explicit knowledge of the model, also when the training set is relatively small and volatility is not too large. More importantly, the numerical results show that the performance of the RL-based strategy is superior to the one of the model based approach when impact parameters are noisily estimated from data. The superiority of RL is observed when volatility is not too low, because otherwise the parameter estimation of the model based approach is extremely accurate and therefore the method is equivalent to an exact optimization of a well specified DGP. It is important to stress that the central question of the paper is therefore not whether manipulation exists theoretically, but whether an RL algorithm can independently learn to exploit these market vulnerabilities, when they are present, and if it can outperform a traditional model based approach.

These results highlight both the effectiveness of RL in solving complex control problems and the potential risks associated with its unsupervised deployment in financial markets. In particular, they underscore the need for a careful design of reward functions and regulatory safeguards to prevent potentially unintended market manipulation adopted by learning algorithms. From a broader perspective, this work extends the literature on non-linear market impact and dynamic arbitrage by demonstrating how modern learning algorithms can exploit these features in practice. At the same time, it connects to the growing body of research on the safety and robustness of machine learning systems, emphasizing the importance of aligning algorithmic objectives with market stability and regulatory requirements.

The rest of the paper is organized as follows. In Section 2, we formulate the problem and demonstrate the existence of a manipulative solution involving only two trading velocities. In Section 3, we introduce the SLSQP numerical optimizer and present its manipulative solution under full information of the trading environment. We also illustrate the complexity of the problem and assess its performance under partial information. In Section 4, we introduce the DDPG algorithm. Finally, in Section 5, we compare all the methods.

2 Framework of the Problem↩︎

In this section, we provide a general framework for the problem of dynamic arbitrage. We define the market model with non-linear permanent impact in discrete time, the trading cost structure, and finally we provide a solution to the case where the trader can trade with just two velocities.

We consider the problem of a risk-neutral agent seeking an optimal trading strategy over a finite time horizon, subject to zero initial and terminal inventory. The agent is allowed to both buy and sell during the trading period, and therefore aims to construct a round-trip strategy. The trading horizon is divided into \(T\) subintervals, each assumed, for simplicity, to have length \(\Delta t=1\).

We assume that the price dynamics follow an AC model with non-linear permanent impact, namely: \[\begin{align} \tag{1} S_t & = S_{t-1}+f(v_t)\Delta t+\sigma\sqrt{\Delta t} \xi_t,~~~\xi_t\sim\mathcal{N}(0,1)\\\tag{2} \tilde{S}_{t} & = S_{t-1}+\kappa v_{t},~~~~~~\kappa>0 \end{align}\] where \(S_t\) is the market price and \(\tilde{S}_t\) is the execution price at time \(t\).

Assuming that the trader trades at each \(t \in [T]\),² we denote by \(q_t\) and \(v_t\) the inventory and the trading velocity at time \(t\) respectively. The trading velocity \(v_t\) represents the traded volume at time \(t\). Clearly, it holds that \(q_0=q_T=0\). The permanent market impact is non-linear and it is \[\label{eq:32market32impact} f(v_t) = \theta~\text{sign}(v_t)|v_t|^{\delta},~~~~~~\theta>0,\delta\in(0,1)\tag{3}\] when \(\delta=1\), the permanent impact is linear as in the standard AC framework. The coefficient \(\kappa\) is the temporary impact coefficient, hence we assume a linear temporary impact. We observe that a buy order pushes the price up, both the market and the execution price, since \(f(v_t)>0\) and \(\kappa v_t>0\). Similarly, a sell order pushes both prices down.

Now, as in other optimal execution problems, we have that: \[\sum_{t=1}^Tv_t = q_0 = 0.\] In such a problem, the final cash flow, will be: \[X_T = -\sum_{t=1}^Tv_t\tilde{S}_t\] The expected cash flow is then (see Appendix 7): \[\label{eq:32final32cash-flow95main32text} \mathbb{E}[X_T] = -\Big[\sum_{t=1}^{T-1} f(v_t)\sum_{j=t+1}^Tv_j+\kappa\sum_{t=1}^T v_t^2\Big].\tag{4}\]

As the main aim of the paper is to study the existence, feasibility and learnability of dynamic arbitrages, we briefly recall its main definition used throughout the text.

Therefore, finding an optimal solution to the trading problem defined above amounts to finding a solution \(\vec{v}_*\in\mathbb{R}^T\) such that the function \(\mathbb{E}[X_T]:=c(\vec{v}_*)\) is maximized, where by \(c(\vec{v}_*)\) we denote a function of the optimal trading trajectory \(\vec{v}_*\). To this end, we first need to make sure that there exists a solution to the problem we pose. The following proposition guarantees that the problem is not ill-posed.

The above proposition does not guarantee that there is a round-trip strategy but that the problem we want to solve is not ill-posed, since the best strategy might still be to not trade.

3 Dynamic arbitrage strategies↩︎

In this section, we investigate the existence and characteristics of dynamic arbitrage strategies of the previously introduced impact model. Due to the non-convexity of the problem, the investigation is mainly numerical. However in the next subsection we show that, by restricting to trading strategies with only two velocities, it is possible to find and characterize price manipulation strategies.

3.1 Price-manipulation with only two trading velocities↩︎

Before considering more complex cases, we start by analysing the case where the trader can only select two trading velocities. Leveraging on [12], we denote by \(\mathcal{A} \subset [-b,b]\subset\mathbb{R}\) the set of all possible trading actions across time step and by \(|\mathcal{A}|\) the cardinality of \(\mathcal{A}\). Notice that for a trading strategy \(\{v_t\}_{t\in[T]}\) there are at most \(T\) different trading velocities thus \(|\mathcal{A}|\leq T\). We observe that for a round-trip strategy it must have \(T\geq 2\) (i.e. at least two time steps). In the rest of this subsection we consider that \(|\mathcal{A}|=2\). In this case it is \[v_t = \begin{cases} \alpha,~~\text{if}~t\in S\subset[T]\\ -\beta,~~\text{if}~t\in [T]\setminus S \end{cases}\] where \(S\) is any set of indexes in \([T]\). To be a round-trip strategy, \(\alpha\) and \(\beta\) must have the same sign and it should be that: \[\label{eq:constraints95pump95dump} \sum_{t=1}^T v_t = |S|\alpha-(T-|S|)\beta=0\Leftrightarrow |S|\alpha = (T-|S|)\beta.\tag{5}\] In the following lemma, we compute the expected cash flow when one trading velocity is used up to a certain time and then replaced by a different trading velocity for the remainder of the trading horizon.

Now that we have the expected cash flow for the case when there are only two trading velocities we can compute which are the optimal ones that maximize the cash flow. The following lemma proves this result.

Notice that we do not have a closed-form analytical solution for the optimal value of \(n\), but only for the optimal value of \(\alpha\), which itself depends on \(n\). Therefore, when \(|\mathcal{A}|=2\), we first determine the optimal trading velocity \(\alpha^*\) for each fixed \(n\), and then choose the value of \(n\) that maximizes the expected cash flow. Finally, for a fixed \(n\), there exists a unique strategy that maximizes the expected cash flow when \(\alpha>0\). Throughout the remainder of the paper, we refer to this strategy as the pump-and-dump strategy (PnD).

It is important to stress that nonlinearity is the main source of manipulation. Fig. (1) shows that the long position in the optimal PnD strategy decreases as \(delta\) increases. This relationship illustrates how stronger nonlinear price impact increases the size of the position selected by the optimal manipulation strategy.

3.2 Numerical optimization via SLSQP↩︎

Assuming only two distinct trading velocities, as in subsection 3.1, clearly restricts the space of possible strategies and the revenues generated from trading. A richer set of trading volumes and strategies could emerge if \(|\mathcal{A}|\) were allowed to include more than just two velocities. However, in that case the problem is not convex any more and there are no closed-form solutions. We therefore resort to numerical methods to obtain an approximate solution to the problem. We solve the constrained optimization problem using the Sequential Least Squares Programming (SLSQP) algorithm proposed by [27]. Therefore, the resulting optimization problem reads as follows: \[\begin{align} \vec{v}_* = \text{argmax}_{\vec{v}\in\mathbb{R}^T}\mathbb{E}[X_T],~~~~~s.t~~\sum_{t=1}^Tv_t = 0, \end{align}\] The objective function \(\mathbb{E}[X_T]\) is defined in Eq.(4 ), and the parameter values used to determine it are reported on Table 1. We initialize the optimization from \(N=500\) different starting points, sampling uniformly at random from \(U[-b,b]^T\), in order to identify the best solution.

Clearly, in this first step, to use the numerical solution we assume complete knowledge of the parameters governing the evolution of the trading environment.

Fig. (2) illustrates the best numerical solution obtained via SLSQP. The top-left panel shows the optimal trading velocity together with the corresponding inventory trajectory, depicted by the red line. The bottom-left panel reports the associated expected cash flow, from which we observe the existence of a manipulative strategy since \(\mathbb{E}[X_T] > 0\). The right panel displays the histogram of \(\mathbb{E}[X_T]\) obtained from 500 different SLSQP initializations. These results highlight the complexity of the optimization problem, which stems from the non-linear form of the permanent market impact function. In Appendix 9 we shed light on the landscape of the various trading strategies found by the SLSQP starting from the \(N\) different initial points.

3.3 Effect of impact parameter estimation on profitability↩︎

The previous numerical results were based on the assumption that the parameters governing the market were fully known when optimizing with SLSQP. In practice, this corresponds to an idealized setting that cannot realistically be solved under full information. In Section 3.3.1, we study the sensitivity of SLSQP to parameter misspecification, while in Section 3.3.2, we present a method of impact parameters estimation on a finite sample of optimal execution strategies, based on [26].

3.3.1 Sensitivity of PnL to parameter misspecification↩︎

Usually, most of the times in real market scenarios, impact parameters values are not available, as by definition they are latent quantities and therefore difficult to be estimated directly. Therefore, numerical solutions obtained with estimated parameters might be affected by estimation errors as magnitudes might be misspecified. To see this, Fig. (3) illustrates the expected cash as a function of parameter misspecification across all three parameters. For the computation of the cash flow, we find the optimal strategy by using \(\hat{\delta}\) and then we substitute this strategy to Eq.(4 ) where we use the parameter \(\delta\), similarly for the other two cases when we do not have knowledge of parameters \(\theta\) and \(\kappa\). Specifically, panel (a) shows the impact of misspecification in \(\delta\), panel (b) in \(\theta\), and panel (c) in \(\kappa\).

Two key observations emerge. First, clearly the highest expected cash is achieved in the absence of misspecification, and performance deteriorates as the magnitude of the misspecification increases. Second, performance improves when the true value of \(\delta\) is closer to 1 rather than to 0. A similar pattern is observed in the remaining panels, where the highest cash is again attained under no or minimal misspecification.

3.3.2 Estimation of impact parameters↩︎

Here we present a parameter estimation method based on [26]. In [26], the authors used real metaorders³, thus here we estimate the parameters by simulating metaorders from a TWAP strategy in a time window \([1,T]\) leaving the dynamics of the market unchanged from previous numerical experiments. Specifically, we assume that for the metaorder \(i\) the fundamental and the execution price, respectively, are \[\begin{align} S_t^{(i)} & = S_{t-1}^{(i)}+\theta\text{sign}(v^{(i)})|v^{(i)}|^{\delta}+\sigma\xi_t^{(i)}\\ \tilde{S}_t^{(i)}& = S_{t-1}^{(i)}+\kappa v^{(i)},~~~~~t=1,\cdots,T~~\text{and}~~i=1,\cdots,N. \end{align}\] We assume that at the \(i\)-th metaorder we want to execute \(q_i\) shares⁴ at a constant speed for any order \(i\) and it is \(v^{(i)} = \frac{q_i}{T}\) with \(q_i \sim U[1,20]\).

For each metaorder \(i\) the fundamental price at time \(t\) can be written as: \[S_{t}^{(i)} = S_0 + t\theta(v^{(i)})^{\delta}+\sigma\sum_{s=1}^{t}\xi_{s}^{(i)}\] thus, at the end of metaorder \(i\), at time \(T\) the fundamental price it is \[\begin{align} S_T^{(i)} & = S_0+T\theta (v^{(i)})^{\delta}+\sigma\sum_{s=1}^{T}\xi_{s}^{(i)}\Leftrightarrow\\ \underbrace{\frac{S_T^{(i)}-S_0}{S_0}}_{I_i} & = \frac{T\theta(v^{(i)})^{\delta}}{S_0}+\text{noise} \end{align}\] The execution price at time \(t\) can be written as: \[\tilde{S}_t^{(i)} = S_0+(t-1)\theta (v^{(i)})^{\delta}+\sigma\sum_{s=1}^{t-1}\xi_{s}^{(i)}+\kappa v^{(i)}\] and the realised price (following the terminology in [26]) for the metaorder \(i\), due to the TWAP strategy, is \[\Bar{S}_i = \frac{\sum_{t=1}^Tv^{(i)}\tilde{S}_{t}^{(i)}}{\sum_{t=1}^Tv^{(i)}} = \frac{\sum_{t=1}^T\frac{q_i}{T}\tilde{S}_{t}^{(i)}}{\sum_{t=1}^T\frac{q_i}{T}} = \frac{\frac{1}{T}q_i\sum_{t=1}^T\tilde{S}_{t}^{(i)}}{q_i}\] \[\begin{align} \Bar{S}_i & = \frac{1}{T}\sum_{t=1}^T \tilde{S}_{t}^{(i)}\\ & = \frac{1}{T}\sum_{t=1}^T\Big[S_0+(t-1)\theta (v^{(i)})^{\delta}+\kappa v^{(i)}+\sigma\sum_{s=1}^{t-1}\xi_{s}^{(i)}\Big]\\ & = S_0+\theta \frac{T-1}{2}(v^{(i)})^{\delta}+\kappa v^{(i)}+\text{noise}\Leftrightarrow\\ \underbrace{\frac{\Bar{S}_i-S_0}{S_0}}_{J_i} & = \frac{T-1}{2T}I_i+\frac{\kappa v^{(i)}}{S_0}+\text{noise} \end{align}\] Therefore, we end up with the following two regressions⁵: \[\begin{align} I_i & = \frac{T\theta(v^{(i)})^{\delta}}{S_0}+\text{noise}\\ J_i-\frac{T-1}{2T}I_i & = \frac{\kappa v^{(i)}}{S_0}+\text{noise} \end{align}\] To find an estimate for \(\delta\) we minimize the residuals sum of squares (RSS): \[\hat{\delta} = \text{argmin}_{\delta\in(0,1)}\sum_{i=1}^N\Big(I_i-\hat{\theta}(\delta)\frac{T}{S_0}\Big(\frac{q_i}{T}\Big)^{\delta}\Big)^2\] where \(\hat{\theta}(\delta)=\frac{S_0}{T}\frac{\sum_{i=1}^N(\frac{q_i}{T})^{\delta}I_i}{\sum_{i=1}^N(\frac{q_i}{T})^{2\delta}}\) (see Appendix 10 for a derivation of this formula). Then \(\hat{\theta} = \hat{\theta}(\hat{\delta})\). Similarly, for the estimation of \(\kappa\) it is \[\hat{\kappa} = S_0\frac{\sum_{i=1}^N(\frac{q_i}{T})(J_i-\frac{T-1}{2T}I_i)}{\sum_{i=1}^N(\frac{q_i}{T})^2}.\]

3.4 Numerical experiments↩︎

We now perform numerical experiments to show how the estimation method proposed above works in finite sample. The parameter of the impact model are those in Table 1. For the choice of the volatility \(\sigma\), we adopt a realistic estimate by using real data. First we use an estimate of the daily realized volatility (RV) (see [28]) of MSFT from April 2018 and we set \(\sigma_\text{daily} = 0.026\). Similarly to [26] we assume that an average metaorder lasts \(0.39\) days, thus \[\sigma_\text{metaorder}:=\sigma = \sigma_\text{daily}\sqrt{0.39}=0.0168.\] Moreover for the impact parameters; \(\theta\) and \(\kappa\), we choose them in a way such that the ratios \(\frac{\sigma}{\theta}\) and \(\frac{\sigma}{\kappa}\) are 56 and 167 respectively. These ratios are the same with the ones reported on Table B from [26] in which they report the summary statistics from a sample of real metaorders. Finally, we consider the estimation on samples of size \(N\in\{500,5000\}\).

The results are shown in Table 2. As expected, the estimation is more precise for larger samples⁶ and for smaller volatility. The interesting aspect is that the estimation of the parameter \(\delta\) is very noisy and usually upward biased. As suggested by Fig. (1), this overestimation leads to more conservative strategies (i.e. with smaller long position), reducing the potential PnL of the dynamic arbitrage strategy when the DGP is known but the impact parameters must be estimated from a finite sample of data.

4 Dynamic Arbitrage via deep RL↩︎

Having set the stage with numerical techniques, we now pass to studying the problem with deep RL. By means of RL we aim at assessing whether a learner, who has no direct knowledge of either the impacts structure or of the values for the impacts, would be able to learn a dynamic arbitrage strategy, employing as little information on the market as possible. In a spirit similar to that of [20] and [21], we solve the round trip optimal trading problem in the context of the model reported in Eq.@eq:eq:fund32price with DDPG.

4.1 Environment, states and reward↩︎

The deep RL experiment runs over a number \(M\) of training episodes of trading where the agent learns the optimal strategy. The agent, for each time step \(t\), has access to a simple state representation of the environment called \(\mathcal{S}_t= (q_t, t, S_t)\), that is the tuple made by current inventory at time step \(t\), time step \(t\) and permanently impacted price at time step \(t\), i.e. \(S_t\).

The price dynamics evolve as in Eq.@eq:eq:fund32price and the values of the parameters are the same as the ones described on Table 1, therefore the agent has to make a trading decision \(v_t\) (i.e. buy/sell action to adopt) based on the knowledge of the state \(\mathcal{S}_t\), such that the reward at each time step is maximal. The reward, seen at each trading step is defined as: \[\label{eq:reward} r_t = S_t v_t-\kappa v_t^2\tag{6}\] Where \(v_t\) is the traded volume at time \(t\), \(S_t\) is the permanently impacted price as in Eq. 1 and \(\kappa\) is the temporary impact parameter. The reward and the actions are added to the tuple \(\mathcal{S}_t\) when they are stored in the memory used by the algorithm to train its neural networks (further details below).

4.2 The learning algorithm↩︎

The DDPG algorithm is composed by two main neural networks: a critic and an actor network. It allows for continuous controls and combines deterministic policy gradients with deep function approximation. Originally introduced by [14], DDPG extends the deterministic policy gradient framework of [13] by leveraging neural networks to approximate both the policy (actor) and the action-value function (critic). The actor network deterministically maps states to actions, while the critic estimates the expected return. To stabilize training, DDPG employs target networks and an experience replay buffer, borrowing key ideas from Deep Q-Network. The critic is trained by minimizing the Bellman error using targets constructed from the reward signal and the discounted estimate of future returns, while the actor is updated via the deterministic policy gradient, maximizing the critic’s evaluation of the chosen actions.

4.2.1 The actor and critic networks↩︎

The critic and actor neural networks are feed-forward networks with four and three input neurons, respectively, a number of \(l_{\text{NN}}\) layers of \(d_{\text{NN}}\) hidden nodes, each with SiLu activation function, in the Actor network, the final layer has tanh activation function.

We refer to the actor network as \(\pi\), since it is the network that parametrises the policy directly, and to the critic network as \(Q\), as the architecture of the critic part of the algorithm relies fully on double deep Q-learning.

At the beginning of the training loop we initialise the networks \(\pi, Q\) with random weights \(\mu_\pi, \mu_Q\). As training unfolds, \((\mu_\pi, \mu_Q) \to (\mu^*_\pi, \mu^*_Q)\), where \((\mu^*_\pi, \mu^*_Q)\) are the optimal weights that correspond to the optimal policy that maximises the the sum of discounted rewards.

Therefore, during training the \(Q\) network, in order to avoid an overestimation of the \(Q\)-values for the considered state-action pairs, we use a \(Q_{\text{tgt}}\) network that is initialised at the beginning of the training routine as a copy of the \(Q\) network with weights \(\mu_{Q_{\text{tgt}}} = \mu_{Q}\), thus employing double deep Q-learning for the Critic part of the algorithm, the weights of the \(\mu_{Q_{\text{tgt}}}\) are then periodically set equal to \(\mu_Q\). In order to do so, we have adopted the soft update technique as described in the original DDPG paper by [14], with the soft update parameter set to \(0.001\).

During training, as the trading unfolds we chose to keep memory of the states seen and the actions, volumes traded, at each time step into a memory of length \(m\). As soon as the memory has at least length \(B\), which is the length of a batch, the algorithm starts training the two networks via gradient descent. In any case, every time the networks are trained, batches of past trading experiences are randomly sampled from the memory.

4.2.1.1 The critic.

For each training iteration, we train a critic network \(Q\). To this end, we add an additive exploration zero mean noise to the output of the actor network \(\pi\) (whose training is described below), whose variance declines at each iteration⁷. The additive noise, \(\varepsilon\), helps the Actor neural network to explore the range of inventory holdings to be held depending on the state. Thus, while training the \(Q\) network, \(v_{{t+1}} = \pi(\mathcal{S}_t|\mu_\pi) + \mathcal{N}(0, \varepsilon)\), where \(\pi(\mathcal{S}_t|\mu_\pi)\) is the output of the Actor network. Clearly, as the training unfolds and \(\varepsilon\) becomes smaller, the exploration noise diminishes and the policy chosen by the network \(\pi\) becomes deterministic. In this way, given the weights \(\mu^*_\pi\), found at the end of the training, the Critic network directly maps a state into an action such that: \[v^{*}_t = \pi(\mathcal{S}_t|\mu^*_\pi)\] Once the volume is obtained, it is used to calculate the reward for each time-step using the equation Eq.@eq:eq:reward .

To train the \(Q\) network, we update the weights \(\mu_Q\) such that \(\mu_Q = \text{argmin}_{\mu_Q} \L_1(\mu_Q; \mu_{Q_{\text{tgt}}})\), by taking a single gradient step on the loss

\[\begin{align} \L_1 &= \frac{1}{b} \sum^b_{i=1} (Q(\mathcal{S}_t, v_{{t+1}}|\mu_Q) - y_t)^2\\ y_t &= r_{t} + \gamma Q_{\text{tgt}}(\mathcal{S}_t', \pi(\mathcal{S}_t'|\mu_\pi)|\mu_{Q_{\text{tgt}}}) \end{align}\]

Where \(\mathcal{S}_t'\) is the next state where the environment evolves after the agent has taken the action \(v_t\). The weights of the Critic network \(\mu_Q\) are then updated using gradient descent.

4.2.1.2 The actor.

For each training iteration we train the actor network \(\pi\), which is the neural network responsible for choosing the action \(v_t\). To do this, we feed the \(\pi\) network with a batch of \(B\) states \(\mathcal{S}_t\). The \(\pi\) network returns a volume to be sold or bought \(v_t\), which is in turn fed to the Critic \(Q\) network, along with the state \(\mathcal{S}_t\). The output of the \(Q\) network is the \(Q\) value which is maximized during training. Specifically, the weights \(\mu_\pi\) of the Actor network are found by maximizing the output of the critic network with fixed weights \(\mu_Q\). As before we perform a single gradient step to minimise the loss:

\[\L_2 = - \frac{1}{b} \sum^b_{i =1} Q(\mathcal{S}_t, \pi(\mathcal{S}_t'|\mu_\pi)|\mu_Q)\] where \(\pi(\mathcal{S}_t|\mu_\pi)\) is the output of the actor network with current weights \(\mu_\pi\). The weights \(\mu_\pi\) are updated using the policy gradient theorem. The derivative of the loss function \(\L_2\) with respect to the weights of the Actor network is: \[\nabla_{\mu_\pi}\L_2 = - \frac{1}{b}\sum^b_{i=1}\left[ \nabla_{a}Q(\mathcal{S}_t, a^{(i)}|\mu_Q)|_{a^{(i)} = \pi(\mathcal{S}_t'|\mu_\pi)} \nabla_{\mu_\pi}(\mathcal{S}_t|\mu_\pi) \right]\] where the first part in the square brackets tells us how sensitive the \(Q\) network is to changes in the action chosen by the Actor \(\pi\), this is the gradient of the \(Q\)-value with respect to the action and evaluated at the action suggested by the Actor network, in this context for ease of notation we use \(a^{(i)}=v^{(i)}_t\) to denote the action for the \(i\) state among the \(b\) considered in the batch. The second part inside the brackets is the gradient of the the actor’s policy and accounts for how much the action \(v_t\) changes with respect to changes in the \(\pi\) network parameters. We update the weights for the \(\pi\) network with gradient descent. We repeat this training routine a number \(l\) of times in order to assess how the quality of the actions chosen by the Actor changes and, in this way, to thoroughly explore weights combinations for the \(\pi\) network.

The complete training algorithm for both the Critic and the Actor neural networks is reported in Algorithm 5. Both the Actor and the Critic neural networks are feed-forward fully connected neural networks, for the gradient descent we use the Weighted ADAM with learning rate of \(0.1\). As training unfolds, we adopt a scheduler for the learning rate, thus diminishing its value as the agent makes experience and the neural networks are trained, once very \(100\) episodic iterations (i.e. time step over training episodes) the learning rate is diminished following the rule \(lr \leftarrow lr \times 0.99\). The features for the DDPG algorithm are normalised in the domain \([-1, 1]\). Relevant parameters for the networks and the market model are reported in Table 3.

4.2.1.3 Training results.

We train the agent using the procedure described above and terminate training after episode 500, since no noticeable improvement in the learned policy is observed beyond this point. The top panel of Fig. (4) shows a heatmap of the actions selected at each time step and in each episode during one run of training. At the beginning of the training phase, the agent alternates between exploratory buy and sell actions. As training progresses, however, the policy converges toward a directional buying strategy over the first 20 time steps, followed by a sell execution at the final step to satisfy the round-trip constraint. The bottom panel shows the final learned strategy.

To characterize how the strategies learned by DDPG evolve across episodes, we define the action aggressiveness in episode \(e\) as: \[\label{eq:32aggressiveness} A_e = \frac{1}{20}\sum_{t=1}^{20}|v_{e,t}|\tag{7}\] and the signed directionality in episode \(e\) as: \[\label{eq:32directionality} D_e = \frac{\sum_{t=1}^{20} v_{e,t}}{\sum_{t=1}^{20} |v_{e,t}|},~~~~~\text{with}~~-1<D_e<1\tag{8}\] where \(v_{e,t}\) is the action chosen (i.e. the volume traded) in episode \(e\) at time \(t\). The quantity \(A_e\) measures how aggressive in absolute terms is the pump phase of the manipulative strategy, while \(D_e\) tells us how consistently the strategy buys or sells in the pump phase. The left panel of Fig. (6) displays the evolution of \(A_e\) during 4 training runs. We observe that, as training progresses, DDPG converges toward less aggressive strategies. The right panel of Fig. (6) shows \(D_e\) across the same runs. During the first episodes, this quantity is close to zero, indicating no clear directional preference in the first 20 time steps. As training proceeds, however, the strategy converges toward a positive direction over the first 20 time steps. Notice that this symmetry breaking is consistent with the form of the reward function in Eq. (6 ). At each time step \(t\), DDPG seeks an action \(v_t\) that maximizes the instantaneous reward. Therefore, by differentiating the reward with respect to \(v_t\) and setting the derivative equal to zero, we obtain the action that maximizes the reward at time \(t\): \[v_t^* = \frac{S_t}{2\kappa}>0.\] If on the other hand, one defines the reward as: \[r_t = -S_tv_t-\kappa v_t^2\] then we will observe a negative direction at the final episodes (see Fig. (12) in Appendix 11).

5 Discovering price manipulation strategies in finite data samples↩︎

We now consider the problem of a trader who tries to find a price manipulation strategy in a market where dynamic arbitrage is allowed by the nonlinearity of permanent impact. Critically, we assume that the trader has a finite amount of data to identify the strategy, and we compare two approaches.

In the first one, the agent has \(N\) TWAP metaorders to estimate the parameters of the AC nonlinear model with the regression method presented above. Then the agent uses the estimated parameters in an SLSQP optimizer to find the optimal price manipulation and it tests it on 1000 out of sample simulations of manipulative strategies. We consider \(N=500\) and \(N=5000\) and we term these strategies \(\text{regSLSQP}_{500}\) and \(\text{regSLSQP}_{5000}\), respectively. In the second approach, the agent is agnostic about the underlying impact model and uses the RL approach described above to learn the manipulation strategy on \(N=500\) episodes. As before, the agent tests it on 1000 out of sample simulations of manipulative strategies. We term this strategy DDPG.

As benchmarks we use the PnL obtained (i) with the optimal strategy with only two trading velocities and (ii) with the optimal strategy obtained with SLSQP. Clearly, in both cases we use the correct DGP and the true parameters. The two strategies are termed PnD and SLSQP, respectively. As above, we consider three levels of volatility.

Table 4 summarizes, for each method, the average terminal cash \(X_T\), the corresponding standard error, and the results of the t-test together with the associated p-values over 1000 out of sample implementations of the learned strategies. For the regSLSQP methods, we follow a different approach. Since there is a large variability in the estimated impact parameters from a finite number of TWAP executions, the PnL in the out of sample set depends strongly on the specific realization of 500 or 5000 TWAPs used in the estimation. To mitigate this effect, we replicate the whole procedure 100 times and we report the mean and the average standard deviation in these 100 experiments. Moreover, the t-test column reports the percentage of the replications (among the 100) that reject the null-hypothesis of zero expected cash.

The results show that when volatility is high (\(\sigma = 0.0168\)), none of the methods among PnD, SLSQP and DDPG produces a statistically significant positive terminal cash. This is due to the finiteness of the test set and the large volatility. The regSLSQPs produce a statistically significant negative expected terminal cash in \(46\%\) and \(33\%\) of the cases when \(500\) and \(5000\) metaorders are used, respectively, for the estimation of the impact parameters. Thus in these conditions, even if manipulation is possible, large volatility prevents its exploitation.

For intermediate volatility values (\(\sigma = 0.003\)) PnD, \(\text{regSLSQP}{5000}\), and DDPG achieve statistically significant positive cash flows. PnD is the best-performing method followed by DDPG and then by \(\text{regSLSQP}{5000}\). Moreover, SLSQP even though it has complete knowledge of both the functional form of the market-impact model and of the parameters does not produce statistically significant positive cash. On the contrary, the model based approach \(\text{regSLSQP}{500}\) estimated on a smaller sample provides negative expected cash flow. Importantly, notice that the same sample size is sufficient for the RL based approach DDPG to learn a systematically profitable strategy.

Finally, when volatility is low (\(\sigma = 0.002\)), all methods except \(\text{regSLSQP}_{500}\) generate statistically significant positive terminal cash. However, among the two sample based methods, \(\text{regSLSQP}_{500}\) outperforms DDPG. This is somewhat expected because with low volatility and relatively large estimation set (\(N=5000\)), the parameters of the impact model are accurately estimated, while the DDPG is still trained on \(N=500\) episodes.

As expected, volatility is the key parameter for the effective profitability of dynamic arbitrage strategies. When the model and the parameters are known (SLSQP and PnD), large volatility prevents the exploitation of the arbitrage, at least in a finite testing sample. When the model and/or the parameters are unknown, the volatility also enters in the estimation (regSLSQP) and learning (DDPG). The surprising result is that for intermediate volatility and small sample a model agnostic method is more effective of one knowing the model but not the parameter in identifying profitable strategies. Even more surprising is that in this volatility regime a ten-fold larger training set of the regSLSQP is not sufficient to beat the performance of DDPG. Finally, even for small volatility, DDPG outperforms regSLSQP when the training tests are equally small.

In conclusion, the DDPG algorithm generates positive cash flows while relying on only minimal information: the current time step, the remaining inventory, and the permanently impacted asset price. Despite this simple state representation, the agent achieves performance comparable to numerical optimization methods, with relatively low variability in terminal cash \(X_T\). This suggests that reinforcement learning algorithms can discover round-trip trading strategies that create dynamic arbitrage opportunities.

6 Conclusion↩︎

This paper studied whether a model-free reinforcement learning algorithm can learn to exploit price-manipulation opportunities more effectively than a traditional model-based approach when both methods are trained on finite samples of market data. We considered a single-asset optimal execution problem with concave permanent market impact, a setting in which dynamic arbitrage may arise through round-trip trading strategies.

We first established, both analytically and numerically, the existence of profitable manipulative strategies in discrete time. While these strategies can be derived under full knowledge of the market-impact model, implementing them in practice requires estimating latent impact parameters from data, introducing an additional source of uncertainty. This observation motivates the central comparison of the paper: a model-based optimization procedure with correctly specified dynamics but estimated parameters versus a model-free reinforcement learning agent that learns trading policies directly from interaction with the environment.

Our numerical experiments show that the DDPG agent is capable of learning profitable round-trip strategies without explicit knowledge of the underlying market-impact model. More importantly, when both approaches are constrained by the same amount of training data, reinforcement learning frequently outperforms the model-based procedure because it avoids the intermediate and error-prone parameter-estimation step. The advantage is most pronounced in regimes where parameter uncertainty materially affects optimization, whereas the model-based approach naturally regains superiority when estimation becomes sufficiently accurate.

The broader implication of these findings is methodological as much as financial. The paper does not argue that reinforcement learning creates manipulation where none exists; rather, it shows that, whenever the market environment admits exploitable dynamic arbitrage, a model-free learning algorithm can identify such opportunities directly from data and may do so more efficiently than a correctly specified parametric model estimated on finite samples. This highlights an important distinction between the theoretical existence of manipulation and its practical discoverability.

From a market-design perspective, the results also carry regulatory implications. Reinforcement learning agents optimize the objective they are given, without distinguishing between economically desirable profits and profits generated through behaviour that may undermine market integrity. Consequently, environments that inadvertently admit manipulative incentives may be particularly vulnerable to increasingly sophisticated learning algorithms. As reinforcement learning becomes more widely adopted in automated trading systems, ensuring that reward functions, market design, and regulatory constraints are aligned with no-manipulation principles will become increasingly important.

Several directions for future research naturally follow from this work. Extending the analysis to more realistic market-impact models, including transient or stochastic impact, would help assess the robustness of the results beyond the Almgren-Chriss framework. Likewise, studying richer market environments with limit-order-book dynamics, multiple strategic agents, or cross-asset interactions would provide a more comprehensive understanding of how learning algorithms exploit market structure. Finally, integrating explicit no-manipulation constraints or safe reinforcement learning techniques into optimal execution algorithms represents a promising avenue for developing trading systems that remain both profitable and market compliant.

7 Derivations for section 2↩︎

In this appendix we first derive the expected cash flow and then we give the proof of Proposition (1).

We derive Eq.(4 ) by substituting the execution price into the expression for the terminal cash flow and expanding the result: \[\begin{align} X_T & = -\sum_{t=1}^T v_t\tilde{S}_t\nonumber\\ & = -\sum_{t=1}^T v_t(S_{t-1}+\kappa v_t)\nonumber\\ & = -\Big[\sum_{t=1}^Tv_tS_{t-1}+\sum_{t=1}^T\kappa v_t^2\Big] \end{align}\] The recursive Eq.(1 ) is written as \[S_{t-1} = S_0+\sum_{j=1}^{t-1}f(v_j)+\sigma\sum_{j=1}^{t-1}\xi_j\] hence the terminal cash-flow can be written as: \[X_T = -\sum_{t=1}^Tv_t\Big[\sum_{j=1}^{t-1}f(v_j)+\sigma\sum_{j=1}^{t-1}\xi_j\Big]-\kappa\sum_{t=1}^T v_t^2.\] We observe that \[\sum_{t=1}^T v_t\sum_{j=1}^{t-1} f(v_j) = \sum_{j=1}^{T-1}f(v_j)\sum_{t=j+1}^T v_t,~~~~~~v_0=0\] thus it is \[\label{eq:32Exp32IS} \mathbb{E}[X_T] = -\Big[\sum_{t=1}^{T-1} f(v_t)\sum_{j=t+1}^Tv_j+\kappa\sum_{t=1}^T v_t^2\Big].\tag{9}\] The following proof establishes Proposition (1).

8 Derivations for section 3.1↩︎

We first prove lemma 1

We then prove lemma 2

9 Properties of the local maxima of the optimization problem↩︎

We analyze the trading strategies obtained by SLSQP from \(N\) different initializations, focusing on how similar the resulting strategies are in terms of their final cash flow. To compare two strategies \(\vec{v}_i,\vec{v}_j\in\mathbb{R}^T\), we use the cosine similarity⁸ [30] \[c(i,j)=\frac{\langle \vec{v}_i,\vec{v}_j\rangle}{\|\vec{v}_i\|\|\vec{v}_j\|},\] which measures the angle between the two vectors and lies in \([-1,1]\). Since a strategy and its sign-reversed counterpart produce the same final cash position, we use the sign-invariant distance \[d(i,j)=1-|c(i,j)|.\] This distance treats aligned and anti-aligned strategies as equivalent. We then examine the pairwise distance matrices for all SLSQP terminal strategies. As expected (see Fig. (7)), nearby strategies in the ordering tend to have smaller distances. However, the plots alone do not reveal whether similar strategies necessarily yield similar expected cash. Therefore, for all pairs satisfying \(d(i,j)\leq \eta\) with \(\eta=0.1\), we compute \(|\mathbb{E}_i[X_T]-\mathbb{E}_j[X_T]|\) (see Fig. (8)). The resulting scatter plot shows that strategies can be close under the sign-invariant distance while still producing different expected cash, indicating structural differences despite high similarity. Finally in Fig. (9) are illustrated two strategies with small sign-invariant distance but relatively large cash discrepancy.

10 Estimating the Parameters in SLSQP↩︎

In this Appendix we justify the derivation for the estimation of the permanent impact parameter \(\theta\) as a function of the exponent impact parameter \(\delta\). Since \[\hat{\delta} = \text{argmin}_{\delta\in(0,1)}\underbrace{\sum_{i=1}^N\Big(I_i-\hat{\theta}(\delta)\frac{T}{S_0}\Big(\frac{q_i}{T}\Big)^{\delta}\Big)^2}_{\text{RSS}(\hat{\theta})}\] for the simplification in the notation we use: \(\hat{\theta}(\delta):=\hat{\theta}\). The RSS is a concave function in \(\hat{\theta}\) thus there exist unique minimizer, it is then enough to consider the first derivative: \[\begin{align} \frac{\partial \text{RSS}}{\partial\hat{\theta}} & = 0 \Leftrightarrow\\ -2\frac{T}{S_0}\sum_{i=1}^N\Big(\frac{q_i}{T}\Big)^{\delta}\Big(I_i-\hat{\theta}\frac{T}{S_0}\Big(\frac{q_i}{T}\Big)^{\delta}\Big) & = 0\Leftrightarrow\\ \hat{\theta}(\delta) & = \frac{S_0}{T}\frac{\sum_{i=1}^N(\frac{q_i}{T})^{\delta}I_i}{\sum_{i=1}^N(\frac{q_i}{T})^{2\delta}}. \end{align}\]

Fig. (10) depicts the estimated impact parameters as a function of the number of metaorders \(N\) for all three values of \(\sigma\). We observe that the estimators are consistent since the standard errors get decrease as \(N\) increases and approach the true value of the parameters. Fig. (11) illustrates the strategy obtained with the \(\text{regSLSQP}_N\) for \(N=500\) and \(N=5000\) metaorders when the volatility is \(0.0168\) in panels (a) and (d), when the volatility is \(0.003\) in panels (b) and (e) and when the volatility is \(0.002\) in panels (c) and (f). Clearly the expected cash-flow is positive since we use the complete information of the price dynamics for its computation.

11 The training phase of DDPG↩︎

Here we plot the actions’ aggressiveness (see Eq.(7 )) on the left plot of Fig. (12) and the actions’ directionality (see Eq.(8 )) on the right plot of Fig. (12) during the training phase of DDPG when the reward is \[\label{eq:32rev32reward} r_t = -S_tv_t-\kappa v^2_t\tag{13}\]

We observe that, although the level of aggressiveness remains approximately the same throughout the training episodes with that of Fig. (6), the directionality of the actions is reversed due to the negative sign in the reward function, relative to the directionality obtained when the reward is defined as in Eq.(6 ).

References↩︎