1 Introduction↩︎

Currently, there is a large focus on research to deliver the energy transition to combat climate change. In the case of electricity, research on the energy transition focuses on the integration of Distributed Energy Resources (DER)s. In this context, there is a large potential in the agricultural sector for the integration of DERs, due to its energy intensive nature and limited use of smart grid technologies.

In an electrical grid, there are multiple, often competing, objectives that need to be balanced in the smart grid [1], such as grid compliance, minimisation of cost and minimisation of carbon emissions. Although previous approaches have been proposed for grid multi objective energy optimisation (MOO) [2], [3], their performance is limited by the precision of their models. In this context, Deep Reinforcement Learning (DRL) methods can be used for energy optimisation in conditions with limited information of the system dynamics. The applications of DRL in smart grids include maximising the use of renewable energy [4] and minimising costs in electric vehicles [5]. Among DRL methods, Proximal Policy Optimisation (PPO) emerges as an ideal method for energy management [6].

MMO applications in smart grid have been presented with some limiations [7], [8]. For example, in [9] MOO is applied to reduce line congestion and costs, however, this approach did not account for multiple generators and storages in the grid. In [10], a MOO framework is proposed for cost minimisation and voltage regulation, however, this approach is limited to the retail electricity market.

To further develop smart grid applications in the rural sector, our proposed control framework combines heuristic optimisation with DRL agents to maximise the use of distributed generation and affordability in dairy farms, accounting for uncertain electricity prices and multiple generation and storage sources. The following section describes the proposed energy management framework.

2 Methodology↩︎

The proposed control system is divided into two layers. The lower layer performs local optimisation in the distribution system using a multi-agent system, where each agent controls a single battery based on the price of electricity provided by the upper layer. The upper layer adjusts the internal price of electricity to regulate the interaction with the main grid. These control layers form a distributed control [11] system as shown in Figure 1.

2.1 Multi-Objective Layer↩︎

The upper control layer performs dynamic pricing by solving the next MOO problem using heuristic optimisation, namely differential evolution:

where \(w_{1}\) and \(w_{2}\) are weights for the objectives \(F_{1}\) and \(F_{2}\), at each time step \(t\) for a duration of \(T\). The first objective is the cost minimisation of the entire system and the second objective is the minimisation of power flow \(P_{grid}\) with the main grid, expressed as:

\[F_{1} = c_{1}\sum P_{grid}(t)\pi_{grid}\]

\[F_{2} = c_{2}\sum (P_{grid}(t))^2\]

where parameters \(\alpha\) and \(\beta\) modify the internal price of electricity using predictive and corrective measures of internal generation, and \(c_1\) and \(c_{2}\) normalise the objectives to one for \(\alpha, \beta = 0\). The power flow \(P_{grid}\) with the main grid depends on the line losses and power Load, \(P_L\), internal Power generation \(P_G\) and Power from batteries \(P\):

\[P_{grid}(t) = P_L(t) - P_G(t) - P\]

The predictive measure, defined as \(\pi_{p}\) in \(euro/kW\), is based on a forecast with length \(K\) of the internal load and generation, \(P_{Lf}\) and \(P_{Gf}\) , calculated as follows:

\[\pi_{p}(t+k) = (1-\alpha) \pi_{grid}(t+k) + (P_{Lf}(t+k) - P_{Gf}(t+k))\alpha, \forall k \in K\]

The internal price \(\pi_{internal}\) is further modified by a corrective measure to account for forecast errors and the reaction of other agents, based on \(P_{grid}\) expressed in \(euro/kW\), as follows:

\[\pi_{internal}(t) = (1-\beta)\pi_{p}(t) + \frac{P_{grid}(t)+P_{grid}(t-1 )}{2}\beta\]

This modifies the internal price according to the current net demand. To increase the price and grid stability, the price changes with a moving average model, using the current and previous price.

2.2 Multi-Agent Layer↩︎

The lower layer uses a distributed control system, with multiple independent DRL agents, to control each battery in the distribution circuit. In this paper, we use PPO for its ability to handle continuous action and state spaces and its stability during training [6], [12]. The DRL agent learns a policy to interact with its environment, in this case the circuit, observing a state \(S\) and taking an action \(a\) sampled from \(A\), to maximise the accumulated reward \(R\).

The objective function \(L\) of PPO during training is described as follows [12]:

\[L(a) = \mathbb{E}_{t}[min( r_{t}(a) [R-R_{exp}],clip(r_{t}(a),1-\delta,1+\delta )[R-R_{exp}]]\]

where \(\mathbb{E}\) is the expectation function at time \(t\); \(r_{t}\) is the ratio of probabilities of the new policy over the old policy of selecting action \(a\); \(R_{exp}\) is the expected reward observing the state \(S\) and \(\delta\) is a hyperparameter that limits policy changes between updates. In this paper, the states observed by the DRL agent are expressed as follows: \[S = \{P_{RB}, SOC, \pi_{internal} \}\] where \(P_{RB}\) is a rule-based power schedule and \(SOC\) is the battery state of charge. The rule-based battery power schedule charges at the lowest forecasted price and discharges at the highest.

Based on the state observed at each hour, the agent selects a power reference \(P\) for the battery sampled from the action space \(A \in [-P_{max},P_{max}]\). This is used to define the power schedule of each battery. During training, the power schedule is used to calculate the agent’s reward. In this case, the reward is the total cost of energy, that is:

where \(SOC_{max}\) is the maximum \(SOC\). \(SOC\) is calculated as follows with charge and discharge efficiencies, \(\eta_{c}\) and \(\eta_{d}\):

\[\label{soc} SOC(t+1)= \left\{ \begin{array}{ll} SOC(t)+\eta_{c}P \quad P>0 \\ SOC(t)+\eta_{d}P \quad \text{else} \end{array} \right.\tag{1}\]

3 Test Case↩︎

The Multi-Objective layer implements differential evolution using Scipy with parameters specified in Appendix A, and the DRL agents are trained using PPO from stable baselines 3 (SB3). \(w_1\) is varied between 0.1 and 0.9 in increments of 0.1 to test different weights of each objective, running each configuration 10 times using random seeds. We tested the Irish single market price data from 2022 publicly available from the Single Electricity Market Operator (SEMO), with equal buying and selling prices. The circuit simulation model is shown in Figure 1 using PyPSA and the distributed generation and storage capacity connected to each Bus, that is, a voltage measurement point, is presented in Appendix B. \(\eta_{c}\) and \(\eta_{d}\) are set to \(0.95\) and \(1/0.95\), equivalent to a round-trip battery efficiency of 90%, found in Tesla Powerwalls. The simulated load represents 50 Irish farms distributed across the circuit model [8], and the simulated renewable generation is based on Ireland. The nominal voltage is \(10KV\) and the lines have an impedance of \(0.157+i0.123 \; \Omega/km\). The price forecast is provided to each DRL agent with lengths of 0 (no forecast), 6, 12 and 24 hours with 0, 5, 10 and 15% added error to represent realistic conditions. PV generation and load forecast is provided similarly for dynamic pricing.

The DRL agent uses two fully connected layers, testing 5 cases: 1) using only the rule-based model. 2) using the DRL agent with 7-day trajectories for training with 64 neurons in each layer. 3) 14-day trajectories with 62 neurons in each layer. 4) 30-day trajectories with 128 neurons in each layer. 5) 30-day trajectories with 256 neurons in each layer. The training lasts 2 million steps, where each step represents one hour. In total, 640 configurations are tested.

4 Simulation Results↩︎

During peak renewable generation during 4th and 5th of May of 2022, the simulation results show that voltage and phase angle variations are in compliance with the limits of the Irish grid of 1 volt per unit (p. u.) \(\pm 10\%\) and phase angle of 0 \(\pm 0.69\) radians , on a base of 230 Volts per phase, established in the EN 50160 standard. The simulated scenario represents the highest variation in the electrical response for the data tested and is shown in Figure 2.

In terms of energy arbitrage in each battery, results show highest profits using a 24-hour price forecast in case 5 with 1866 euros. In comparison, using only the rule-based model, the highest profit achieved is 1569 euros. The results using the DRL agents also show that errors of 5% do not significantly affect total costs. Similarly, Figure 2 shows that there is no significant difference between the 12-hour and 24-hour forecast. The total cost of energy for the entire system for one year is 55, 263 euros and the power variability is 34.73 \(MW^2\).

The simulation results show that the Multi-Objective layer can reduce power variability of the entire system with the main grid by 0.85\(MW^2\), without disrupting the cost minimisation achieved by the Multi-Agent layer, using dynamic pricing with parameters \(\alpha, \beta = [0.022,0.305]\), as shown in Figure 2.

5 Conclusion↩︎

This paper presents a hierarchical and distributed energy management framework for the rural sector combining Deep Reinforcement Learning and differential evolution for Multi-Objective Optimisation. The use of DRL agents achieves up to 18% increase in profits from energy arbitrage compared to using a rule-based power schedule. The use of dynamic pricing can reduce the power variation with the main grid by \(0.85 MW^2\) and maintain a similar cost compared to using only the DRL agents, which promotes the use of distributed generation and voltage stability. In addition, the simulation of the voltage response shows that the proposed control system is in compliance with the Irish grid standard EN 50160. Furthermore, simulation results show that a relatively small difference in the accuracy of the electricity price forecast did not significantly affect profits from energy arbitrage.

Appendix A↩︎

The parameters for differential evolution used in the Multi-Objective Layer are described in Table 1.

Appendix B↩︎

The distributed generation and storage capacity in the circuit model shown in Figure 1 is described in Table 2.

References↩︎