Stochastic battery optimisation by hand: market price vs continuation value

Imagine a battery entering a market interval with 0.5 MWh of energy. The current price is 20 Euro/MWh, but the expected marginal value of having another MWh available in the next stage is 55 Euro/MWh.

Even though the current price is positive, the optimal decision is to charge i.e. spend money right now.

I wanted to understand where this decision comes from before letting an optimization library solve it for me. So I worked through a small three-stage battery problem using stochastic dynamic programming by hand.

The Julia package SDDP.jl models decision points as nodes. In a linear policy graph, these nodes correspond to stages along the time axis: https://sddp.dev/stable/tutorial/first_steps/

Model and information

With $e_t$ the SoC of the battery in MWh entering stage $t$ and $p_t$ the grid-side battery power in MW. Grid-side means: $p_t > 0$ the battery discharges and vice versa.

To keep things simple, for a one hour settlement period and 100% efficiency:

Since $\Delta t = 1$ h, I suppress $\Delta t$ below. More generally:

and stage P&L is $\pi_t p_t \Delta t$.

Let’s go through this for a 1 MW / 1 MWh battery:

State (SoC) transitions are therefore:

At the beginning of each stage, before making our trading decision, we observe the price of the given stage $\pi_t$. Before observing the price of the stage, we track multiple price scenarios per stage, e.g.:

For this toy example, assume the price scenarios are independent across stages.

Across all stages we want to maximize our expected P&L:

The Bellman equation or value to go at time $t$, i.e. the maximal expected P&L from stage $t$ onward after the current price $\pi_t$ has been observed:

subject to:

Here, we choose dispatch / market position $p_t$. Once we choose $p_t$, the SoC at the beginning of stage $t+1$ is determined by $e_{t+1}=e_t-p_t$, so let’s write:

Let’s introduce a new term that summarizes the expectation:

• $V_t(e_t,\pi_t)$: I have entered stage $t$, I know my SoC, and I have already observed the current price. What is the maximum P&L I can make from here onward?
• $Q_t(e_t)$: I know my SoC entering stage $t$, but I have not yet observed the stage-$t$ price. What is my expected value from here onward?

In my notation, $Q_t(e_t)$ is the expected value entering stage $t$ before observing $\pi_t$. Viewed from stage $t-1$, $Q_t(e_t)$ is the expected continuation value attached to the outgoing state $e_t$.

SDDP.jl calls this expected successor value the value function, explicitly distinguishing it from $V_t(e_t,\pi_t)$.

With this we can rewrite the Bellman equation as:

Let’s solve a three-stage problem using this Bellman formulation.

Suppose at stage 1 we already observed price:

For stage 2 we have the following price scenarios:

For stage 3 our price scenarios are:

Stage 3: terminal value

With dynamic programming we always start at the end of the time axis, i.e. we define the terminal continuation value right after the last stage we are interested in:

So for any SoC at the end of stage 3, which would be the SoC we enter stage 4 with, we prescribe zero continuation value. We do so because in this toy problem we assign no terminal or salvage value to energy left after stage 3.

With this, we can now write the value to go for stage 3 as:

At this point, the beginning of stage 3, both $e_3$ and $\pi_3$ are known since at the beginning of each stage we resolve the current-stage price uncertainty and observe the price.

The only unknown here now is $p_3$, which is constrained by the limits of the battery:

So we cannot simply choose an arbitrary $p_3$ here. We need to respect where the SoC of the battery comes from at the beginning of stage 3, $e_3$.

Imagine $e_3=0.6$ MWh. From the SoC constraint our feasible power range becomes:

What would we choose for the different price scenarios in stage 3?

So the value of the SoC entering stage 3, $e_3$, is given by how much we can discharge in stage 3, because leaving energy in the battery has no terminal value, and the price we observe in stage 3, $\pi_3$.

Before observing the price of stage 3 we can only work out the expected continuation value from and including stage 3:

For an arbitrary beginning of stage 3 SoC this is:

This is the continuation value function of stage 3. Taking its derivative with respect to the SoC at the beginning of stage 3:

This derivative tells us that each additional MWh stored in the battery at the beginning of stage 3 is worth an additional 55 Euro in expected future P&L. That 55 Euro/MWh is the marginal continuation value of stored energy.

Stage 2: charge or discharge?

Now let’s move to stage 2: Suppose we enter stage 2 with SoC $e_2=0.5$ MWh and observe price $\pi_2=20$ Euro/MWh. What does the Bellman problem tell us we should do?

The Bellman equation / the value to go for stage 2 is:

This is the objective we are maximizing at stage 2:

subject to power and energy constraints:

and:

This gives us the following constraints we need to respect:

So our feasible power range in stage 2 is:

meaning our most extreme choices are charge, negative $p_2$, at 0.5 MW and discharge, positive $p_2$, at 0.5 MW.

The objective and constraints tell us to charge the battery in stage 2 up to its maximum SoC of 1 MWh:

This is not surprising because we already worked out that the marginal continuation value of stored energy in stage 3 is:

Given that charging in stage 2 costs us 20 Euro/MWh while the marginal continuation value of that extra MWh at the beginning of stage 3 is 55 Euro/MWh, we obviously charge in stage 2. Charging another MWh adds:

of expected P&L.

Imagine we had observed $\pi_2=80$ Euro/MWh coming into stage 2 at the same SoC $e_2=0.5$ MWh.

Then the objective would have been to maximize:

and it would have been more lucrative to discharge 0.5 MW in stage 2. The difference between the current market value in stage 2 and marginal continuation value of stage 3 gives the same result:

Since $p_2>0$ means discharge, the positive coefficient tells us to discharge.

So the tradeoff we consider at each stage is: market value of our energy in the current stage vs its marginal continuation value in the next stage.

Let us formulate the pre-price value function and derive the marginal continuation value for stage 2.

For every $e_2\in[0,1]$, our 1 MW battery can reach either SoC boundary within the one-hour stage. At 20 Euro/MWh we therefore charge to $e_3=1$, which gives:

At 80 Euro/MWh we discharge to $e_3=0$, which gives:

Then:

Realize that $V_2(e_2,20)$ and $V_2(e_2,80)$ are two different $\pi_2$ observations that lead to two different decision branches. For the stage 2 pre-price value function $Q_2(e_2)$ we average over them.

So the marginal continuation value in stage 2 is:

Stage 1: optimal expected P&L

Let us now go to the beginning of our optimization horizon, stage 1, where we already observed:

At stage 1 the marginal continuation value of stored energy is 50 Euro/MWh, while the observed market price is 40 Euro/MWh:

So if possible given our SoC we will charge our battery in stage 1.

Let us finish the three-stage problem. Suppose we begin stage 1 with:

The stage 2 examples above looked locally at $e_2=0.5$ MWh. Starting the complete three-stage problem from $e_1=0.5$ MWh, the Bellman problem at stage 1 is:

Given $e_1=0.5$ MWh, our feasible power range is again:

So we charge:

and enter stage 2 with:

Our optimal expected P&L for the complete three-stage problem is therefore:

It has been fun working through this toy example by hand as it has taught me some intuition regarding:

• Across our sequential decision making horizon we always need to think in terms of what info we know coming into a stage, before current-stage price uncertainty is resolved, and the state we create at the end of that stage.
• Important objects $Q_t$, which is the expected value from stage $t$ before observing the stage-$t$ price, and $V_t$, the value from stage $t$ after the current-stage price has been observed.
• At each stage our decision essentially boils down to comparing the current observed market price and the marginal continuation value of energy in the next state.

Market applications

Where can we apply this to market channel optimization for batteries and other steerable assets?

To my mind, but I need to think about this further, we can probably apply this in at least two phases:

• The day-ahead positioning of our battery where we need to decide how much of our battery’s limited resources to allocate to market channels such as FCR, aFRR capacity, and day-ahead wholesale. The problem certainly would need to be reformulated so would look differently from the above toy example.
• Intraday continuous / real time market trading where this could probably be helpful in making decisions near-term in the liquid phase of the limit order book and weigh those decisions against further out less liquid phases of the same limit order book.

In this toy problem we could derive the continuation value functions exactly by hand. For anything approaching a realistic multistage problem, we cannot.

To use this stochastic optimization approach for any meaningful work I will next dive into formulating the above toy example using SDDP.jl and see that this gives me the same results as I got by hand, including the 47.5 Euro optimal expected P&L.