leaderboarding
/ˈliːdəˈbɔːdɪŋ/
verb
1. Organising, comparing and ranking a series of items, objects or commodities based on their quantitative performance in a pre-agreed area or sector.
2. The process of creating, maintaining, and updating the Modo Leaderboard codebase.
e.g. “I spent 16 hours leaderboarding today, and now I am fatigued.”
Leaderboarding is a labour of love. Since its launch two years ago, we’ve been working constantly to refine, improve and expand the Modo Leaderboard. In doing this, we've increased the complexity of the processes and methodologies that go into making the Leaderboard what it is. It's worth noting here that to build a Leaderboard that provides real value and insight, you can't simply take £/MW/h ancillary service figures and multiply them by the number of hours in the month. We take pride in our Leaderboard, which leads us down rabbit holes of physical asset performance, market participation and energy throughput. At Modo, we've been burrowing further into these rabbit holes since December 2019. And it's very dark down here.
At Modo, we put transparency at the forefront of everything we do. With that in mind, here’s a detailed explanation of how we build the Leaderboard, and the assumptions that we make in doing so.
Purpose of the Leaderboard
The Leaderboard is all about visibility.
In isolation it’s too crude a tool to be used to rank route-to-market providers, and it’s important when looking at the Leaderboard that you read our accompanying analysis (e.g. this piece on the rise of Dynamic Containment).
The Leaderboard is an independent, consistent measure of how much specific assets are earning, and of where revenues are being generated. This information is vital for existing and potential owners of battery energy storage (as well as their operators).
To produce the Leaderboard each month we follow the 10 steps outlined below in Figure 1.
Step 1: Data collection
We collate and curate data from numerous sources, including National Grid ESO, DNOs (e.g. UK Power Networks), ELEXON (BMRS), EPEX SPOT and Nord Pool. We also use Modo's proprietary data, which includes our battery asset database, market ID maps, and asset performance information from previous Leaderboard months.
We collect data via several methods, including using APIs (Application Programming Interfaces), FTPs (File Transfer Protocols), and in some cases scraping algorithms, e.g. determining Distribution Use of System charges (DUoS) by scraping the Schedule of Charges spreadsheets from each DNO.
This data is continually updated on our servers to support the Modo platform.
Step 2: Market schedule
Next, we build a picture of what each market ID is doing across the month. So, for each market ID (e.g. FFR ID, DC ID, BMU ID, CMU ID), we produce a schedule of operation on a settlement period basis throughout the month.
This stage is vital for identifying gaps in operation or changes in asset IDs later on in the Quality Assurance (QA) process. When creating the market schedule, our algorithms let us know of any new battery asset IDs that have entered the market (which might still be unaccounted for - if we haven’t already found them, that is).
Step 3: Energy throughput calculation
Based on the market schedule (Step 2), we calculate the energy throughput (import and export) for each market ID.
- For frequency response services, we model the contracted response for each service using the physical (measured) grid frequency grid data provided by National Grid ESO at the end of the month.
- Physical reported volumes (final physical notifications (FPNs)) in the Balancing Mechanism (BM) are used in an integral function to determine energy throughput in wholesale markets.
- Bid-Offer Acceptances (BOAs) are collected for each BMU ID from the BM, to calculate the total bid (import) and offer (export) volumes.
- Where BOAs coincide with reported physical volumes in the BM, traded volume in the BM is calculated by removing the physical throughput from the BOA volumes.
- Costs associated with state-of-charge management are estimated through algorithms using all the throughputs calculated above.
Step 4: Market revenues
Once we’ve calculated the energy throughput, we assign revenues to each market ID.
For ancillary services, we look up accepted clearing prices (£/MW/h) and apply them to each market ID for each settlement period. As there are many frequency response services, auction systems, and periods, each service is calculated separately.
For the Balancing Mechanism, BOA volumes are multiplied by the accepted bid/offer prices (£/MWh) to determine revenues.
For activity in the wholesale markets, revenues are calculated based on two things - energy throughput (a discrete integration under the FPN to determine volume) and the price series in the most applicable power exchange (typically Nord Pool’s day-ahead continuous auction).
For non-BMU registered assets, calculating wholesale revenues is more complex. The Modo team analyses these assets on a settlement period basis, determining whether they could potentially be trading or are out-of-market when there is no public visibility of a site’s operation. Wholesale revenues are estimated by calculating the largest spread in Nord Pool day-ahead markets (depending on site duration) across days when the team believes assets are pursuing merchant trading activity. Revenues are assigned based on these estimations under the assumption each asset operates using one battery cycle per day. Assets are then reviewed by site duration against BMU-registered assets the team has visibility on, to determine the credibility of our estimations and make any final adjustments.
Capacity Market (CM) revenues are calculated on a monthly basis, by applying the monthly weighting factor by the total yearly CM revenue applicable to each individual asset. CM revenues are identified and calculated by going through all awarded contracts for each auction year, and mapping them to assets (as you can imagine, this is one of our team’s favourite tasks). We also account for secondary traded contracts where we can see volumes have moved between CMUs. In some cases, (bilateral) agreements between CMUs in the secondary trading process include a fee to assign the volumes (particularly with aggregator counter-parties). These fees are invisible to us, so we do not discount secondary traded volumes and revenues to account for this.
Step 5: Map to physical assets
Once all market IDs have been assigned both revenues and energy throughputs for each settlement period of the month, each market ID is mapped to a physical asset. Most assets have more than one market ID associated with them and these can change over time, so this process is vital in ensuring all physical assets receive their correct allocation of revenue streams from all markets.
Step 6: Net throughput calculation
Once all assets have been assigned to their market IDs, revenues, and contracted energy throughputs, net energy throughputs are aggregated to 30-minute settlement periods. These aggregated volumes are used as inputs for costs/charges calculations for assets.
Step 7: Use-of-system charges
We calculate the use-of-system charges payable to/from all assets for each settlement period throughout the month.
- Fixed DUoS - we assign a fixed £/kVA amount depending on the asset’s connection (taken from Modo's asset database). Key input variables are connection voltage, type, and distribution network operator (DNO).
- Variable DUoS - we calculate the energy-based DNO costs associated with the asset's connection by multiplying our estimated net energy throughput by the respective green, amber and red tariffs applicable for each settlement period. DUoS charges change annually, and duos rates, bands and periods vary between DNOs and connection types.
- Balancing Services Use of System charge (BSUoS) - we calculate the applicable BSUoS costs associated with the asset by multiplying the energy throughput (import and export) by the respective BSUoS rate.
- Transmission Network Use of System (TNUoS) - asset-specific TNUoS tariffs are calculated for both generation and demand. The generation costs are a fixed rate applied each month, and demand charges are calculated each month depending on energy throughput during specific settlement periods.
The final process is filtering use-of-system charges payable to/from assets. This is determined by their liability to pay specific charges which are decided by factors such as geographical location and grid connection.
Step 8: Aggregate data
For each asset, we aggregate the revenues, costs and energy throughputs across the month.
Step 9: Quality Assurance
Quality Assurance is by far the most critical stage in the process. The research team reviews each asset individually, creating a view of each asset’s market activity for the month.
Before we publish the Leaderboard, the Modo team gets around the kitchen table and discusses each asset in detail, checking for periods of unavailability or unusual behaviour (e.g. maintenance, testing, market pre-qualification). When we publish it, we must be confident in the Leaderboard, including any methodology revisions, upgrades, and policy or market changes.
Step 10: Rankings
We then normalise each asset’s earnings on a £/MW basis and rank in the Modo Leaderboard from highest to lowest earnings for that month. We also calculate year-to-date earnings and annualised earnings.