---
title: "Time Series Forecasting"
book: time-series-forecasting
edition: 2026.11.11
verified: 2026-11-20
edition_code: 1RPH-3A4D
canonical: https://rollingedition.com/time-series-forecasting
license: CC BY-NC 4.0
more_permissions: https://rollingedition.com/policies/reuse
attribution: "Time Series Forecasting, Edition 2026.11.11, examples passed Nov 20, 2026. By Rolling Edition, https://rollingedition.com/time-series-forecasting. Text CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/), with added permissions (https://rollingedition.com/policies/reuse). Code MIT, copyright Rolling Edition and contributors. Written with AI. Verified by people."
---
Edition 2026.11.11. Examples passed Nov 20, 2026. 41 of 41 examples pass. Written with AI. Verified by people.

To cite a section, name Edition 2026.11.11 and link to the section's permanent link: https://rollingedition.com/time-series-forecasting/s/{section_id}

# Forecast tomorrow's demand

This chapter builds the first forecast of tomorrow's demand, and the simple baseline that every later forecast has to beat. You'll load one client's hourly demand, forecast its next day with a seasonal naive baseline and with one statistical model, and score both on the same 7 days. The result is rough on purpose: no ranges, no tuning, and one client to judge by. Each later chapter makes one part of it solid. You need Python, pandas, the mean of a sample and what an error is, and nothing else; the core terms are defined in a line where they first appear.

Tomorrow's electricity has to be bought today, before the day starts, one hour at a time. Whoever buys it needs a forecast of how much its clients will use in each of those hours, and every hour the forecast misses is an hour the purchase plans around the wrong number. Two questions therefore come before any method is trusted. Is a forecast made only from readings recorded before the purchase close to what the clients actually used? And is it better than the simplest forecast anyone could make, one that copies yesterday?

## The case and its data {#the-case-and-its-data}

This book follows one case throughout: an electricity supplier that buys power a day ahead for its clients, which we call the supplier.

The supplier commits to its purchases before the day starts, and it needs a range as well as a single number, because buying too little costs more than buying too much. This book builds the forecast that such a purchase starts from. A forecast here is the expected demand of one client in each hour, made from readings recorded before the forecast was made.

Three terms fix what a forecast is. The moment a forecast is made is its cutoff. A forecast uses only the readings before its cutoff, and it reaches forward a fixed number of hours, its horizon. A window is one cutoff with its horizon and the forecast it produces. Here the horizon is 24 hours, so each window covers one day: the 24 hours that start at its cutoff.

The timing matters as much as the method. Every window in this book has its cutoff at 00:00 on the day it predicts, so the last reading it can use is the one for 23:00 on the day before. The case buys before the day starts, so a midnight cutoff is a simplification. The book keeps it so that each window needs only the day before. A supplier that buys earlier would cut earlier and forecast further ahead. A forecast that reads one hour past its cutoff would use information the supplier does not have when it buys, and its score would flatter it.

The data is public. It is the ElectricityLoadDiagrams20112014 dataset from the UCI Machine Learning Repository, by Trindade (2015), with the DOI 10.24432/C58C86. It is licensed under CC BY 4.0. The book ships a cut of it, `electricity_hourly.csv.gz`, with the licence beside it in the same folder. The [data card](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/data/README.md) gives the source, the cut and its limits.

The raw file holds 370 clients, with a reading every 15 minutes. The cut averages the four readings of each hour, which gives the energy used in that hour in kWh, and it keeps two whole years, 2013 and 2014. One client, `MT_223`, has no non-zero reading in that window, so the cut drops it. The 369 clients that remain are sorted by id, and 30 are kept at evenly spaced positions. A reader who cuts the raw file must drop `MT_223` and take the same 30 evenly spaced positions, or the counts in this book will not match.

The clients have anonymous ids such as `MT_014`. The dataset does not say what they are, so the book calls them clients. It does not describe them as companies or households.

Two groups of clients recur through the book. The thirty clients are every client in the cut. The 8 clients of `SUBSET8` are every fourth client of the thirty, starting with the second. Later chapters use that subset where a full run would take too long, and it includes one client that starts late. The lead client, `MT_014`, is the first client of `SUBSET8`. This chapter and its single-client examples use it.

The test period is the last week of 2014, from Dec 25 to Dec 31, 2014. The forecasts in this chapter are scored on it, and it is called the held-out week. Each forecast uses only readings from before its cutoff, so no forecast learns from the day it predicts. The week is not quiet. Dec 25, 2014 is Christmas Day, a national holiday in Portugal, where the data's clock is set. This chapter scores the week as it is, and the scores show what the holiday does. Chapter 5 tests a holiday flag on holiday windows and does not keep it, so the book does not repair the holiday here. Chapter 10 shows its cost.

A forecast says what demand is likely to be. It does not say why. The data records how much each client used. It cannot say what demand would have been under a different price or contract. That is a causal question, and it needs a different design.

**Working rule:** Check the cut against the data card before any forecast is made. The file should hold 30 clients, and the example counts how many of them have a full two years of hourly rows. Those counts should match the card. Confirm that `MT_223` is missing and explained. A cut that does not match the card is different data, and every number computed from it describes that data, not this one.

## Load one client's hourly demand {#load-one-clients-demand}

A time series is a sequence of readings in time order. In the book's file, each row is one client in one hour, and each client's hours are in time order. To load one client, keep that client's rows. The helper `load` in `examples/lib/electricity.py` does that and returns four columns. `unique_id` names the client. `ds` is the hour. `y` is the energy used in that hour, in kWh. `quirk` tells you whether the row needs special handling.

The `ds` column is the start of the hour, in local clock time with no time zone. The data card gives this label, and the book's examples treat the index as one naive hourly index. Each row's `quirk` is one of four labels:

- `normal` is an ordinary hour.
- `before_first_reading` is an hour before a client's first non-zero reading. Only a late-starting client has these rows.
- `clock_change_gap` is 01:00 on a March clock-change day. The dataset's note says it reads zero for every client. The data card measures it: of 57 such hours with non-zero neighbours, 55 read below 5% of them.
- `clock_change_repeat` is 01:00 on an October clock-change day. It holds about two hours of energy under one label.

The book keeps every value as the meter recorded it. Nothing is filled, smoothed, removed or rescaled. A zero in a gap hour and a doubled October hour are therefore still in the file, flagged but not corrected. Chapter 3 decides what each flagged row means for a model. This chapter only needs to know the labels exist, and to check that none of them falls inside the windows it scores.

The lead client has readings in every hour of the window, so none of its rows are `before_first_reading`. Over the two years it has 17516 ordinary rows. It has 2 gap rows and 2 repeat rows, one of each in each year.

Example `load-lead-client` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Loads the lead client's hourly demand, counts its quirk labels, and checks the eight-week history before each
held-out cutoff. The point rests on the last printed line: no history holds a clock-change row."""
from lib.backtest import held_out_cutoffs, history_before
from lib.electricity import load
from lib.settings import LEAD, show


def fmt(ts):
    return ts.strftime("%Y-%m-%d %H:%M")


FULL_ROWS = 2 * 365 * 24  # two years of hours; 2013 and 2014 are not leap years

everyone = load()
show("clients in the file", everyone["unique_id"].nunique())
show(f"clients with {FULL_ROWS} rows", int((everyone.groupby("unique_id").size() == FULL_ROWS).sum()))

df = load([LEAD])
show("rows", len(df))
show("first reading", fmt(df["ds"].min()))
show("last reading", fmt(df["ds"].max()))
for quirk in ["normal", "before_first_reading", "clock_change_gap", "clock_change_repeat"]:
    show(f"{quirk} rows", int((df["quirk"] == quirk).sum()))

# history_before keeps the HISTORY_HOURS hours (eight weeks) before each cutoff, and raises if a count is off.
windows = [history_before(df, cutoff) for cutoff in held_out_cutoffs()]
show("history rows per window", min(len(w) for w in windows))
show("longest history rows", max(len(w) for w in windows))
show("first history reading", fmt(windows[0]["ds"].min()))
# Counts only the two clock-change labels.
show("clock-change rows in held-out histories",
     sum(int(w["quirk"].isin(["clock_change_gap", "clock_change_repeat"]).sum()) for w in windows))
```

Output:

```text
clients in the file: 30
clients with 17520 rows: 30
rows: 17520
first reading: 2013-01-01 00:00
last reading: 2014-12-31 23:00
normal rows: 17516
before_first_reading rows: 0
clock_change_gap rows: 2
clock_change_repeat rows: 2
history rows per window: 1344
longest history rows: 1344
first history reading: 2014-10-30 00:00
clock-change rows in held-out histories: 0
```

The file holds 30 clients. The lead client has 17520 rows, from 2013-01-01 00:00 to 2014-12-31 23:00.

Two years of hours is 17520 rows for each client, and 30 of 30 clients have exactly that many, so no hour is missing from the file for those clients. A window with fewer hours than its history should hold has a gap, and the gap would be the first thing to explain. Check the client count and the row count against the card.

A forecast needs history. The book's history is 8 weeks, which is 1344 hours, and each cutoff in the held-out week gets exactly that much before it. The shortest history has 1344 rows, and the longest has 1344.

For the first cutoff of the held-out week, the history begins at 2014-10-30 00:00. That is after the clock change of Oct 26, 2014, which the data card dates. So none of the 7 histories holds a clock-change hour, and the example counts 0 clock-change rows across all seven.

The rest of this chapter can leave the clock-change rows alone for that reason. Chapter 3 meets them, and Chapter 4 meets the late starter.

**Working rule:** Choose the client and the window first, then count the quirks in that window's history before any model sees it. If a history holds a clock-change label or a `before_first_reading` row, do not hand it to a model unchanged, because the model will read the label's value as demand. Chapter 3 and Chapter 4 give the rules for those rows. A window that holds neither can use its rows as they are, as this chapter does.

## Forecast the next day with seasonal naive {#forecast-with-seasonal-naive}

The first forecast is the simplest one to make, and every later forecast has to beat it. It uses the case's cutoff and horizon from [the case and its data](#the-case-and-its-data), and the history that [loading one client](#load-one-clients-demand) describes.

Seasonal naive says that each future value equals the last observed value from the same season. Hyndman and Athanasopoulos define it for a season of any length `m`: the forecast for a future value is the value observed one season earlier ([Hyndman and Athanasopoulos, *Forecasting: Principles and Practice*, ch. 5](https://otexts.com/fpp3/toolbox.html#sec-simple-methods)). With hourly data and a daily cycle, the season is 24 hours, so the forecast for an hour of tomorrow is the reading at the same hour today. In this book's notation, the forecast for hour `t` is `y(t - 24)`.

Nothing in this rule is fitted. The forecast is a copy of the data, so it needs no estimation step. It never reads ahead of its cutoff, whichever cutoff it is. For each hour from the cutoff to 23 hours after it, `t - 24` falls between 24 hours before the cutoff and the hour before it, so the copy uses only history.

The held-out week has 7 windows, one for each day from Dec 25 to Dec 31, 2014, each made at 00:00. The supplier's question is tomorrow's demand for `MT_014`. For Dec 25, the forecast copies the readings of Dec 24 at each hour, and the same rule gives every other day of the week its forecast from the day before.

Take the window at 00:00 on Dec 25 as a worked case. Its forecast for 00:00 on Dec 25 is the reading at 00:00 on Dec 24, and its forecast for 23:00 on Dec 25 is the reading at 23:00 on Dec 24. Of the 1344 hours in its history, this method uses only the last 24, because statsforecast's `_seasonal_naive` repeats the last season of values (statsforecast 2.1.1, `statsforecast/utils.py`). The example still passes all of them, because the same call serves the models that do use more of the history, and one call with one history length is one form to learn, not two.

Why a day and not a week? A season of 168 hours would copy the same hour of the same weekday from the week before, which is another valid baseline for a series with a weekly rhythm. This chapter uses the daily season because the question is the next day, and because the day-ahead forecast is the one the case needs.

Example `seasonal-naive-next-day` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Forecasts the lead client's next day with seasonal naive for each day of the held-out week, then scores it.
The point rests on the MAE and RMSE lines, which score a copy of yesterday, and on the missing forecasts line."""
import pandas as pd
from statsforecast import StatsForecast
from statsforecast.models import SeasonalNaive

from lib.backtest import forecast_windows, held_out_cutoffs, scoring_counts, score_windows, window_mae
from lib.electricity import load
from lib.settings import HORIZON, LEAD, show


def day_mean(day):
    return df[df["ds"].dt.normalize() == pd.Timestamp(day)]["y"].mean()


# statsforecast reads a frame with three columns: the series id, the time and the value.
df = load([LEAD])[["unique_id", "ds", "y"]]
# season_length 24 is the daily cycle of hourly data; freq "h" is pandas' code for hourly steps; one job runs in one process.
sf = StatsForecast(models=[SeasonalNaive(season_length=24)], freq="h", n_jobs=1)
forecasts = forecast_windows(sf, df, held_out_cutoffs())
scores = score_windows(df, forecasts, "SeasonalNaive")
counts = scoring_counts(df, forecasts, "SeasonalNaive")

show("horizon hours", HORIZON)
show("windows", counts["windows"])
show("scored hours", counts["scored_hours"])
show("missing forecasts", counts["missing_forecasts"])
# The readings at the forecast hours, which are the hours the scores cover.
show("mean demand", df[df["ds"].isin(forecasts["ds"])]["y"].mean())
show("mean demand 2014-12-24", day_mean("2014-12-24"))
show("mean demand 2014-12-25", day_mean("2014-12-25"))
show("MAE", scores["mae"])
show("RMSE", scores["rmse"])
for day, mae_day in window_mae(df, forecasts, "SeasonalNaive").items():
    show(f"MAE {day:%Y-%m-%d}", mae_day)

# The check is run on a history with one hour removed, to show it refuses a short history.
cutoff = held_out_cutoffs()[0]
gap = df[df["ds"] != cutoff - pd.Timedelta(hours=1)]  # drops the hour just before the first cutoff
try:
    forecast_windows(sf, gap, [cutoff])  # history_before raises ValueError unless every client has exactly HISTORY_HOURS rows before the cutoff.
    refused = "no"
except ValueError:
    refused = "yes"
show("history with one hour removed refused", refused)
```

Output:

```text
horizon hours: 24
windows: 7
scored hours: 168
missing forecasts: 0
mean demand: 46.62
mean demand 2014-12-24: 47.58
mean demand 2014-12-25: 38.40
MAE: 8.36
RMSE: 10.87
MAE 2014-12-25: 9.41
MAE 2014-12-26: 12.16
MAE 2014-12-27: 5.10
MAE 2014-12-28: 11.16
MAE 2014-12-29: 8.68
MAE 2014-12-30: 5.26
MAE 2014-12-31: 6.75
history with one hour removed refused: yes
```

Each forecast is scored against the readings that followed, and every one of the 168 scored hours has a forecast. Those hours average 46.62 kWh, the level against which every error in this chapter should be read.

Seasonal naive is a baseline, and when a daily cycle dominates the series it can be a strong one. Its weakness is that it copies a single day. An unusual day becomes the forecast for the next, and the method has no way to average it out.

**Working rule:** Keep seasonal naive in every comparison for an hourly series with a daily cycle. It needs no fitting, so it runs on every window at no extra cost. A model that cannot beat it on the same windows has not earned its extra work. Check the forecasts for missing values before you read their score, because a missing reading in the last season becomes a missing forecast. Chapter 3 gives the rule for filling those hours before any model runs.

## Forecast the next day with one model {#forecast-with-a-statistical-model}

The baseline above answers what a forecast gets with no model at all. The question here is whether a fitted model does better on the same 7 windows, and the model is AutoETS. statsforecast 2.1.1 describes AutoETS as selecting the best ETS (error, trend and seasonality) form with an information criterion (`statsforecast/models.py`, class `AutoETS`). The criterion AutoETS uses is AICc, the Akaike information criterion corrected for small samples. It is set in `ets_f`, in `statsforecast/ets.py`, and `AutoETS` offers no other.

The `model` string sets which forms the model may take, with one letter each for error, trend and season. `Z` lets AICc choose, and `A` fixes the form to additive. This chapter uses `ZZA`: AICc chooses the error and the trend, and the season is additive, so every candidate has a daily cycle. The default, `ZZZ`, also lets AICc choose the season. On this client and these 7 cutoffs it chose no season in every window, a flat form, ETS(A,N,N), so its forecast is one value repeated for the day. This chapter asks whether a model with the daily cycle beats a copy of yesterday, so it does not use that form. Other seasonal forms score differently on this week, and Chapter 4 looks at the forms and the choice.

A fit does three things, and each one matters for what the score means. At a cutoff, AutoETS reads the 1344 hours before it and compares the candidate forms, keeping the one with the lowest AICc. AICc rewards a close fit to the history and charges for each parameter the form uses, so a form with more parameters must fit much better to win. The model then projects that form 24 hours ahead. The held-out week therefore means 7 separate selections, not one. In this week 7 windows chose ETS(A,N,A): additive error, no trend and an additive daily season. The 7 fits still differ in their parameters. The book does not quote them, because each is a detail of one fit, and a reader should not take it as a finding.

The comparison is fair only when nothing else changes. It uses the same 7 cutoffs and the same 1344 hours of history before each one, and it refits the model at every cutoff using only the history before that cutoff. Only the model differs from the baseline.

Example `autoets-next-day` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Forecasts the lead client's next day with AutoETS for each day of the held-out week, then scores it.
The point rests on the MAE and RMSE lines, read against the seasonal naive example, and on the form counts."""
import numpy as np
from statsforecast import StatsForecast
from statsforecast.models import AutoETS

from lib.backtest import chosen_forms, forecast_windows, held_out_cutoffs, scoring_counts, score_windows, window_mae
from lib.electricity import load
from lib.settings import HORIZON, LEAD, SEED, show

# Seeded by convention. AutoETS's point forecast draws no random numbers, so this line changes nothing here.
np.random.seed(SEED)
# statsforecast reads a frame with three columns: the series id, the time and the value.
df = load([LEAD])[["unique_id", "ds", "y"]]
# model="ZZA": AICc chooses the error and trend forms; the seasonal form is fixed to additive, so every form has a day cycle.
sf = StatsForecast(models=[AutoETS(season_length=24, model="ZZA")], freq="h", n_jobs=1)
forecasts = forecast_windows(sf, df, held_out_cutoffs())
scores = score_windows(df, forecasts, "AutoETS")
counts = scoring_counts(df, forecasts, "AutoETS")
# chosen_forms refits at each cutoff to read the form each fit picked; the forecasts above do not report it.
forms = chosen_forms(sf, df, held_out_cutoffs())

show("horizon hours", HORIZON)
show("windows", counts["windows"])
show("scored hours", counts["scored_hours"])
show("missing forecasts", counts["missing_forecasts"])
for form, n in forms.items():
    show(f"windows that chose {form}", n)
show("MAE", scores["mae"])
show("RMSE", scores["rmse"])
for day, mae_day in window_mae(df, forecasts, "AutoETS").items():
    show(f"MAE {day:%Y-%m-%d}", mae_day)
```

Output:

```text
horizon hours: 24
windows: 7
scored hours: 168
missing forecasts: 0
windows that chose ETS(A,N,A): 7
MAE: 8.86
RMSE: 10.91
MAE 2014-12-25: 13.51
MAE 2014-12-26: 11.21
MAE 2014-12-27: 4.17
MAE 2014-12-28: 10.03
MAE 2014-12-29: 9.53
MAE 2014-12-30: 6.33
MAE 2014-12-31: 7.25
```

The counts match the baseline's, so the two scores cover the same 168 hours, and all 7 windows chose the same form. The model makes 7 fits, one at each cutoff, and each sees only what was known at its cutoff. The next section sets the scores side by side.

**Working rule:** Add a model only with a baseline it has to beat on the same windows. Refit the model at each cutoff, using history from before that cutoff only. A model fitted once on the whole series and then scored on that series has seen the answers, and its score is not a forecast score.

## Score both on a held-out week {#score-on-a-held-out-week}

Both examples have now made 7 day-ahead forecasts, from the two sections above. A score turns the errors of a set of forecasts into one number. The error of an hour is the actual demand minus the forecast. This chapter uses two scores. The mean absolute error, MAE, is the average size of the miss: `MAE = mean(abs(actual - forecast))`. The root mean squared error, RMSE, is `RMSE = sqrt(mean((actual - forecast)^2))`. Both are in kWh, and they answer different questions. Hyndman and Athanasopoulos, in [*Forecasting: Principles and Practice*, ch. 5](https://otexts.com/fpp3/toolbox.html#sec-accuracy), note that a forecast minimising MAE is a forecast of the median, and one minimising RMSE is a forecast of the mean. Because RMSE squares each error, a few large misses raise it more than they raise MAE.

Both scores come from utilsforecast (0.2.17). Its `mae` is the mean of the absolute errors, and its `rmse` is the square root of the mean of the squared errors (`utilsforecast/losses.py`, functions `mae` and `rmse`). The scores pool the 168 scored hours. Each hour counts once, and each of the 7 windows contributes 24 of them.

Read the table against the mean demand of 46.62 kWh an hour: a lower number means forecasts closer to the readings.

| Model | MAE (kWh) | RMSE (kWh) |
| --- | --- | --- |
| SeasonalNaive | 8.36 | 10.87 |
| AutoETS | 8.86 | 10.91 |

AutoETS's errors are higher than the baseline's on both scores, as the table shows, with MAE at 8.86 kWh against 8.36. The MAE is the number to plan with, because it is the typical miss in each hour. RMSE is always at least MAE. For normal errors RMSE is about 1.25 times MAE, so a ratio well above that means a few large misses. This section reads the two scores together and goes no further, because one client over one week cannot show the spread of the misses.

Both models do worst around Christmas, but on different days. The daily lines show where: the baseline's worst day is Dec 26, the day after Christmas, when it copies Christmas Day, at 12.16 kWh an hour, the largest of the 7. AutoETS's worst day is Dec 25 itself, at 13.51 kWh, against 9.41 for the baseline.
A gap of this size on one client over one week is not a verdict. Seven forecasts make a small sample, and a ranking built on one week can flip on the next. Chapter 3 backtests seasonal naive over a year of Sunday cutoffs.

The score also has to match the decision, and the purchase is lopsided: as [the case](#the-case-and-its-data) says, buying too little costs more than buying too much. MAE and RMSE treat a miss in either direction alike, because they use the size of each miss and not its sign. That is enough for this chapter, which asks whether a forecast is close. It is not enough for the purchase. Chapter 8 builds ranges for that decision, and scores them against what happened.

**Working rule:** Score each model on forecasts it made without the scored hours in its history, over more than one day, and report the number of windows and hours behind each score. Compare models only on the same windows. Read MAE and RMSE together. If they disagree about which model is better, look at the largest misses in the windows before you choose, because a few large misses can decide the result.

## What goes wrong in production {#what-goes-wrong-ch1}

The forecasts and scores above can go wrong in 5 ways, and only some of them print an error. The [cutoff rule](#the-case-and-its-data) and the [seasonal naive rule](#forecast-with-seasonal-naive) say what each window must hold.

The first failure is a label that is off by an hour, and no score in this chapter detects it. If a reader takes `ds` as the end of its hour rather than the start, the forecasts and the readings they are scored against carry the same shift, so the score does not change. The error is in the purchase instead: the forecast would cover different hours from the ones the supplier buys for. The data card fixes the label, so it is the first thing to check when a forecast is moved to new data.

[Leakage](#keep-the-future-out-of-history) flatters a window whose history includes the hours it predicts, and nothing in the score says so. [Hewamalage, Ackermann and Bergmeir (2022)](https://arxiv.org/abs/2203.10716) warn that rolling-origin evaluation is prone to this leak when it is not done with care. The checks here are the cutoff rule and the history length. Each history must end before its cutoff and hold 1344 rows. `history_before` raises an error when a history is short or long. A history with one hour removed is refused: the check at the end of the seasonal naive example prints yes for it.

Seasonal naive copies yesterday, so an unusual day becomes tomorrow's forecast, and the held-out week holds one. Dec 25, 2014 is Christmas Day, and the lead client's mean demand that day is 38.40 kWh an hour, against 47.58 kWh on Dec 24. The window for Dec 26 copies that day, so it has the largest MAE of the 7 windows, 12.16 kWh. That is the failure, and the scores show it. One holiday is too few to measure a holiday effect, so the book leaves it alone, as [the case and its data](#the-case-and-its-data) explains. Chapter 10 shows its cost.

A missing reading gets through unchecked. The statsforecast seasonal naive does not check its input. It repeats the last season's values, so a missing value in that season becomes a missing forecast (statsforecast 2.1.1, `statsforecast/utils.py`, function `_seasonal_naive`). The seasonal naive example counts missing forecasts, 0 here, because a hole in the forecast is an hour the score does not describe. Chapter 3 fills the hours that need filling, from data before the cutoff, before any model runs.

Last, the window itself can flatter the method. A week with no unusual days can make almost any forecast look good, and a week with one can make every forecast look bad. The held-out week has a holiday in it, and it is still one week for one client, so its scores describe that week more than the method. The fix is breadth: compare models on several windows of different kinds before you trust a ranking.

**Working rule:** Before you trust a score, check three things in this order: each window's first forecast hour is its cutoff, each history ends before its cutoff and holds the full history length, and no forecast is missing. The helpers raise an error for the first two, and the examples print the third. A score counts as a finding only after all three pass.

## Decisions this chapter made {#decisions-ch1}

This chapter made nine choices. Most are settings in `examples/lib/settings.py`, so a later edition that changes one changes it in every chapter that uses it. The rest are stated in the data card or in this chapter. Chapter 2 starts from these two forecasts and their two scores, and asks whether either is good enough to act on.

- Each window's cutoff is at 00:00 on the day it predicts, and the horizon is 24 hours, so each window covers that day.
- The history before each cutoff is 1344 hours, which is 8 weeks.
- The lead client is `MT_014`. The held-out week is Dec 25 to Dec 31, 2014: 7 windows and 168 scored hours.
- The scores are MAE and RMSE, pooled over those scored hours and printed to two decimals.
- The models are seasonal naive with a season of 24 hours, and AutoETS with the model string `ZZA` and a season of 24 hours. Each run uses statsforecast with one job. Neither point forecast draws random numbers, so the seed the AutoETS example sets, by convention, has no effect on it.
- The cut drops `MT_223` and keeps 30 clients, as the data card explains.
- The clock-change and late-starter rows stay as they are, because no history in this chapter contains them. Chapter 3 and Chapter 4 handle them.
- Every number the prose quotes is bound to a labelled line in its example's recorded output, so the text and the output cannot disagree silently.
- The chapter builds one client's forecast and no ranges. Chapter 8 is where ranges begin.

# Set the baselines a model must beat

In this chapter you compute three baselines for a client's day-ahead forecast, naive, seasonal naive and average of weeks, and scale a client's error with MASE, using a scale that reads only the readings before the forecast week.

[Chapter 1](#score-on-a-held-out-week) ended with two scores on the held-out week, and neither can be judged yet, because an error in kWh means nothing on its own. The supplier, whose purchase rests on each hour's forecast, has to set each model's error against what the simplest forecast scores on the same hours, and against how much the client's demand moves from one day to the next. This chapter builds both comparisons. The supplier needs them before it can say that a model is worth its cost.

Every error is a miss in kilowatt hours for one hour, and the scores here are hourly averages of those misses. A baseline's score is the same kind of number. It says how many kilowatt hours of error the simplest rule leaves for an average hour of the day, and a model that cannot beat that figure has not earned its extra cost. So each score is read against the lead client's mean demand as well as against the other rows.

## Why a baseline comes first {#why-a-baseline-comes-first}

A baseline is a forecast that needs no fitting. It copies a value the history already holds, so it costs almost nothing to run, and it sets a bar that any model has to clear. A model earns its place only by beating a forecast that cost nothing. Fitting time, tuning and keeping a model running are all costs, and the baseline's score measures what those costs buy.

Three terms matter here. The [cutoff](#the-case-and-its-data) is the hour at which a forecast is made. The cutoff is the first hour the forecast covers, and the history is every reading before the cutoff. The [horizon](#the-case-and-its-data) is how far ahead a forecast reaches, here 24 hours. A [window](#the-case-and-its-data) is one cutoff together with the horizon it covers, and a backtest is the score of many windows, each made as it would have been made on its day.

The comparison has to be fair, or it measures the setup and not the model. A baseline scored on different hours, with a different history or a different scoring routine, says nothing about the model beside it. So every method in this chapter takes its cutoffs from `held_out_cutoffs()`, the seven daily cutoffs at 00:00 of the held-out week, reads its history through `history_before()`, which takes the eight weeks of hours before a cutoff, and is scored by the same code. These are the seven windows [Chapter 1](#score-on-a-held-out-week) used.

**Working rule:** Score every baseline on the same windows, histories and scoring code as the model it will be compared with, and count each baseline's missing forecasts before you read its score. A model that beats a baseline scored on different hours has not beaten it.

## Three baselines to beat {#three-baselines-to-beat}

Each baseline copies the history in a different way. Naive carries the last hour it has seen forward across the whole day. Seasonal naive copies the day before, hour for hour. Average of weeks takes the mean of that hour on that weekday over the four weeks before.

Every rule covers the 24 hours after a cutoff. Call the cutoff `T`, the hour at which the forecast is made, and let `k` run from 0 to 23 across the hours the forecast covers. The three rules are these:

- Naive repeats the last reading before the cutoff: `forecast(T + k) = y(T - 1)`.
- Seasonal naive repeats the reading one day earlier: `forecast(T + k) = y(T + k - 24)`.
- Average of weeks takes the mean of the same hour on the same weekday over the four weeks before: `forecast(T + k) = (y(T + k - 168) + y(T + k - 336) + y(T + k - 504) + y(T + k - 672)) / 4`.

The lags are in hours, so 168 is one week. Every reading a rule uses comes from before the cutoff: naive's is the hour before it, seasonal naive's are the 24 hours before it, and average of weeks reaches back one to four weeks. Take the window at 00:00 on Dec 25, 2014 as a worked case. Naive gives every hour of Dec 25 the reading at 23:00 on Dec 24, so the forecast is flat. Seasonal naive gives 00:00 on Dec 25 the reading at 00:00 on Dec 24, and so copies the whole previous day's shape. Average of weeks gives 00:00 on Dec 25 the mean of the 00:00 readings on Dec 18, Dec 11, Dec 4 and Nov 27. That mean smooths out one unusual day, but it also averages in any level change across the four weeks.

Each rule fails in a known way. Naive gives every hour of the day the same value, so it cannot follow the daily peak at all. Seasonal naive copies one day, so an unusual day is copied into the forecast for the next day, as the per-day errors later in this section show on the held-out week. Average of weeks needs four clean weeks, and a holiday in any of them enters the mean with the same weight as an ordinary day. A baseline's score therefore describes its rule on the windows it was run on, and these failures show up only on the windows where they happen.

Naive and seasonal naive come from statsforecast (2.1.1), whose classes `Naive` and `SeasonalNaive` are in `statsforecast/models.py`. The seasonal naive rule is the one [Hyndman and Athanasopoulos](https://otexts.com/fpp3/toolbox.html#sec-simple-methods) give for seasonal data: the forecast is the last observed value from the same season. Average of weeks is not a library method, so `examples/lib/baselines.py` computes it in pandas, where the rule can be read, and each window reads only its own history. A lag missing from that history raises an error instead of being skipped. One script scores all three rules through those seven windows, and its first lines count the windows, scored hours and missing forecasts, so you can check them before you read a score.

Example `naive-baselines-held-out-week` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Score the three baselines for the lead client over the held-out week, on the same seven windows. The point rests on the missing-forecasts line, which must be zero for the scores to compare, and on the three MAE lines."""
import pandas as pd

from lib.backtest import held_out_cutoffs, score_windows, window_mae
from lib.baselines import BASELINES, baseline_forecasts
from lib.electricity import load
from lib.settings import LEAD, show

cutoffs = held_out_cutoffs()                                 # seven daily cutoffs at 00:00; every baseline uses these same windows
df = load([LEAD])[["unique_id", "ds", "y"]]                  # unique_id, ds and y only: the baselines read no other column
forecasts = baseline_forecasts(df, cutoffs)                  # one row per forecast hour, one column per baseline; each window reads its own history
labels = ["naive", "seasonal naive", "average of weeks"]     # printed names, in the order of BASELINES, which the loops below zip with

show("windows", len(cutoffs))
show("scored hours", len(forecasts))
show("missing forecasts", int(forecasts[BASELINES].isna().sum().sum()))   # NaN marks a forecast that could not be made; the two sums count them over every column
show("mean demand", df[df["ds"].isin(forecasts["ds"])]["y"].mean())       # the mean over the scored hours only, not over the whole history
for model, label in zip(BASELINES, labels):
    show(f"{label} MAE", score_windows(df, forecasts, model)["mae"])      # pooled over every scored hour; the result is keyed by metric name
for model, label in zip(BASELINES, labels):
    per_window = window_mae(df, forecasts, model)                         # one MAE per window, keyed by the day its forecast hours fall on
    show(f"{label} worst window MAE", per_window.max())
    show(f"{label} worst window", per_window.idxmax().strftime("%b %d, %Y"))
    for day in ["2014-12-25", "2014-12-26"]:                              # Dec 25 is the holiday and Dec 26 the day after: the two windows the prose reads
        show(f"{label} MAE {pd.Timestamp(day):%b %d} window", per_window[pd.Timestamp(day)])   # windows are keyed by midnight of their day, so a Timestamp at 00:00 finds one
```

Output:

```text
windows: 7
scored hours: 168
missing forecasts: 0
mean demand: 46.62
naive MAE: 8.16
seasonal naive MAE: 8.36
average of weeks MAE: 8.96
naive worst window MAE: 11.01
naive worst window: Dec 30, 2014
naive MAE Dec 25 window: 5.84
naive MAE Dec 26 window: 10.36
seasonal naive worst window MAE: 12.16
seasonal naive worst window: Dec 26, 2014
seasonal naive MAE Dec 25 window: 9.41
seasonal naive MAE Dec 26 window: 12.16
average of weeks worst window MAE: 22.09
average of weeks worst window: Dec 25, 2014
average of weeks MAE Dec 25 window: 22.09
average of weeks MAE Dec 26 window: 12.33
```

The three scores are comparable only because they cover the same hours: 168 scored hours in 7 windows, one for each hour of the seven days, and 0 missing forecasts across the three rules. That count is the one the [working rule in the first section](#why-a-baseline-comes-first) asks for, and zero is the only value that lets the three be compared on the same hours.

Read each error below against the lead client's mean demand of 46.62 kWh. Each score below is a [MAE](#score-on-a-held-out-week), the mean absolute miss in kWh that [Chapter 1](#score-on-a-held-out-week) defines. The three baselines score:

- naive: 8.16 kWh
- seasonal naive: 8.36 kWh
- average of weeks: 8.96 kWh

The simplest rule has the lowest error on this week, and average of weeks, which averages four weeks, is the weakest. Nothing here explains that order, and this is one client and one week, so it is a result for this week only. Chapter 3 is where the book backtests seasonal naive across many cutoffs.

The totals hide where the errors sit. The week begins on Dec 25, 2014, a Portuguese national holiday, and its first window is where the rules part company. Naive has the lowest error on it, at 5.84 kWh, against 9.41 kWh for seasonal naive. Average of weeks is worse still, and its largest error of the week falls on that window, at 22.09 kWh, because its four lags are ordinary days and the holiday has no match among them.

The day after the holiday is where copying costs seasonal naive. Its largest error of the week falls on Dec 26, 2014, at 12.16 kWh, because that window copies the holiday's pattern. On that day naive is still the lowest, at 10.36 kWh.

We recommend reporting all three baselines every time. Use seasonal naive as the main reference for [MASE](#scale-errors-with-mase), because it carries the daily cycle that makes hourly demand predictable, and keep naive beside it, because on some weeks the flat rule is the stronger one.

**Working rule:** Start from naive and seasonal naive, both from the library, and add average of weeks when the series has a weekly rhythm worth testing. Check one forecast by hand against its formula before you rely on a baseline: pick one cutoff, compute one hour from the readings in its history, and compare the result with the forecast.

## Scale errors with MASE {#scale-errors-with-mase}

A kilowatt-hour error means different things for different clients. A miss of 8.36 kWh an hour is small for a client that uses 46.62 kWh an hour, and the same miss would be large for a hypothetical client that uses 4 kWh an hour. Scaling the error removes that difference. A scaled error divides each absolute error by a scale, the typical error of a simple method on the same client's own history. MASE, the mean absolute scaled error, is the mean of the scaled errors. [Hyndman and Koehler](https://robjhyndman.com/papers/mase.pdf) introduced it in a 2006 paper in the *International Journal of Forecasting*. The link is the authors' preprint, dated 2005. They noted that the scaled error does not depend on the units of the data, so the same number can be read across clients of very different size. This chapter scores one client, so that comparison across clients is the reason for the scale, not a result shown here.

The 2006 paper scales by the one-step naive error: the mean absolute difference between each reading and the one before it. That is the natural scale for non-seasonal series. For hourly demand, this book uses the seasonal form that [Hyndman and Athanasopoulos](https://otexts.com/fpp3/toolbox.html#sec-accuracy) give for seasonal data, because the comparison a reader wants is with seasonal naive: under the seasonal scale, the denominator is seasonal naive's error on the client's own history. A MASE below 1 then says the forecast's error on the scored hours is smaller than seasonal naive's error on that history. A one-step scale would compare it with a method this book does not use for hourly demand.

The two formulas are these:

- `scale = mean of |y(t) - y(t - 24)|` over every reading t before the first test day, with the pairs that touch a masked clock-change hour left out
- `MASE = mean of |actual(t) - forecast(t)| / scale` over the scored hours

The scale reads every reading before the first test day, 17352 hours, about two years. That is longer than each window's eight-week history, and a longer history gives a steadier estimate of the client's usual swing. One scale for the week also means every window is read against the same number. The function `seasonal_scale` in `examples/lib/baselines.py` computes it for one client or for many, and returns one value per client. To see whether the cut matters, score the three baselines on the held-out week twice: with the scale from history, and with the scale from the whole series, both with the clock-change readings masked.

Example `mase-scale-leak-check` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Score the lead client's held-out week in MASE with a scale from history and with one from the whole series, both with the clock-change readings masked. The point rests on the two last-reading lines, which show where each scale stops, and on the MASE ratio lines."""
import pandas as pd

from lib.backtest import held_out_cutoffs
from lib.baselines import BASELINES, baseline_forecasts, mase_scores, scale_history, seasonal_scale
from lib.electricity import load
from lib.quirks import mask_clock_changes
from lib.settings import HELD_OUT_WEEK, LEAD, show

LABELS = ["naive", "seasonal naive", "average of weeks"]    # printed names, in the order of BASELINES
first_day = pd.Timestamp(HELD_OUT_WEEK[0])                   # the first test day: the history scale may read only readings before it

full = load([LEAD])                                          # every reading of the lead client, with the quirk column that marks clock-change hours
df = full[["unique_id", "ds", "y"]]                          # the three columns the baselines and mase_scores read; quirk is for masking only
forecasts = baseline_forecasts(df, held_out_cutoffs())       # the same windows and history rule as naive-baselines-held-out-week.py
history = scale_history(full, first_day)                     # keeps only rows strictly before the first test day, so no test hour reaches this scale
masked_history = mask_clock_changes(history)                 # clock-change hours set to NaN, so the pairs that touch them drop out of the mean
masked_whole = mask_clock_changes(full)                      # the same masking on the whole series, so the test week is the only difference
test_rows = len(full[full["ds"] >= first_day])               # readings from the first test day on: the test week's rows

show("scale rows from history", len(history))
show("last scale reading from history", history["ds"].max().strftime("%b %d, %Y %H:%M"))   # this must be the hour before the first test day; the line is the check
show("scale rows from whole series", len(full))
show("last reading of whole series", full["ds"].max().strftime("%b %d, %Y %H:%M"))
show("test week share of whole-series rows in percent", 100 * test_rows / len(full), digits=3)
show("clock-change rows in scale history", int(history["quirk"].ne("normal").sum()))   # every quirk but normal; for the lead client these are the clock-change hours, as it has no late start
show("test week own mean day-to-day difference", seasonal_scale(full[full["ds"] >= first_day - pd.Timedelta(hours=24)]).loc[LEAD], digits=4)   # starts 24 hours early, so its first pair is the first test day against the day before
show("scale from history as recorded", seasonal_scale(history).loc[LEAD], digits=4)          # seasonal_scale returns one value per client, so .loc picks the lead client; the clock-change pairs are kept here
show("scale from history clock changes masked", seasonal_scale(masked_history).loc[LEAD], digits=4)   # pairs with a NaN are left out of the mean
show("scale from whole series clock changes masked", seasonal_scale(masked_whole).loc[LEAD], digits=4)

hist = mase_scores(df, forecasts, BASELINES, masked_history).loc[LEAD]   # the scale comes from the rows passed as train, not from df; that choice is what this example tests
whole = mase_scores(df, forecasts, BASELINES, masked_whole).loc[LEAD]
for model, label in zip(BASELINES, LABELS):
    show(f"{label} MASE from history scale", hist[model])
    show(f"{label} MASE from whole-series scale", whole[model])
    show(f"{label} MASE ratio whole-series over history", whole[model] / hist[model], digits=4)   # both MASEs share the same errors, so this ratio is the history scale over the whole-series scale
show("seasonal naive MASE from history scale at four decimals", hist["SeasonalNaive"], digits=4)   # two decimals hide the leak, so the last two lines print four
show("seasonal naive MASE from whole-series scale at four decimals", whole["SeasonalNaive"], digits=4)
```

Output:

```text
scale rows from history: 17352
last scale reading from history: Dec 24, 2014 23:00
scale rows from whole series: 17520
last reading of whole series: Dec 31, 2014 23:00
test week share of whole-series rows in percent: 0.959
clock-change rows in scale history: 4
test week own mean day-to-day difference: 8.3607
scale from history as recorded: 7.1121
scale from history clock changes masked: 7.1005
scale from whole series clock changes masked: 7.1126
naive MASE from history scale: 1.15
naive MASE from whole-series scale: 1.15
naive MASE ratio whole-series over history: 0.9983
seasonal naive MASE from history scale: 1.18
seasonal naive MASE from whole-series scale: 1.18
seasonal naive MASE ratio whole-series over history: 0.9983
average of weeks MASE from history scale: 1.26
average of weeks MASE from whole-series scale: 1.26
average of weeks MASE ratio whole-series over history: 0.9983
seasonal naive MASE from history scale at four decimals: 1.1775
seasonal naive MASE from whole-series scale at four decimals: 1.1755
```

The scale from history is 7.1005 kWh, the average gap between each hour and the same hour the day before, and it reads against the mean demand the way an MAE does. A scale that is a large share of the mean says the client's own day-to-day swing is large, and every error for that client is read against that swing. Dividing seasonal naive's MAE of 8.36 kWh by the scale gives its MASE, 1.18. utilsforecast's `mase`, reached through the helper `mase_scores`, computes this MASE by dividing the same MAE by the same seasonal scale (utilsforecast 0.2.17, `utilsforecast/losses.py`, function `mase`), so the division by hand gives the same figure.

utilsforecast (0.2.17) computes MASE with its `mase` function in `utilsforecast/losses.py`. The function takes its scale from the `train_df` it is given, and it shifts that frame by rows, not by time, so a missing hour moves every later lag and nothing reports it. It also needs sorted input. Its time filter runs only when the scored frame has a `cutoff` column (function `_create_train_with_cutoffs`). So the library does not decide which readings the scale may read. The caller does, and the next section is about that choice.

**Working rule:** Report MASE beside MAE whenever the clients you compare differ in size, and say in the same sentence which history the scale comes from. Read MASE as a ratio of two mean errors, one from the scored hours and one from the history. It is not a percentage gain over seasonal naive, even when the MASE is well below 1.

## Take the MASE scale from history {#take-the-mase-scale-from-history}

A scale describes the history before the first test day, not the series as a whole. If the scale reads the whole series, it also reads the hours the forecast is scored on, so the held-out week enters the denominator of its own error. That is a [leak](#keep-the-future-out-of-history), and it raises no error. The MASE it produces looks like any other. The rule in this book is that a scale reads only the readings before the first test day of the week it scores, which for the held-out week is Dec 25, 2014.

A whole-series scale looks like the safer choice, because it uses more readings. It uses more readings from the wrong period. The extra readings are the ones the forecast is judged on, so the scale is no longer a measure of what was known at the cutoff. More data does not fix that. Only the cut does.

The script in [the previous section](#scale-errors-with-mase) prints the rows each scale reads, and those lines are the check. The history scale reads 17352 readings, and its last reading is Dec 24, 2014 23:00, the hour before the first test day. A later date there would mean test hours reached the scale.

The whole-series scale reads 17520 readings, through Dec 31, 2014 23:00, which covers the whole held-out week. Both scales have their clock-change readings masked, so the held-out week is the only difference between them.

The whole-series scale is the larger one, so every MASE computed with it is smaller. At two decimals both scales give seasonal naive 1.18, so the leak does not show. At four decimals it does, with seasonal naive's MASE at 1.1775 against 1.1755, and the factor below is the lesson, not the rounding.

The scales are 7.1005 kWh from history and 7.1126 kWh from the whole series, and the same factor, 0.9983, shrinks each of the three rules.

The leak is small on this week because the held-out week is a small share of the series. It makes up 0.959 percent of the whole-series readings, so the whole-series scale is roughly the history scale moved toward the held-out week's own swing by that share. The held-out week's own mean day-to-day difference is 8.3607 kWh, against 7.1005 kWh for the history, so the week swings more than the history does. The share times that gap is the leak, and the ratio above is its size. The leak grows in two ways. It grows when more of the scale's rows are test hours, as in a test period much longer than one week, such as the year of cutoffs in Chapter 3. It also grows when the held-out week's swing sits further from the history's. A quieter week would pull the whole-series scale down and make every MASE larger, so the same leak can flatter a model in one week and penalise it in another.

The clock changes are a second, smaller point, and this chapter handles them as Chapter 3 does. The history holds 4 clock-change readings: the two gap hours and the two repeat hours of the March and October changes in 2013 and 2014. The gap hours read zero, and the repeat hours hold the energy of two, as the [data card](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/data/README.md) describes; [Chapter 3](#the-october-hour-that-holds-two) shows the effect. Kept as recorded, those pairs give a scale of 7.1121 kWh. Masked, the scale is 7.1005 kWh, because the pairs that touch a masked reading drop out of the mean. A masked reading is neither zero nor a copy; it is missing. The chapter's scale masks them, and [Chapter 3 gives the reason](#meet-the-clock-changes), which sets them to missing in every training set and every score.

**Working rule:** Pass a scale only the readings before the first test day, mask its clock-change readings, and print the last reading it uses. Check that this reading is the hour before the first test day. A small difference between the two scales does not show the procedure is right. The row check does.

## Read a baseline's score {#read-a-baselines-score}

MASE compares a forecast with seasonal naive's error on the client's own history. Below 1, the forecast's error on the scored hours is smaller than the error seasonal naive made on that history; above 1, it is larger. [Hyndman and Koehler](https://robjhyndman.com/papers/mase.pdf) state this reading in their paper for the one-step scale, and the arithmetic is the same for the seasonal scale. The book gives no threshold for a good score. What counts as good depends on what a miss costs the supplier, which the supplier knows and this book does not.

The table puts the three baselines side by side, one row each. The MAE is in kWh, and the MASE uses the history scale with the clock changes masked. The leak from [the previous section](#take-the-mase-scale-from-history) does not show at two decimals, so the table has one MASE column.

| Baseline | MAE (kWh) | MASE |
| --- | --- | --- |
| Naive | 8.16 | 1.15 |
| Seasonal naive | 8.36 | 1.18 |
| Average of weeks | 8.96 | 1.26 |

All three MASE values are above 1, so every baseline missed by more than seasonal naive's typical error on the history. For seasonal naive itself, the week's day-to-day swing was larger than the history's. Its MAE equals the held-out week's own day-to-day difference, 8.3607 kWh, against 7.1005 kWh for the history. For naive and average of weeks, the scale compares them with seasonal naive, not with their own past, so this table does not say how those two rules did on the history. The week opens on a national holiday, and the [earlier section](#three-baselines-to-beat) shows the holiday window as the largest error for average of weeks. The book does not test the remaining days here.

The order is the same on both measures, naive first, seasonal naive second and average of weeks third. That agreement is not a finding. MASE divides every MAE by the same scale, so the order cannot differ between MAE and MASE for one client.

The table also raises the supplier's question: is the simplest rule good enough to buy on? That depends on what an hour of error costs, and the MASE does not price it. What the MASE gives the supplier is a comparison it can repeat. A later model can be scored with the same helpers, so its MASE sits beside these rows without a change of units.

Take the table in this order. First, confirm from the [earlier section](#three-baselines-to-beat) that every row scores the same hours, with no missing forecasts. Second, read the MASE column against 1. Third, compare each row with the others, not with a number from a textbook. A fitted model that is only as good as the best row has not shown much, and one that is better by less than the printed rounding has not shown anything at all.

**Working rule:** Read each baseline's MASE against 1 and against the other baselines, and do not name a winner on one week. A model counts as better only when it beats the best baseline on more than one set of windows, and when the margin is larger than the rounding of the printed scores.

## What goes wrong in production {#what-goes-wrong-ch2}

Score a baseline on other hours, or with another history, and the comparison stops meaning what it says. A team may score its model on a year of cutoffs and its baseline on one week, or give the model a longer history than the baseline. The comparison then measures the setup, and the printed numbers look like evidence. The check is the one this chapter uses. Every method takes its cutoffs from `held_out_cutoffs` and its history from `history_before`, and the count of missing forecasts is printed for each run.

[The leak in the scale](#take-the-mase-scale-from-history) leaves a MASE that looks like any other, and raises no error. In this chapter it multiplies every seasonal naive MASE by 0.9983, a change that two decimals hide. A larger leak in another week would look the same on the page. The check is the printed row count and the last reading the scale uses, both on one line of output that a reviewer can read without rerunning anything.

A gap in the seasonal lag gives a wrong scale that no output marks. With a missing hour, a lag of 24 rows reaches 25 hours back, so each pair for the next day compares the wrong hours. The library's `mase` has no gap check of its own. The helper `seasonal_scale` checks that each client's hours are unbroken, and `mase_scores` runs the same check on the scale's rows before the library reads them. The check raises an error rather than using an assertion, because Python removes assertions under `-O`. The scored hours are checked only for a reading and for duplicates. A production run should count the hours in the feed before it computes any scale, because a check inside the helper reports the problem after it has reached the scale.

The seasonal lag can also be chosen for the wrong cycle. A lag of 24 suits hourly demand with a daily cycle. Daily data has a weekly cycle, and hourly data with a weekly rhythm may need a lag of 168 instead. A scale computed with the wrong season is still a number, so nothing in the output shows the mistake. The check is to name the season beside the scale in the prose and to confirm it against the data's cycle before the scale is used.

Picking a setting on the held-out week spoils the comparison. Average of weeks has a choice in it: how many weeks to average. If that number is picked because it gives the best score on the held-out week, the baseline has been fitted to the week it is judged on, and the comparison with the models is no longer fair. This book fixed four weeks before any score was printed, and this chapter keeps it for every window. A change to it belongs in the decision list at the end of the chapter, made before the scores are seen.

**Working rule:** Check a baseline's inputs, and its scale's, in this order: the same windows for every method, a scale that reads only earlier readings with its last reading printed, an unbroken hourly history for the seasonal lag, and a season that matches the data's cycle. Each check is one line of output or one count, and a failed check stops the comparison until it is fixed.

## Decisions this chapter made {#decisions-ch2}

Seven choices shaped this chapter, and the last is its hand-off to Chapter 3. Most are in `examples/lib/baselines.py`, the baseline helpers, so a fix there reaches every chapter that calls them. The history length and the horizon are the settings in `examples/lib/settings.py`, the same ones [Chapter 1](#decisions-ch1) used, so no baseline here reads a different history from the forecasts beside it.

- The MASE scale is seasonal, with a lag of 24 hours. The one-step scale of the 2006 paper is not used.
- The scale is one number per client for the held-out week, from every reading before Dec 25, 2014. It is longer than each window's history, as set out above. utilsforecast can take a scale per cutoff when the frame carries a cutoff column, but this chapter does not use that option. If you add one, know that its filter keeps `ds <= cutoff`, so the cutoff hour itself is included. A cutoff in this book is the first hour a forecast covers, and its reading is not history. A cutoff column built from this book's cutoffs must therefore be shifted back one hour first.
- Average of weeks uses the four weeks before each forecast day, is computed in pandas, and reads only its own eight-week history.
- Naive and seasonal naive are the statsforecast classes. Scores are pooled over every scored hour of the week and printed to two decimals.
- The scale masks its 4 clock-change readings, following Chapter 3's rule. The printed history scale is then 7.1005 kWh.
- The baselines draw no random numbers, so the examples set no seed.
- Chapter 3 keeps seasonal naive and these helpers and runs them over a year of Sunday cutoffs, with one scale per client taken from every reading before the first cutoff, the rule this chapter sets out. One week cannot give that test.

# Backtest without looking ahead

This chapter teaches you to score a day-ahead forecast over a year of cutoffs without letting the future into its history, to handle the two clock-change hours in the data, and to run a shift test that catches a feature which reads past its cutoff. You need what [Chapter 2](#take-the-mase-scale-from-history) gave you: the seasonal naive baseline, MAE, and a scale taken from history. The method stays the same. What changes is how many times you score it, and how carefully you cut the data each time.

## Why one test week misleads {#why-one-test-week-misleads}

So far the evidence is the held-out week of [Chapter 1](#score-on-a-held-out-week): seven day-ahead forecasts for one client, 168 scored hours in all. A sample that size can be lucky. A forecast can match a calm week and miss the next, and one week cannot tell the two apart. Before the supplier trusts a method with each day's purchase, it needs to know how a forecast does across many days, the unusual ones included.

A backtest answers that question with the data you already have. It makes a forecast at many past moments, each with only the readings available at that moment, and scores each one against the readings that followed. Each such moment is a [cutoff](#the-case-and-its-data). Each cutoff, with the hours it forecasts, is a [window](#the-case-and-its-data). The scores of all the windows together say how the method behaves across the year, not on one week.

This matters most for the clock changes. A year of cutoffs contains the March change, when the 01:00 hour does not occur on the clock, and the October change, when one 01:00 label holds two hours of energy. A single week, such as the last week of December, contains neither. A backtest that does not reach those dates cannot show what the method does on them, so the sample has to be chosen with them in mind.

**Working rule:** Prefer a year of cutoffs before you trust a difference between two methods. When cost forces fewer, say so beside each score, as Chapters 4 to 9 do. Make sure the sample reaches the odd days of the year. Report how many windows and how many scored hours stand behind each number. A figure from one week is one sample, and a reader should be able to see that.

## Roll the cutoff forward {#roll-the-cutoff-forward}

A window has three parts: the cutoff, the horizon and the history. The cutoff is the moment the forecast is made. The [horizon](#the-case-and-its-data) is how far ahead it reaches, which is 24 hours here, so the forecast covers the 24 hours that start at the cutoff. The [history](#why-a-baseline-comes-first) is the readings before the cutoff that the model may use, which the book sets at 1,344 hours, or eight weeks. Move the cutoff by a step and you have the next window. The step is one week for the Sunday sample, and one day for the daily samples in the two clock-change subsections, [The March hour that reads zero](#the-march-hour-that-reads-zero) and [The October hour that holds two](#the-october-hour-that-holds-two).

In words, each window's targets are the 24 hours from its cutoff, and its history is the readings from 1,344 hours before the cutoff up to, but not including, the cutoff. The next window moves both by one step. A reading can therefore be a target in one window and history in a later one. That is correct, because by the later cutoff the reading lies in the past, and it is the same reading in both places.

The Sunday sample runs seasonal naive over every Sunday of 2014, from Jan 5 to Dec 28, 2014. The EU puts every clock change on the last Sunday of March and of October ([Directive 2000/84/EC](https://eur-lex.europa.eu/eli/dir/2000/84/oj), Articles 2 and 3), and both changes of 2014, on Mar 30 and Oct 26, are in the sample. The sample uses [`SUBSET8`](#the-case-and-its-data), the book's eight-client subset. Here the year is scored on those eight clients, window by window:

Example `backtest-sundays-2014` (Python 3.13.16 · holidays 0.110.dev20261118063242 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Seasonal naive over the 52 Sunday cutoffs of 2014, for eight clients, with no clock-change rule: how much history each
window gets, then the year's scores. The prose's headline rests on the mean per-client MASE line."""
import pandas as pd
from holidays import country_holidays
from statsforecast import StatsForecast
from statsforecast.models import SeasonalNaive

from lib.backtest import forecast_windows, history_before
from lib.electricity import load
from lib.quirks import year_scores
from lib.scoring import book_scale, clients_left_out
from lib.settings import HISTORY_HOURS, HORIZON, SUBSET8, show

COLS = ["unique_id", "ds", "y"]
cutoffs = pd.date_range("2014-01-05", "2014-12-28", freq="W-SUN")  # 52 Sundays, both clock changes among them
df = load(SUBSET8)[COLS + ["quirk"]]
sf = StatsForecast(models=[SeasonalNaive(season_length=24)], freq="h", n_jobs=1)  # 24 hourly readings a day: each hour copies the same hour of the day before
forecasts = forecast_windows(sf, df[COLS], cutoffs, h=HORIZON)
scored = forecasts.merge(df, on=["unique_id", "ds"])  # attaches each forecast hour's reading and its quirk flag
scale = book_scale(df, cutoffs[0])  # the MASE scale: every reading before the first cutoff, from 2013 on
sizes = pd.concat([history_before(df, c).groupby("unique_id").size() for c in cutoffs])  # history rows per client in each window
assert sizes.min() == sizes.max() == HISTORY_HOURS  # every window's history is the same length, the prose's claim
holidays = country_holidays("PT", years=[2014])

show("windows", len(cutoffs))
show("history hours per series, min over windows", sizes.min())
show("history hours per series, max over windows", sizes.max())
show("Sunday windows on a Portuguese holiday", sum(c.date() in holidays for c in cutoffs))
show("client left out of MASE", ", ".join(clients_left_out(df, cutoffs[0])))
for label, value in year_scores(scored, scale).items():  # pooled MAE and RMSE, per-client MASE, the largest client's share
    show(label, value)
```

Output:

```text
windows: 52
history hours per series, min over windows: 1344
history hours per series, max over windows: 1344
Sunday windows on a Portuguese holiday: 1
client left out of MASE: MT_115
scored hours: 9936
scored hours before a first reading: 48
MAE: 382.02
RMSE: 1369.57
largest client by absolute error: MT_370
largest client's share of absolute error, percent: 88.69
clients in MASE: 7
mean MASE: 0.97
lowest client MASE: 0.77
highest client MASE: 1.26
```

Every window's history holds the same 1344 hours for each client, the eight weeks the book specifies, and the example asserts that the length is the same in every window. The 52 windows score 9936 hours: eight clients, 24 hours each, times the windows, less the hours of one late starter before its first reading, which the next paragraphs explain.

The headline is the [mean per-client MASE](#measure-what-each-feature-adds), not the pooled error. The MASE scale is one number per client, and `scoring.book_scale` computes it for the whole book: every reading before the first cutoff, with the clock-change hours masked and each client's leading zeros removed. The first cutoff is Jan 5, 2014, so this sample's scale is read from Jan 1, 2013 to Jan 4, 2014. The mean of the per-client MASE values is 0.97, with the lowest at 0.77 and the highest at 1.26, over 7 clients.

The [pooled MAE](#measure-what-each-feature-adds), 382.02 kWh over the scored hours, compares methods on the same clients and windows, but it cannot compare clients with each other. Its largest client, MT_370, carries 88.69 percent of the absolute error, because that client uses thousands of kilowatt hours an hour, so its errors are in the thousands too. The RMSE is 1369.57 kWh.

Client `MT_115` has no reading before Jan 14, 2014. Until then the meter reports zeros, and the data card says those zeros are not demand. The code leaves the 48 hours before its first reading out of every score, which are the first two windows. Its history still holds those zeros, because the [start rule](#start-each-series-at-its-first-reading) for histories belongs to Chapter 4. Its MASE is left out for a different reason: its scale is read from the readings before Jan 5, 2014, and it has none. The output lists that client as left out of MASE, and the start rule is left to that chapter.

The statsforecast library has a function for this loop, `cross_validation`, and it is possible to call it without the window you wanted. At version 2.1.1, the training slice of each window starts at the first reading of the series unless you pass `input_size`. The history then grows from one window to the next, and the eight-week rule is not in force. [Hyndman and Athanasopoulos](https://otexts.com/fpp3/toolbox.html#sec-accuracy) state the principle that the training set consists only of observations before the test set (*Forecasting: Principles and Practice*, 3rd ed., chapter 5). The code behind `cross_validation` decides the length of that set (statsforecast 2.1.1, `statsforecast/core.py`). Its cutoff is the last hour a window trains on, where this chapter's cutoff is the first hour it forecasts, so the two names differ by one hour. This chapter builds each window's history itself, from the cutoff backwards. That gives the same 1,344 hours at every window and makes the rule visible in the code.

**Working rule:** Set the history length explicitly, in hours, and check that it is the same for every window. When a library loops over windows for you, read how it sets the training slice before you trust its output, because its default may be an expanding window rather than the length you chose.

## Keep the future out of the history {#keep-the-future-out-of-history}

A backtest leaks when a window's model sees a reading from the hours it is asked to forecast. The reading can reach the model in several ways: through the history, through a feature, or through a scale computed from data that includes the target. [Hewamalage, Ackermann and Bergmeir](https://arxiv.org/abs/2203.10716) describe rolling-origin evaluation as prone to exactly this, because the test data of one window becomes training data of the next. In forecasting, they note, the leak is easier to make and harder to avoid than in other machine learning tasks (2022, section 3.4).

One rule closes the history path. Each window's history is the readings whose timestamps fall in the eight weeks before its cutoff, and nothing at or after the cutoff. In the book's code, `history_before` keeps the rows with `ds` from the cutoff minus 1,344 hours up to, but not including, the cutoff. The reading at the cutoff itself belongs to the target day, so the same comparison excludes it. Membership is decided by timestamps, not by row positions, so a missing row cannot shift a window by an hour. `history_before` also raises an error unless every client has exactly 1,344 rows in the window.

The window's own first hour is the cutoff, so a cut that is one hour too generous passes the first target reading into the history, and nothing in a forecast shows it. A row count can catch it, and the example below counts the rows.

Example `history-cut-check` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""One window's history, cut by its cutoff: its first and last timestamps, and what a cut that keeps the cutoff hour adds.
The prose's timestamps rest on the first two lines, and its off-by-one point on the row counts
with the cutoff hour in and out."""
import pandas as pd

from lib.backtest import history_before
from lib.electricity import load
from lib.settings import HISTORY_HOURS, LEAD, show

df = load([LEAD])  # the lead client's readings only: one series is enough to show the cut
cutoff = pd.Timestamp("2014-12-29")  # the window's cutoff at midnight, which is also its first forecast hour
hist = history_before(df, cutoff)  # the 1,344 hours before the cutoff; the cutoff hour itself is not in it
show("history rows", len(hist))
show("first history timestamp", hist["ds"].min().strftime("%Y-%m-%d %H:%M"))
show("last history timestamp", hist["ds"].max().strftime("%Y-%m-%d %H:%M"))

wrong = df[(df["ds"] >= cutoff - pd.Timedelta(hours=HISTORY_HOURS)) & (df["ds"] <= cutoff)]  # the off-by-one cut: <= keeps the cutoff hour
show("history rows with ds at or before the cutoff", len(wrong))
show("target-day readings in that cut", int((wrong["ds"] == cutoff).sum()))
```

Output:

```text
history rows: 1344
first history timestamp: 2014-11-03 00:00
last history timestamp: 2014-12-28 23:00
history rows with ds at or before the cutoff: 1345
target-day readings in that cut: 1
```

Keeping the cutoff hour adds one row, and the count shows it. The history for the cutoff of Dec 29, 2014 runs from 2014-11-03 00:00 to 2014-12-28 23:00 and holds 1344 rows. A cut with `ds <= cutoff` holds 1345 rows, because the target day's first reading, the cutoff hour, has entered the history. A count one row too high is the sign of that error.

The same care applies to the scale. `evaluate` in `utilsforecast` takes a training frame for its scale, and it cuts that frame by time only when the forecast frame carries a `cutoff` column. Then it keeps the training rows with `ds` at or before the cutoff, which includes the cutoff hour itself (utilsforecast 0.2.17, `evaluation.py`). This chapter does not use that option. Its scale comes from `scoring.book_scale`, which passes only the rows before the first cutoff, and the frame it scores has no `cutoff` column.

**Working rule:** Slice each window's history by its cutoff, never by the start of its test window, and pass a library only the rows it may use. Check the first and last timestamps of every history, and make sure the last one is an hour before the cutoff.

## Meet the clock changes {#meet-the-clock-changes}

The data has two hours that break the simple picture of one reading per clock hour. On the March change day, the local clock goes from 00:59 to 02:00, so the 01:00 hour does not exist on the clock. Mainland Portugal is on Greenwich Mean Time in winter, and the EU moves the clocks at 01:00 Greenwich Mean Time, which is 02:00 local time ([Directive 2000/84/EC](https://eur-lex.europa.eu/eli/dir/2000/84/oj), Article 2). On the October change day, the clock goes back from 02:00 to 01:00, so the 01:00 hour happens twice. The file has one row for 01:00 per day, flagged in the book's data as `clock_change_gap` or `clock_change_repeat`.

The book reads the file under one stated assumption, which every later chapter relies on. A reading's timestamp `ds` is the start of its hour in local clock time, and the index is one naive hourly index with no time zone. The March evidence supports this reading. The dataset's own note, in [Dataset 321 of the UCI Machine Learning Repository](https://doi.org/10.24432/C58C86), says the 01:00 values on the March change day are zero for all clients. The file mostly agrees, as the first subsection shows, and the second shows the two hours in the October label.

### The March hour that reads zero {#the-march-hour-that-reads-zero}

On Mar 30, 2014, the 01:00 label reads zero for 5 of the eight clients in the subset. The others read a small amount, not an hour of demand. The census covers all thirty clients.

Example `clock-change-census` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""The clock-change hours of all thirty clients, each against its neighbours. The point that the March hours read near zero
rests on the gap rows below 5 percent of their neighbours, with the zero count and the median ratio beside them."""
from lib.electricity import load
from lib.quirks import clock_ratios
from lib.settings import show

r = clock_ratios(load())  # each clock-change hour divided by the mean of the hour before and the hour after
gap = r[r["quirk"] == "clock_change_gap"]  # the file's flag: 01:00 on the March change day, an hour the clock skips
repeat = r[r["quirk"] == "clock_change_repeat"]  # the file's flag: 01:00 holds two hours on the October change day
show("gap rows", len(gap))
show("gap rows that read zero", int((gap["y"] == 0).sum()))
show("gap rows that read above zero", int((gap["y"] > 0).sum()))
show("gap rows below 5 percent of their neighbours", int((gap["ratio"] < 0.05).sum()))  # a NaN ratio, where both neighbours are zero, counts in neither line
show("gap rows at or above 5 percent of their neighbours", int((gap["ratio"] >= 0.05).sum()))
show("median gap ratio to neighbours", gap["ratio"].median(), digits=3)
show("repeat rows", len(repeat))
show("median repeat ratio to neighbours", repeat["ratio"].median(), digits=3)
```

Output:

```text
gap rows: 57
gap rows that read zero: 40
gap rows that read above zero: 17
gap rows below 5 percent of their neighbours: 55
gap rows at or above 5 percent of their neighbours: 2
median gap ratio to neighbours: 0.000
repeat rows: 58
median repeat ratio to neighbours: 1.948
```

Most flagged hours read near zero: 55 of the 57 flagged client-hours on the two March change days sit below 5 percent of their neighbouring hours, which is what this chapter means by near zero. The median ratio to the neighbours is 0.000.

The file mostly agrees with the dataset's note: 40 of the 57 read exactly zero, and 17 read above zero.

The data say zero or near zero, enough for the rule. The meter recorded nothing for an hour that did not occur on the clock, so the reading is not demand.

Seasonal naive copies the last day's values forward, so the forecast for 01:00 on Mar 30 is the demand of Mar 29 at 01:00. That is an ordinary hour, and where the reading is zero, the error is the whole forecast.

Make a guess before the next run. The cutoff of Mar 31 looks back over 24 hours, and those hours include the zero. How large is the error at 01:00 on Mar 31 compared with an ordinary hour? The run covers the 15 daily cutoffs from Mar 23 to Apr 6, 2014, first with no rule and then with the rule from the next section:

Example `clock-gap-daily-cutoffs` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Seasonal naive around the March clock change, for eight clients: the raw failure, then the rule. The prose's claims rest
on the gap-hour and next-day 1am lines, with and without the rule, the median client ratios, and the count of missing
forecasts without the fill."""
import pandas as pd

from lib.backtest import history_before
from lib.electricity import load
from lib.quirks import filled_history, mae_of, mask_clock_changes, median_client_ratio, seasonal_naive_at
from lib.settings import SUBSET8, show

cutoffs = pd.date_range("2014-03-23", "2014-04-06", freq="D")  # 15 daily cutoffs around Mar 30, 2014
next_day = pd.Timestamp("2014-03-31 01:00")  # 01:00 the day after the change: the forecast that copies the zero
df = load(SUBSET8)[["unique_id", "ds", "y", "quirk"]]
masked = mask_clock_changes(df)  # a copy with the gap and repeat hours set to NaN

raw = seasonal_naive_at(df, cutoffs, lambda c: history_before(df, c))  # no rule: the history keeps the zeros as the file has them
gap = raw["quirk"] == "clock_change_gap"  # the file's flag marks the gap hours
at_next_day = raw["ds"] == next_day
show("gap hours in actuals", int(gap.sum()))
show("gap hours that read zero", int((raw[gap]["y"] == 0).sum()))
show("gap-hour MAE with no rule", mae_of(raw[gap]))
show("other-hour MAE with no rule", mae_of(raw[~gap]))
# per client: the mean error at gap hours over the mean at other hours, then the median across clients
show("median client ratio, gap hour to other hours, no rule", median_client_ratio(raw, gap, ~gap))
show("next-day MAE at 1am with no rule", mae_of(raw[at_next_day]))
# a forecast of exactly zero is the March zero copied forward from the day before
show("next-day 1am forecasts that are zero, no rule", int((raw[at_next_day]["SeasonalNaive"] == 0).sum()))
show("median client ratio, next-day 1am to other hours, no rule", median_client_ratio(raw, at_next_day, ~gap))

no_fill = seasonal_naive_at(masked, cutoffs, lambda c: history_before(masked, c))  # masked history, no fill: a NaN copies forward and stays NaN
show("NaN forecasts without the fill", int(no_fill["SeasonalNaive"].isna().sum()))

fixed = seasonal_naive_at(masked, cutoffs, lambda c: filled_history(masked, c))  # the rule: masked hours filled from the history before each cutoff
show("next-day MAE at 1am with the rule", mae_of(fixed[fixed["ds"] == next_day]))
show("other-hour MAE with the rule", mae_of(fixed[fixed["quirk"] != "clock_change_gap"]))
```

Output:

```text
gap hours in actuals: 8
gap hours that read zero: 5
gap-hour MAE with no rule: 2322.69
other-hour MAE with no rule: 234.21
median client ratio, gap hour to other hours, no rule: 7.93
next-day MAE at 1am with no rule: 2393.01
next-day 1am forecasts that are zero, no rule: 5
median client ratio, next-day 1am to other hours, no rule: 9.92
NaN forecasts without the fill: 8
next-day MAE at 1am with the rule: 274.41
other-hour MAE with the rule: 228.31
```

For the median client, the error at 01:00 on Mar 31 is 9.92 times its error at other hours. The forecast for that hour copies the zero for 5 clients.

The gap hour itself fails as well. Its pooled error is 2322.69 kWh, against 234.21 kWh at other hours, and the median client's ratio is 7.93. Both pooled figures are set mostly by the largest client, while the per-client median counts each client once.

Pooled, the next day's error at 01:00 is 2393.01 kWh, about as much as the gap hour's. The method is wrong on the change hour, and wrong by about as much the day after, because it copied the zero.

**Working rule:** Treat a 01:00 reading of zero on a March clock-change day as a missing hour, not as zero demand. Keep it out of a model's history and out of the scores, and do not let a seasonal method copy it into the next day.

### The October hour that holds two {#the-october-hour-that-holds-two}

On Oct 26, 2014, the 01:00 label holds about two hours of energy. The census puts the median ratio of that hour to its neighbours at 1.948, over the 58 flagged client-hours in the file. The UCI note agrees: the 01:00 values aggregate two hours of consumption.

The failure is the reverse of the March one. On the change day, seasonal naive forecasts the 01:00 label from the day before, which is an ordinary hour, and the actual is about double that. The cutoff of Oct 27 copies the doubled value, so the forecast for 01:00 on Oct 27 is about twice an ordinary hour, and the actual is ordinary.

The same comparison runs over the 15 daily cutoffs from Oct 19 to Nov 2, 2014:

Example `clock-repeat-daily-cutoffs` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Seasonal naive around the October clock change, for eight clients: the raw failure, then the rule. The prose's claims rest
on the repeat-hour and next-day 1am lines, the median client ratios, and the count of missing forecasts without the fill."""
import pandas as pd

from lib.backtest import history_before
from lib.electricity import load
from lib.quirks import filled_history, mae_of, mask_clock_changes, median_client_ratio, seasonal_naive_at
from lib.settings import SUBSET8, show

cutoffs = pd.date_range("2014-10-19", "2014-11-02", freq="D")  # 15 daily cutoffs around Oct 26, 2014
next_day = pd.Timestamp("2014-10-27 01:00")  # 01:00 the day after the change: the forecast that copies the doubled value
df = load(SUBSET8)[["unique_id", "ds", "y", "quirk"]]
masked = mask_clock_changes(df)  # a copy with the gap and repeat hours set to NaN

raw = seasonal_naive_at(df, cutoffs, lambda c: history_before(df, c))  # no rule: the history keeps the doubled 01:00 value
repeat = raw["quirk"] == "clock_change_repeat"  # the file's flag: two hours of energy in one 01:00 label
at_next_day = raw["ds"] == next_day
show("repeat hours in actuals", int(repeat.sum()))
show("repeat-hour MAE with no rule", mae_of(raw[repeat]))
show("other-hour MAE with no rule", mae_of(raw[~repeat]))
# per client: the mean error at the repeat hour over the mean at other hours, then the median across clients
show("median client ratio, repeat hour to other hours, no rule", median_client_ratio(raw, repeat, ~repeat))
show("next-day MAE at 1am with no rule", mae_of(raw[at_next_day]))
show("median client ratio, next-day 1am to other hours, no rule", median_client_ratio(raw, at_next_day, ~repeat))

no_fill = seasonal_naive_at(masked, cutoffs, lambda c: history_before(masked, c))  # masked history, no fill: a NaN copies forward and stays NaN
show("NaN forecasts without the fill", int(no_fill["SeasonalNaive"].isna().sum()))

fixed = seasonal_naive_at(masked, cutoffs, lambda c: filled_history(masked, c))  # the rule: masked hours filled from the history before each cutoff
show("next-day MAE at 1am with the rule", mae_of(fixed[fixed["ds"] == next_day]))
show("other-hour MAE with the rule", mae_of(fixed[fixed["quirk"] != "clock_change_repeat"]))
```

Output:

```text
repeat hours in actuals: 8
repeat-hour MAE with no rule: 637.18
other-hour MAE with no rule: 397.96
median client ratio, repeat hour to other hours, no rule: 6.98
next-day MAE at 1am with no rule: 1524.26
median client ratio, next-day 1am to other hours, no rule: 7.53
NaN forecasts without the fill: 8
next-day MAE at 1am with the rule: 184.22
other-hour MAE with the rule: 394.23
```

The pooled figures understate the change-hour failure. On the change hour the forecast is an ordinary hour and the actual is doubled, so the pooled error is 637.18 kWh, against 397.96 kWh at other hours. The gap looks smaller than it is, because the largest client's errors are large at every hour. For the median client, the change-hour error is 6.98 times its error at other hours.

The doubled value does the most damage the next day. The forecast for 01:00 on Oct 27 copies it, so the pooled error there is 1524.26 kWh, the largest of the three, and the median client's ratio is 7.53. A reader who treats the raw October label as an ordinary hour will under-forecast one hour and over-forecast the next.

**Working rule:** Treat the October 01:00 label as two hours of energy in one slot, and never as an ordinary hour. Do not halve it to make it look like one, for the reason the next section gives.

## Mark clock-change hours as missing {#mark-clock-change-hours-as-missing}

The rule the book uses is a choice, and this section states it in one place so that every later chapter can call the same code. Both 01:00 hours, the March gap and the October repeat, are set to missing in the history and in the scores, so a masked hour is never scored. A model that takes missing values skips the hour. A model that cannot, such as seasonal naive, reads a filled value instead, because a missing value would copy into the next day's forecast. Either way the hour is never scored, and the forecast still produces a value for its label, because the output grid must stay complete. The fill is the mean of the hour before and the hour after, taken from the history before the cutoff only. It is a rule of thumb for an hour that sits between two ordinary ones, and it is labelled as one.

Halving the October label instead would give each real hour half of its reading, which assumes both hours used the same energy. The data card does not show that, so the book does not do it.

Without the fill, the forecast for 01:00 on the day after each change is missing for 8 of the eight clients in March, and for 8 in October.

With the fill, the next-day error at that hour falls from 2393.01 to 274.41 kWh in March, and from 1524.26 to 184.22 kWh in October, while the error at other hours moves little.

The year of Sunday cutoffs is a different test, because it applies the mask to the scores and the fill plays no part. Scoring the year twice, plain and masked, shows how much the two hours moved the headline.

Example `clock-masked-backtest` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""The Sunday backtest of 2014 for eight clients, scored twice: as the readings stand, and with the clock-change rule, which
leaves the masked hours out of the scores and fills the history. The prose's comparison rests on the masked-hour count,
the two pooled MAE lines, the two mean MASE lines and the two scored-hour lines."""
import pandas as pd

from lib.backtest import history_before
from lib.electricity import load
from lib.quirks import CLOCK_CHANGE_QUIRKS, filled_history, mask_clock_changes, seasonal_naive_at, year_scores
from lib.scoring import book_scale
from lib.settings import SUBSET8, show

cutoffs = pd.date_range("2014-01-05", "2014-12-28", freq="W-SUN")  # the same 52 Sunday cutoffs as the plain backtest
df = load(SUBSET8)[["unique_id", "ds", "y", "quirk"]]
masked = mask_clock_changes(df)  # a copy with the gap and repeat hours set to NaN; the rule's scores skip them
scale = book_scale(df, cutoffs[0])  # book_scale masks the clock-change hours itself, so both runs share this one scale

plain = seasonal_naive_at(df, cutoffs, lambda c: history_before(df, c))  # as the readings stand: the zeros and the doubled hour stay in
ruled = seasonal_naive_at(masked, cutoffs, lambda c: filled_history(masked, c))  # the rule: history filled from before each cutoff
show("masked hours in scored windows", int(plain["quirk"].isin(CLOCK_CHANGE_QUIRKS).sum()))  # counted in the plain run, where the flags are still present
for tag, scored in [("without the mask", plain), ("with the mask", ruled)]:
    result = year_scores(scored, scale)
    show(f"scored hours {tag}", result["scored hours"])
    show(f"MAE {tag}", result["MAE"])
    show(f"mean MASE {tag}", result["mean MASE"])
```

Output:

```text
masked hours in scored windows: 16
scored hours without the mask: 9936
MAE without the mask: 382.02
mean MASE without the mask: 0.97
scored hours with the mask: 9920
MAE with the mask: 380.25
mean MASE with the mask: 0.96
```

The mask takes 16 masked hours out of the scored windows, the March and October hours of the subset, and the pooled MAE moves only a little, from 382.02 kWh to 380.25 kWh.

The mean MASE goes from 0.97 to 0.96, over 9920 scored hours rather than 9936. The difference comes from the removed hours.

The Sunday sample cannot show the fill at all, because seasonal naive reads only the day before each cutoff, which is a Saturday for every Sunday cutoff, and no Saturday holds a clock change.

**Working rule:** Mark both clock-change hours missing in every history and every score, count the masked hours in your report, and fill them from the history before the cutoff only when the model cannot take missing values. If the count of masked hours changes from one run to the next, the sample or the file's flags have changed, and you should find out why before you rerun the scores.

## Check a backtest for leakage {#check-a-backtest-for-leakage}

Every window in a backtest should be built from data available at its cutoff, so the check for a leak is simple to state. A feature for a window must have the same value whether it is computed from the history cut at the cutoff or from the whole series. If the two values differ, the feature read a reading after the cutoff, and the backtest flatters the method. The shift test compares a feature computed from the history cut at the cutoff with the same feature computed from the whole series. It is cheap to run, and it catches a mistake that the backtest cannot catch on its own, because a leaking feature scores well by design.

The test has two halves, and each catches what the other misses. Take the lead client, `MT_014`, and three features over its hourly readings. The first is a trailing mean: the average of the 24 hours up to and including each hour, which reads nothing after its hour. The second is the same mean shifted one hour forward, so each value is the average of a window that ends one hour later, and it reads the first hour after the cutoff. The third is a lag of one hour, the most common day-ahead leak. The history half compares each feature on the rows of the eight weeks before each of the three evaluation cutoffs, Dec 29 to Dec 31, 2014. The target half compares the 24 hours that each cutoff forecasts, computed from the readings before the cutoff only. Both versions start from the same first reading, so the warm-up of a rolling mean matches on both sides. Computed from different starts, the warm-up and floating-point rounding would show as false differences, which is why the check compares with a tolerance:

Example `leakage-shift-check` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3), passing in Edition 2026.11.11:

```python
"""Shift test: a feature must read nothing from its cutoff on, on the history rows and on the target hours. The prose's
point rests on the counts of differing rows: an honest feature shows zero, and a leaking one shows its leak."""
from lib.electricity import load
from lib.leakage import history_test, lag, peeking_mean, target_test, trailing_mean
from lib.settings import EVAL_CUTOFFS, LEAD, show

y = load([LEAD]).set_index("ds")["y"]  # indexed by timestamp, so each feature can be cut at a cutoff by time

compared, honest = history_test(y, trailing_mean)  # rows compared, and the rows where cutting at the cutoff changes the feature
_, peeking = history_test(y, peeking_mean)  # the leaking feature: its last value before each cutoff reads the cutoff hour
_, lag1_history = history_test(y, lag(1))  # lag 1 reads only the past on history rows, so this half passes it
show("cutoffs", len(EVAL_CUTOFFS))
show("rows compared", compared)
show("rows differing for the trailing mean", honest)
show("rows differing for the peeking mean", peeking)
show("rows differing for lag 1", lag1_history)

targets, lag1_target = target_test(y, lag(1))  # target hours compared, and the hours that differ, over the 24 hours of each cutoff
_, lag24_target = target_test(y, lag(24))
show("target hours compared", targets)
show("target hours differing for lag 1", lag1_target)
show("target hours differing for lag 24", lag24_target)
```

Output:

```text
cutoffs: 3
rows compared: 4032
rows differing for the trailing mean: 0
rows differing for the peeking mean: 3
rows differing for lag 1: 0
target hours compared: 72
target hours differing for lag 1: 69
target hours differing for lag 24: 0
```

On the history rows, the shifted mean differs on 3 of the 4032 compared, one at each cutoff: the last hour before each cutoff, whose value depends on the reading at the cutoff itself. The trailing mean differs on 0 of them.

The lag of one hour is the case the history half cannot catch. It differs on 0 history rows, because every row it reads lies in the past, so the history half alone passes it.

On the 72 target hours it differs on 69, since for every target hour after the first, the reading it uses is in the day being forecast. The lag of 24 hours differs on 0 of them, because it reads the same hour of the day before, which was known at each cutoff.

The shift test checks features and nothing else. It does not check the scale, the fill or the model's own code, which is why the earlier sections set those rules separately. Run it on every feature a model reads, at several cutoffs spread through the year, and include a cutoff on a clock-change day if the feature reads the hour before it.

When a feature fails the test, do not fix the score. Move the feature back until it reads only what was known at its cutoff, then run the test again until the count is zero.

**Working rule:** Keep every lag at least the horizon, 24 hours, so that the reading a lag uses for any target hour was known at the cutoff. Before you trust a backtest with features, run both halves of the shift test on every feature, at several cutoffs spread through the year, and require zero differing rows for every feature the model reads. A feature that differs is reading the future, so fix or remove it before you score anything.

## What goes wrong in production {#what-goes-wrong-ch3}

A backtest is a sample of the past, and the sample is chosen. Every clock change falls on the last Sunday of March or October ([Directive 2000/84/EC](https://eur-lex.europa.eu/eli/dir/2000/84/oj), Articles 2 and 3), so a Sunday sample reaches every change. A sample of Mondays would reach the day after each change without reaching the change hour itself, so the choice of days changes what the backtest can see. Record which dates the sample covers, and rerun the clock check when the sample's dates change.

Calendar effects are a separate sample problem. Easter Sunday, Apr 20, 2014, is a Portuguese holiday, which the `holidays` package lists for Portugal, and the run counts it: 1 of 52 windows. The held-out week contains Dec 25, 2014, a national holiday in Portugal, and a week of ordinary days and one holiday gives a different score from a week of ordinary days alone. A backtest that uses holiday features must count the holidays in its windows, and say how many fall in the sample.

A pooled error hides the windows that went wrong. A year of windows can average to a respectable number while one month, or one client, carries the error. Break the scores down by month and by client before you report one figure, and look at the worst windows, not only the average.

The masked count follows the calendar flag, not the readings. It changes when the sample's dates or the file's flags change, and it cannot see drift in the readings. Count the masked hours on every run, and alarm on a change in the count, because it is the cheapest signal that the sample or the file has changed. Drift in the readings needs the census: the gap hours that read zero, and the repeat hours against their neighbours. Chapter 10 returns to this in the running log.

Finally, a backtest that passes the shift test can still leak through the scale or the fill, if either is taken from data after the cutoff. The checks in this chapter cover those paths in the examples and the rules, not in every library a reader might use. A reader who adds a library should read how it cuts its training data, as the [default of `cross_validation`](#roll-the-cutoff-forward) shows.

**Working rule:** Run the clock and shift checks again on every data refresh, and before any change to the sample's dates, the history length or the scale. Keep the masked-hour count, the census counts and the shift-test counts in the run log next to the scores, so that a later change in either can be traced to its cause.

## Decisions this chapter made {#decisions-ch3}

- The backtest uses the Sunday cutoffs of 2014. Every clock change falls on a Sunday, so the sample reaches each change, and both of 2014's changes are in it.
- The history of every window is the 1,344 hours before its cutoff, cut by the cutoff, and the length is set in the code, not left to a library default.
- The timestamp `ds` is the start of its hour in local clock time, with no time zone, and the book reads the file under that assumption.
- Both clock-change hours are masked in every history and every score. A model that cannot take missing values reads the mean of the neighbouring hours from history before the cutoff.
- Halving the October hour is rejected, because it assumes the two real hours used the same energy, which the data does not show.
- The headline is the mean per-client MASE, on the one scale of [Chapter 2](#take-the-mase-scale-from-history) through `scoring.book_scale`. Pooled MAE is the second line, with the largest client's share printed.
- Every figure the prose quotes stays bound to a labelled line of its example's output, as [Chapter 1](#decisions-ch1) set out, so a library update changes the chapter's figure and nobody retypes it.
- A client with no reading before the first cutoff is left out of MASE. A late starter's hours before its first reading are left out of every score. Chapter 4 owns the start rule for histories.
- The shift test runs on every feature, both halves. Every lag is at least the horizon, and a feature that differs at any cutoff is not used.

# Fit a statistical model for each client

Here you fit AutoETS and a fixed seasonal ARIMA for each client over the same backtest, score both against seasonal naive, and learn to start a late-starting client at its first reading. The chapter assumes the baselines and backtests of the two chapters before it.

Each of the supplier's clients needs its own day-ahead forecast, and [Chapter 2](#beat-the-naive-baselines)'s seasonal naive gives every one of them the same answer: the same hour yesterday. That rule keeps no estimate of a client's level or its trend, and the clients differ in both. So this chapter fits a statistical model for each client, and asks whether the extra work pays.

## Why a statistical model comes next {#why-a-statistical-model-comes-next}

[Chapter 2](#beat-the-naive-baselines) set the bar. [Seasonal naive](#forecast-with-seasonal-naive) forecasts each hour as the same hour yesterday, and the scaled error, MASE, compares clients with different loads on one scale. [Chapter 3](#backtest-without-looking-ahead) backtested that forecast over many cutoffs. A [cutoff](#the-case-and-its-data) is the time a forecast is made, and its history ends there. A [window](#the-case-and-its-data) is one cutoff and the 24 hours after it, which is the horizon. Each window's history ends at its cutoff, so no hour is forecast with data from its own day.

A statistical model adds to that loop. A series has a state. The level of a series is its typical reading, the trend is the slow drift of the level, the season is the daily shape of the load, and what none of these explain is error. A statistical model holds an estimate of each part of the state of a series, updates the estimates as each reading arrives, and projects them over the horizon. Seasonal naive has no such state. It copies yesterday, so it gets a client's level only as well as yesterday did.

A per-client model is one fit for each client, on that client's history alone, so the cost is one fit per client per window. The subset this chapter uses, [`SUBSET8`](#the-case-and-its-data), has eight clients. Over the book's three standard cutoffs, Dec 29 to Dec 31, 2014, the last three days of the held-out week and the windows every later chapter reports, that is 24 fits for each model. The code runs the fits in one worker (`n_jobs=1`), which keeps the output identical from run to run. Every model in this chapter gets the same three cutoffs and the same 1,344 hours of history before each one. The cutoffs fall in the held-out week, and the subset holds the one late starter a later section needs.

**Working rule:** Fit one model per client only after a baseline and a backtest are in place. Count the fits before you choose the number of windows: clients, times windows, times refits if you run the model every day. Use the same cutoffs and history for every model in a comparison. Otherwise the comparison measures the windows, not the models.

## Fit exponential smoothing {#fit-exponential-smoothing}

Seasonal naive keeps only yesterday, so the first candidate keeps a level instead. Exponential smoothing moves its level toward each new reading. The simplest form updates the level as `level_t = alpha * y_t + (1 - alpha) * level_(t-1)`. The constant alpha, between 0 and 1, sets how fast the level follows the data. The forecast for every hour of the horizon is the last level. Recent readings count for more than older ones, and the weights fall off with age.

ETS, for error, trend and season, extends the simplest form of exponential smoothing. Each part is chosen from a short list. The trend can be none, additive or damped. The error can be additive or multiplicative, and the season can be none, additive or multiplicative. A name such as ETS(A,N,A) reads in that order: additive error, no trend, additive season. ETS(M,Ad,M) has a multiplicative error, a damped trend and a multiplicative season.

[AutoETS](#forecast-with-a-statistical-model) chooses the ETS form for you. Its default criterion is [AICc](#forecast-with-a-statistical-model), the corrected Akaike information criterion, which charges a model for each parameter and corrects for small samples, as the `AutoETS` docstring in statsforecast 2.1.1 (`statsforecast/models.py`) says. Two of its defaults matter for hourly demand. The season length defaults to 1, which means no season at all, and the model gives no warning when you leave it that way. The `model` argument defaults to `"ZZZ"`, which lets the model choose all three parts. For hourly demand with a daily pattern, the call sets `season_length=24`.

Which form does AutoETS pick for each client, and do its forecasts beat seasonal naive? The lines it prints answer both.

Example `autoets-subset-backtest` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Backtest AutoETS and seasonal naive for the eight clients of SUBSET8 over the three standard cutoffs, and score both.
The fits lines count the forms AutoETS chose, and the MASE and coverage lines are the scores the comparison table quotes."""
from collections import Counter, defaultdict

import numpy as np
import pandas as pd
from statsforecast import StatsForecast
from statsforecast.models import AutoETS, SeasonalNaive

from lib.scoring import book_scale
from lib.electricity import load
from lib.quirks import filled_history, mask_clock_changes
from lib.settings import EVAL_CUTOFFS, HORIZON, SEED, SUBSET8, show
from lib.subsets import score_clients, show_client_lines
np.random.seed(SEED)                                        # the seed from lib/settings.py, set before any fit
masked = mask_clock_changes(load(SUBSET8))                  # clock-change hours become NaN, so they are never scored; filled_history fills them from their neighbours before each fit
scale = book_scale(load(SUBSET8), EVAL_CUTOFFS[0])        # MASE scale: rows before the first cutoff, from each client's first reading
# season_length=24 for hourly demand; the default of 1 would mean no season at all. n_jobs=1 keeps the output the same from run to run.
sf = StatsForecast(models=[AutoETS(season_length=24), SeasonalNaive(season_length=24)], freq="h", n_jobs=1)
parts, forms, by_client = [], Counter(), defaultdict(list)
for c in pd.to_datetime(EVAL_CUTOFFS):
    sf.fit(df=filled_history(masked, c)[["unique_id", "ds", "y"]])   # the eight weeks before c, clock hours filled from them
    part = sf.predict(h=HORIZON, level=[95])                          # level=[95] asks for the 95% interval the coverage line scores
    # the first forecast hour must be the cutoff itself, or the window is misaligned
    if part["ds"].min() != c:
        raise ValueError(f"forecast for {c:%Y-%m-%d} does not start at the cutoff")
    parts.append(part)
    for i, uid in enumerate(sf.uids):
        form = sf.fitted_[i, 0].model_["method"]                    # model 0 is AutoETS; its method name is the form, such as ETS(A,N,A)
        forms[form] += 1
        by_client[uid].append(form)
forecasts = pd.concat(parts, ignore_index=True)
summary, per_client = score_clients(forecasts, masked, ["AutoETS", "SeasonalNaive"], scale, interval_model="AutoETS")
show("horizon hours", HORIZON)
show("windows", len(parts))
for form, n in sorted(forms.items()):
    show(f"fits that chose {form}", n)
# the form reads ETS(error,trend,season): a multiplicative part is an M in any of the three slots
show("fits with a multiplicative part", sum(n for form, n in forms.items() if "M" in form[4:-1].split(",")))
for label, value in summary.items():
    show(label, value, digits=3 if label.startswith("coverage") else 2)   # coverage is a share, to three decimals; the rest are kWh or ratios
show("clients where AutoETS has lower MASE", int((per_client["AutoETS"] < per_client["SeasonalNaive"]).sum()))
show_client_lines(by_client, per_client, ["AutoETS", "SeasonalNaive"])
```

Output:

```text
horizon hours: 24
windows: 3
fits that chose ETS(A,Ad,A): 1
fits that chose ETS(A,N,A): 8
fits that chose ETS(A,N,N): 9
fits that chose ETS(M,Ad,M): 2
fits that chose ETS(M,N,A): 3
fits that chose ETS(M,N,M): 1
fits with a multiplicative part: 6
clients: 8
scored hours per client: 72
MAE AutoETS: 362.11
RMSE AutoETS: 537.20
MASE AutoETS: 1.30
MAE SeasonalNaive: 423.68
RMSE SeasonalNaive: 591.25
MASE SeasonalNaive: 1.16
coverage 95 AutoETS: 0.910
clients where AutoETS has lower MASE: 4
forms MT_014, one per cutoff: ETS(A,N,N) ETS(A,N,N) ETS(A,N,N)
MASE MT_014 AutoETS: 1.18
MASE MT_014 SeasonalNaive: 0.97
forms MT_064, one per cutoff: ETS(A,N,N) ETS(A,N,N) ETS(A,N,N)
MASE MT_064 AutoETS: 2.32
MASE MT_064 SeasonalNaive: 1.08
forms MT_115, one per cutoff: ETS(M,N,A) ETS(M,N,A) ETS(M,N,A)
MASE MT_115 AutoETS: 0.95
MASE MT_115 SeasonalNaive: 1.31
forms MT_166, one per cutoff: ETS(A,Ad,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_166 AutoETS: 1.18
MASE MT_166 SeasonalNaive: 1.29
forms MT_217, one per cutoff: ETS(M,N,M) ETS(M,Ad,M) ETS(M,Ad,M)
MASE MT_217 AutoETS: 1.76
MASE MT_217 SeasonalNaive: 1.25
forms MT_268, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_268 AutoETS: 0.94
MASE MT_268 SeasonalNaive: 0.89
forms MT_319, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_319 AutoETS: 1.11
MASE MT_319 SeasonalNaive: 1.26
forms MT_370, one per cutoff: ETS(A,N,N) ETS(A,N,N) ETS(A,N,N)
MASE MT_370 AutoETS: 0.96
MASE MT_370 SeasonalNaive: 1.25
```

The chosen form changes from client to client, and for some clients from one cutoff to the next. Across the 24 fits, AutoETS chose ETS(A,N,N), with no season, for 9 of them, and ETS(A,N,A), with an additive season, for 8. ETS(A,Ad,A), a damped trend with an additive season, came up in 1 of the fits. The other fits used a multiplicative part, in the error, the season or both. The lines under the totals give each client's three forms. Three clients, `MT_014`, `MT_064` and `MT_370`, take the flat form at all three cutoffs. `MT_014` is the lead client, the one [Chapter 1](#forecast-tomorrows-demand) met.

A flat form winning looks odd when the seasonal form contains it. ETS(A,N,N) is ETS(A,N,A) with the seasonal part switched off, so the best seasonal fit cannot have a lower likelihood than the flat fit on the same history. Fitting both forms on each of the 24 windows shows whether that happens.

Example `flat-form-check` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""For each subset window, fit the flat form ETS(A,N,N) and the additive-season form ETS(A,N,A) on the same history, and count where the seasonal fit has the lower likelihood.
The two counts printed last are the ones the prose quotes."""
import numpy as np
import pandas as pd
from statsforecast import StatsForecast
from statsforecast.models import AutoETS

from lib.electricity import load
from lib.quirks import filled_history, mask_clock_changes
from lib.settings import EVAL_CUTOFFS, SEED, SUBSET8, show

np.random.seed(SEED)                                        # the seed from lib/settings.py, set before any fit
masked = mask_clock_changes(load(SUBSET8))
windows = seasonal_lower_loglik = seasonal_higher_aicc = 0
for c in pd.to_datetime(EVAL_CUTOFFS):
    hist = filled_history(masked, c)[["unique_id", "ds", "y"]]
    # ANN is additive error, no trend, no season: the flat form. ANA is the same with an additive season.
    sf = StatsForecast(models=[AutoETS(model="ANN", season_length=24, alias="flat"),
                               AutoETS(model="ANA", season_length=24, alias="seasonal")], freq="h", n_jobs=1)
    sf.fit(df=hist)
    for i in range(len(sf.uids)):
        flat = sf.fitted_[i, 0].model_                          # models are fitted in the order listed: 0 is the flat fit
        seasonal = sf.fitted_[i, 1].model_                      # 1 is the seasonal fit
        windows += 1
        # ANA contains ANN, so a lower seasonal likelihood points to a fit that stopped short of its best
        seasonal_lower_loglik += int(seasonal["loglik"] < flat["loglik"])
        # AICc charges each parameter, so this count can differ from the likelihood count
        seasonal_higher_aicc += int(seasonal["aicc"] > flat["aicc"])
show("windows", windows)
show("seasonal fit has the lower log-likelihood", seasonal_lower_loglik)
show("seasonal fit has the higher AICc", seasonal_higher_aicc)
```

Output:

```text
windows: 24
seasonal fit has the lower log-likelihood: 9
seasonal fit has the higher AICc: 9
```

The seasonal fit loses on likelihood, not only on AICc. Its log-likelihood is lower in 9 of the 24 windows, and its AICc is higher in 9. Those are the same nine windows in which the default run chose the flat form. AICc charges a model for each parameter, and the seasonal form has 24 more. That charge adds to the gap; it does not cause it, because the seasonal fit's likelihood is already lower before any charge. The seasonal fit most likely stopped short of its best value, and the flat form won by default. The book does not chase the optimiser; it tests the choice with a forced season.

[Chapter 1](#forecast-with-a-statistical-model) forced an additive season on the lead client, with `model="ZZA"`, and seasonal naive still had the lower error there. Does that hold across all eight? The forced run covers the same eight clients, beside the default.

Example `autoets-forced-season-backtest` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Backtest AutoETS with the season forced to additive (model ZZA) and seasonal naive, for the eight clients of SUBSET8 over the three standard cutoffs.
The ZZA lines are the forced-season scores that the comparison table and the forced-season paragraph quote."""
from collections import Counter, defaultdict

import numpy as np
import pandas as pd
from statsforecast import StatsForecast
from statsforecast.models import AutoETS, SeasonalNaive

from lib.scoring import book_scale
from lib.electricity import load
from lib.quirks import filled_history, mask_clock_changes
from lib.settings import EVAL_CUTOFFS, HORIZON, SEED, SUBSET8, show
from lib.subsets import score_clients, show_client_lines
np.random.seed(SEED)                                        # the seed from lib/settings.py, set before any fit
masked = mask_clock_changes(load(SUBSET8))                  # clock-change hours become NaN, so they are never scored; filled_history fills them from their neighbours before each fit
scale = book_scale(load(SUBSET8), EVAL_CUTOFFS[0])        # MASE scale: rows before the first cutoff, from each client's first reading
# ZZA: Z lets the model choose the error and the trend, A forces an additive season. alias names the output columns.
sf = StatsForecast(models=[AutoETS(model="ZZA", season_length=24, alias="ZZA"), SeasonalNaive(season_length=24)],
                   freq="h", n_jobs=1)
parts, forms, by_client = [], Counter(), defaultdict(list)
for c in pd.to_datetime(EVAL_CUTOFFS):
    sf.fit(df=filled_history(masked, c)[["unique_id", "ds", "y"]])   # the eight weeks before c, clock hours filled from them
    part = sf.predict(h=HORIZON, level=[95])                          # level=[95] asks for the 95% interval the coverage line scores
    # the first forecast hour must be the cutoff itself, or the window is misaligned
    if part["ds"].min() != c:
        raise ValueError(f"forecast for {c:%Y-%m-%d} does not start at the cutoff")
    parts.append(part)
    for i, uid in enumerate(sf.uids):
        form = sf.fitted_[i, 0].model_["method"]                    # model 0 is the forced AutoETS; its method name is the form
        forms[form] += 1
        by_client[uid].append(form)
forecasts = pd.concat(parts, ignore_index=True)
summary, per_client = score_clients(forecasts, masked, ["ZZA", "SeasonalNaive"], scale, interval_model="ZZA")
show("horizon hours", HORIZON)
show("windows", len(parts))
for form, n in sorted(forms.items()):
    show(f"fits that chose {form}", n)
for label, value in summary.items():
    show(label, value, digits=3 if label.startswith("coverage") else 2)
show("clients where ZZA has lower MASE", int((per_client["ZZA"] < per_client["SeasonalNaive"]).sum()))
show_client_lines(by_client, per_client, ["ZZA", "SeasonalNaive"])
```

Output:

```text
horizon hours: 24
windows: 3
fits that chose ETS(A,Ad,A): 1
fits that chose ETS(A,N,A): 17
fits that chose ETS(M,N,A): 6
clients: 8
scored hours per client: 72
MAE ZZA: 365.24
RMSE ZZA: 553.11
MASE ZZA: 1.17
MAE SeasonalNaive: 423.68
RMSE SeasonalNaive: 591.25
MASE SeasonalNaive: 1.16
coverage 95 ZZA: 0.925
clients where ZZA has lower MASE: 4
forms MT_014, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_014 ZZA: 1.08
MASE MT_014 SeasonalNaive: 0.97
forms MT_064, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_064 ZZA: 1.27
MASE MT_064 SeasonalNaive: 1.08
forms MT_115, one per cutoff: ETS(M,N,A) ETS(M,N,A) ETS(M,N,A)
MASE MT_115 ZZA: 0.95
MASE MT_115 SeasonalNaive: 1.31
forms MT_166, one per cutoff: ETS(A,Ad,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_166 ZZA: 1.18
MASE MT_166 SeasonalNaive: 1.29
forms MT_217, one per cutoff: ETS(M,N,A) ETS(M,N,A) ETS(M,N,A)
MASE MT_217 ZZA: 1.78
MASE MT_217 SeasonalNaive: 1.25
forms MT_268, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_268 ZZA: 0.94
MASE MT_268 SeasonalNaive: 0.89
forms MT_319, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_319 ZZA: 1.11
MASE MT_319 SeasonalNaive: 1.26
forms MT_370, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_370 ZZA: 1.05
MASE MT_370 SeasonalNaive: 1.25
```

On mean MASE, forcing the season brings AutoETS within a hair of seasonal naive, though not below it: 1.17 against 1.30 for the default and 1.16 for seasonal naive.

The forced season helps some clients and hurts others. It lowers the lead client's MASE, from 1.18 to 1.08, and `MT_064`'s too, but it raises `MT_370`'s, from 0.96 to 1.05. So forcing the season is a change to test on each client, not a fix to apply to all of them.

**Working rule:** Start with AutoETS for hourly demand with `season_length=24`, and print the chosen forms in every backtest. When a client takes a flat form, check that its seasonal form is not failing. If the seasonal fit has the lower likelihood, force the season with `model="ZZA"`, and score both forms on earlier windows before you keep either: cutoffs before the held-out week that the report does not score, such as the four ordinary days in late November and early December 2014, which [Chapter 5](#choose-on-decision-windows) sets aside for choices. The standard cutoffs show the comparison here, but they do not decide it. When a forecast gets worse, look first for a form that changed between cutoffs.

## Fit a fixed seasonal ARIMA {#fit-a-fixed-seasonal-arima}

The second candidate models what seasonal naive leaves behind. The backtests of Chapter 3 are the test for what it adds. An ARIMA model writes the next reading as a weighted sum of earlier readings and earlier errors, after differencing: subtracting a reading from the reading one season earlier. The model is written ARIMA(p,d,q)(P,D,Q)[m]. The non-seasonal part has p autoregressive terms, d differences and q moving-average terms. The seasonal part has the same three, at lags that are multiples of the season length m.

The book's model is ARIMA(1,0,0)(1,1,0)[24]. It has one autoregressive term, no ordinary difference, no moving-average term, one seasonal difference at lag 24, and one seasonal autoregressive term at lag 24. Written out, with B the step back operator, so that B y_t is y_(t-1):

`(1 - phi1 B)(1 - Phi1 B^24)(1 - B^24) y_t = e_t`

The term `(1 - B^24)` replaces each reading with its change from the same hour yesterday. The lag of 24 is the point of the model for hourly demand. Each hour is compared with the same hour yesterday, which is the comparison seasonal naive makes, and the autoregressive terms then model what that comparison leaves behind. The seasonal difference removes the level, so the model has no mean term. statsforecast's `ARIMA` leaves the mean off for differenced series by default (`statsforecast/models.py`, the `ARIMA` docstring).

We fix the order rather than search for it, and [the section on automatic search](#why-the-book-avoids-autoarima) explains why. Before the order goes into a backtest, fit it once with statsmodels 0.15.0 on the lead client's history before the first standard cutoff, and print the names of the fitted parameters. That shows whether the model has the terms you wrote down.

Example `arima-summary-look` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsmodels 0.15.0 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Fit the fixed seasonal ARIMA with statsmodels on the lead client's history before the first standard cutoff, and print its parameter names. The parameter lines are what the prose reads."""
import pandas as pd
from statsmodels.tsa.statespace.sarimax import SARIMAX

from lib.backtest import history_before
from lib.electricity import load
from lib.settings import EVAL_CUTOFFS, LEAD, show

hist = history_before(load([LEAD]), pd.Timestamp(EVAL_CUTOFFS[0]))
y = pd.Series(hist["y"].to_numpy(), index=pd.DatetimeIndex(hist["ds"], freq="h"))
res = SARIMAX(y, order=(1, 0, 0), seasonal_order=(1, 1, 0, 24)).fit(disp=False)
show("hours of history", len(y))
for name in res.params.index:
    show("parameter", name)
```

Output:

```text
hours of history: 1344
parameter: ar.L1
parameter: ar.S.L24
parameter: sigma2
```

The list has three names: `ar.L1`, `ar.S.L24` and `sigma2`. The first is the autoregressive term, which says how much an hour's change from the same hour yesterday carries into the next hour. The second is the seasonal term at lag 24, which says how much a day's change at an hour carries into the same hour the next day. The third is the variance of the error. The list has no intercept. statsmodels adds one to a seasonal ARIMA only when `trend` is set, so the list would show none whether or not the series is differenced, and the seasonally differenced model needs none in any case. A full statsmodels summary also gives each value. This book does not quote them. The values change with the window and the library version, so a reader needs what each term means more than its number.

The fixed model is then one more candidate in the backtest, beside AutoETS and seasonal naive. Its scores come in the [comparison section](#compare-models-on-the-same-cutoffs) below.

**Working rule:** When a search is too slow to run at every cutoff, fix the order and write it in the call. Print the summary once, and check that the terms it lists are the ones you meant. Refit the fixed order at every cutoff, as you would refit any model. The order stays fixed. The coefficients do not.

## Compare models on the same cutoffs {#compare-models-on-the-same-cutoffs}

Four forecasts now cover the same 24 hours at the same three cutoffs, from the same eight weeks of history, for the same eight clients: seasonal naive, AutoETS, AutoETS with the season forced, and ARIMA. Each backtest is a separate script, so each finishes well inside the 45 seconds the book allows an example on two CPUs.

One score is still missing: the fixed ARIMA's, on the same windows and measures.

Example `arima-subset-backtest` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Backtest the fixed seasonal ARIMA(1,0,0)(1,1,0)[24] and seasonal naive for the eight clients of SUBSET8 over the three standard cutoffs.
The MASE gap and the count of clients where ARIMA wins, printed last, are the figures the comparison section quotes."""
import numpy as np
import pandas as pd
from statsforecast import StatsForecast
from statsforecast.models import ARIMA, SeasonalNaive

from lib.scoring import book_scale
from lib.electricity import load
from lib.quirks import filled_history, mask_clock_changes
from lib.settings import EVAL_CUTOFFS, HORIZON, SEED, SUBSET8, show
from lib.subsets import score_clients

np.random.seed(SEED)                                        # the seed from lib/settings.py, set before any fit
masked = mask_clock_changes(load(SUBSET8))                  # clock-change hours become NaN, so they are never scored; filled_history fills them from their neighbours before each fit
scale = book_scale(load(SUBSET8), EVAL_CUTOFFS[0])        # MASE scale: rows before the first cutoff, from each client's first reading
# The order is fixed, not searched: one AR term, one seasonal difference and one seasonal AR term, all at lag 24.
# n_jobs=1 keeps the output the same from run to run.
sf = StatsForecast(models=[ARIMA(order=(1, 0, 0), season_length=24, seasonal_order=(1, 1, 0)),
                           SeasonalNaive(season_length=24)], freq="h", n_jobs=1)
parts = []
for c in pd.to_datetime(EVAL_CUTOFFS):
    sf.fit(df=filled_history(masked, c)[["unique_id", "ds", "y"]])   # the eight weeks before c, clock hours filled from them
    part = sf.predict(h=HORIZON)
    # the first forecast hour must be the cutoff itself, or the window is misaligned
    if part["ds"].min() != c:
        raise ValueError(f"forecast for {c:%Y-%m-%d} does not start at the cutoff")
    parts.append(part)
forecasts = pd.concat(parts, ignore_index=True)

summary, per_client = score_clients(forecasts, masked, ["ARIMA", "SeasonalNaive"], scale)
show("horizon hours", HORIZON)
show("windows", len(parts))
for label, value in summary.items():
    show(label, value)
show("MASE gap, SeasonalNaive minus ARIMA", summary["MASE SeasonalNaive"] - summary["MASE ARIMA"])
show("clients where ARIMA has lower MASE", int((per_client["ARIMA"] < per_client["SeasonalNaive"]).sum()))
```

Output:

```text
horizon hours: 24
windows: 3
clients: 8
scored hours per client: 72
MAE ARIMA: 386.13
RMSE ARIMA: 576.30
MASE ARIMA: 1.09
MAE SeasonalNaive: 423.68
RMSE SeasonalNaive: 591.25
MASE SeasonalNaive: 1.16
MASE gap, SeasonalNaive minus ARIMA: 0.07
clients where ARIMA has lower MASE: 5
```

MAE is in kilowatt hours, and [RMSE](#score-on-a-held-out-week), the root mean squared error, punishes large misses more. Both are computed for each client over its 72 scored hours, then averaged over the eight clients, so each client counts once. [MASE](#scale-errors-with-mase) divides each client's MAE by the MAE that seasonal naive makes in its own history, over every reading before Dec 29, 2014, from each client's first reading and with the clock-change hours left out, then averages over clients. A MASE of 1 means the model is as accurate as seasonal naive was on its own past. The [coverage](#why-one-number-is-not-enough) of AutoETS is the share of scored hours that fall inside its 95% [prediction interval](#why-one-number-is-not-enough), the range the model gives for each hour.

| Model | MAE (kWh) | RMSE (kWh) | MASE | Clients beating seasonal naive on MASE |
| --- | --- | --- | --- | --- |
| SeasonalNaive | 423.68 | 591.25 | 1.16 | not applicable |
| AutoETS | 362.11 | 537.20 | 1.30 | 4 |
| AutoETS, season forced (ZZA) | 365.24 | 553.11 | 1.17 | 4 |
| ARIMA | 386.13 | 576.30 | 1.09 | 5 |

Read the last column as a count out of the eight clients, not as a score. The AutoETS intervals cover 0.910 of the scored hours, and the forced-season intervals cover 0.925. The nominal share is 0.95, so the intervals are a little narrow on this sample. Coverage checks the intervals, not the point forecast. An interval that covers too little is too narrow, and a planner who buys to it will be short more often than its label says. Chapter 8 returns to intervals and shows how to calibrate them on earlier windows.

The two measures disagree about AutoETS. Its mean MAE is the lowest of the four rows, and its mean MASE is the highest. MAE in kilowatt hours gives more weight to the clients with large loads, because their errors are large. MASE scales each client's error by that client's own history, so every client counts about the same. The per-client lines in the AutoETS run show where the MASE comes from. `MT_064` has a MASE of 2.32, against 1.08 for seasonal naive, and it is one of the flat-form clients. `MT_217` has 1.76, against 1.25, and its form changes from one cutoff to the next. The ranking depends on the measure, so the choice of measure is part of the decision.

Three windows are a small sample, as [Chapter 3](#why-one-test-week-misleads) explains. AutoETS beats seasonal naive on 4 of the eight clients, and ARIMA on 5. ARIMA is better than seasonal naive on both measures, and on MASE by 0.07. Three windows can't settle a gap that small.

**Working rule:** Score every candidate on the same cutoffs, with the same history and the same scale, and report MAE and MASE side by side. When the two disagree, say which one the decision needs. We recommend more windows before choosing between two models that differ by less than about a tenth of MASE. That threshold is a judgement from this sample, not a measurement.

## Meet the late starters {#meet-the-late-starters}

One client in the subset starts late. `MT_115`'s first non-zero reading is at 00:00 on Jan 14, 2014. Before that, the meter reports zeros, and the data flags those hours `before_first_reading`. Those zeros are not low use. A model that reads them learns a level the client does not have. A late starter is a client whose first reading falls inside the history its model reads. The fix is the start rule: a series begins at its first reading. This matters for the supplier's decision. A level pulled low means buying too little for that client, the costlier of the supplier's two errors.

At the December cutoffs the eight weeks before each one are all readings, so this section uses February cutoffs: Feb 1, 2014, Feb 15, 2014 and Feb 28, 2014. Their eight-week histories still reach back before the first reading, so the case is in them. [Chapter 6](#forecast-many-clients-at-once) applies the same rule to all thirty clients.

### A history that starts with zeros {#a-history-that-starts-with-zeros}

Start with the raw history, the eight weeks before each February cutoff with its zeros included. The share of zero hours in that history is 68% on Feb 1, 2014, 43% on Feb 15, 2014 and 20% on Feb 28, 2014. The share falls as the cutoff passes the first reading.

Starting the history at the first reading may fix it. Fitting AutoETS twice at each February cutoff, once on the raw eight weeks and once on the start history, which begins at `MT_115`'s first reading, shows whether it does, scored on the 24 hours from each cutoff.

Example `late-starter-trim` (Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Backtest AutoETS for MT_115, a late starter, on three February cutoffs: once on its raw eight weeks and once on its history from its first reading.
The start-rule counts, printed first, and the three-window MAE lines are the figures the point rests on."""
import numpy as np
import pandas as pd
from statsforecast import StatsForecast
from statsforecast.models import AutoETS

from lib.backtest import history_before
from lib.electricity import load
from lib.quirks import fill_clock_change_hours, mask_clock_changes
from lib.settings import HORIZON, SEED, SUBSET8, show
from lib.starts import check_start_rule, history_from_start

LATE = "MT_115"                                             # the one late starter in SUBSET8, first non-zero reading on Jan 14, 2014
# February cutoffs: at the December cutoffs every hour of the history is a reading, so the rule changes nothing there
CUTOFFS = pd.to_datetime(["2014-02-01", "2014-02-15", "2014-02-28"])

np.random.seed(SEED)                                        # the seed from lib/settings.py, set before any fit
dropped, flagged = check_start_rule(load(SUBSET8))           # raises unless the rule drops exactly the flagged rows, so the counts agree
show("start rule drops, SUBSET8", dropped)
show("flagged before_first_reading, SUBSET8", flagged)
masked = mask_clock_changes(load([LATE]))[["unique_id", "ds", "y", "quirk"]]


def fit_and_score(hist, c):
    """Fit AutoETS on one history, forecast the 24 hours from c, and score it on those hours."""
    sf = StatsForecast(models=[AutoETS(season_length=24)], freq="h", n_jobs=1)
    sf.fit(df=fill_clock_change_hours(hist)[["unique_id", "ds", "y"]])   # clock-change hours filled from their neighbouring hours, as elsewhere
    fc = sf.predict(h=HORIZON)
    actual = masked[(masked["ds"] >= c) & (masked["ds"] < c + pd.Timedelta(hours=HORIZON))]   # the day's 24 readings, to score against
    joined = fc.merge(actual[["ds", "y"]], on="ds", how="left")
    # the form's method name, the mean forecast, the day's MAE, and the mean of the readings
    return sf.fitted_[0, 0].model_["method"], float(joined["AutoETS"].mean()), float((joined["y"] - joined["AutoETS"]).abs().mean()), float(joined["y"].mean())


maes = {"raw": [], "start": []}
for c in CUTOFFS:
    raw_hist = history_before(masked, c)                     # the eight weeks before c, zeros included; it insists on exactly 1,344 rows
    # the share of hours that read zero, including the leading zeros before the first reading
    show(f"zero percent before {c:%Y-%m-%d}", round(100 * (raw_hist["y"] == 0).mean()))
    # the start history begins at the first reading, so it is shorter wherever the client starts late
    for label, hist in [("raw", raw_hist), ("start", history_from_start(masked, c))]:
        form, fc_mean, window_mae, actual_mean = fit_and_score(hist, c)
        show(f"{label} history hours {c:%Y-%m-%d}", len(hist))
        show(f"{label} form {c:%Y-%m-%d}", form)
        show(f"{label} mean forecast {c:%Y-%m-%d}", fc_mean)
        show(f"{label} mean actual {c:%Y-%m-%d}", actual_mean)
        show(f"MAE {label} {c:%Y-%m-%d}", window_mae)
        maes[label].append(window_mae)
for label in maes:
    show(f"MAE {label}, three windows", np.mean(maes[label]))
    show(f"MAE {label}, 2014-02-15 and 2014-02-28", np.mean(maes[label][1:]))   # [1:] skips Feb 1, the first cutoff
```

Output:

```text
start rule drops, SUBSET8: 9072
flagged before_first_reading, SUBSET8: 9072
zero percent before 2014-02-01: 68
raw history hours 2014-02-01: 1344
raw form 2014-02-01: ETS(A,N,N)
raw mean forecast 2014-02-01: 16.43
raw mean actual 2014-02-01: 24.06
MAE raw 2014-02-01: 8.81
start history hours 2014-02-01: 432
start form 2014-02-01: ETS(M,Ad,A)
start mean forecast 2014-02-01: 23.88
start mean actual 2014-02-01: 24.06
MAE start 2014-02-01: 3.72
zero percent before 2014-02-15: 43
raw history hours 2014-02-15: 1344
raw form 2014-02-15: ETS(A,N,A)
raw mean forecast 2014-02-15: 24.54
raw mean actual 2014-02-15: 27.11
MAE raw 2014-02-15: 4.29
start history hours 2014-02-15: 768
start form 2014-02-15: ETS(M,Ad,A)
start mean forecast 2014-02-15: 23.84
start mean actual 2014-02-15: 27.11
MAE start 2014-02-15: 4.77
zero percent before 2014-02-28: 20
raw history hours 2014-02-28: 1344
raw form 2014-02-28: ETS(A,N,A)
raw mean forecast 2014-02-28: 23.73
raw mean actual 2014-02-28: 23.54
MAE raw 2014-02-28: 2.93
start history hours 2014-02-28: 1080
start form 2014-02-28: ETS(M,Ad,A)
start mean forecast 2014-02-28: 22.48
start mean actual 2014-02-28: 23.54
MAE start 2014-02-28: 2.73
MAE raw, three windows: 5.34
MAE raw, 2014-02-15 and 2014-02-28: 3.61
MAE start, three windows: 3.74
MAE start, 2014-02-15 and 2014-02-28: 3.75
```

The damage is in the level, and it is worst on Feb 1. AutoETS fits the raw history without an error and forecasts without a warning. On Feb 1, 2014 it chooses the flat form, ETS(A,N,N), and its forecast averages 16.43 kWh an hour over the day. The readings that followed average 24.06 kWh. The MAE on that day is 8.81 kWh. The likely cause is the zeros: they pull the level down, and the model cannot tell that they are not use.

On the other two days the raw history's errors are much smaller, at 4.29 kWh on Feb 15, 2014 and 2.93 kWh on Feb 28, 2014. So the damage depends on how many zeros the window holds. The rule is right whatever the zero share, because leading zeros are not use.

### Start each series at its first reading {#start-each-series-at-its-first-reading}

The start rule says that a series begins at its first reading. It drops every reading before the client's first non-zero reading, and it keeps every reading after that, zeros included, because the zeros after the start are real low use and a model needs them. A window that begins after the first reading keeps all its hours, since the rule removes leading zeros only. The rule lives in `examples/lib/starts.py`, so later chapters call the same code. Its function `history_from_start(df, cutoff)` returns one cutoff's history under the rule: the readings in the eight weeks before the cutoff, from the first reading onward. It finds that first reading from the rows before the cutoff only, so no reading after the cutoff reaches the model. It raises an error if a client has no reading before the cutoff, or fewer than 48 hours of history.

Before the rule is used, check that it does what the flag says. `check_start_rule` counts the rows the rule would drop across the whole series, and raises an error unless those rows are exactly the rows flagged `before_first_reading`. On the subset, the rule drops 9072 hours, and the flag marks 9072. They match. The only late starter in the subset is `MT_115`.

The rule shortens the history most at the first cutoff, 432 hours against 1,344, and less as the cutoff moves past the first reading: 768 hours on Feb 15 and 1080 on Feb 28.

The start history picks ETS(M,Ad,A) at all three cutoffs, a form with multiplicative error, damped trend and additive season. The raw history changes its form between days, from the flat form on Feb 1 to ETS(A,N,A) on Feb 15, 2014 and ETS(A,N,A) on Feb 28, 2014.

The three-window MAE falls from 5.34 kWh to 3.74 kWh. The gain is uneven across the windows, and all of that gain comes from Feb 1, where the MAE falls from 8.81 to 3.72 kWh.

On Feb 15 the rule does worse, and on Feb 28 it does a little better. Over those two days together the raw history does slightly better, with a mean MAE of 3.61 kWh against 3.75.

The fits show how the rule loses on Feb 15. The start history is still short, 768 hours, and its damped trend forecasts lower than the raw fit does: 23.84 kWh an hour on average, against 24.54 for the raw fit, when the readings averaged 27.11 kWh. That low forecast is the cost the case warns about. One client and three days are a small sample, so the figures illustrate the rule rather than prove it.

**Working rule:** When a client's history holds zeros, check whether its first non-zero reading falls inside the history. If it does, apply the start rule before any model sees the history, whatever the zero share, because leading zeros are not use. Run `check_start_rule` on the whole series first, and if it fails, find out why before you trim anything. Print the history length at each cutoff, because a short history is the first sign that the rule has taken too much.

## Why the book avoids automatic ARIMA at season 24 {#why-the-book-avoids-autoarima}

AutoARIMA searches for the orders instead of fixing them. It runs a stepwise search over candidate orders, picks the best by AICc, which is its default, and stops after at most 94 candidate models (statsforecast 2.1.1, `statsforecast/models.py`, and the stop message in `statsforecast/arima.py`). Each candidate is a full state-space fit. With a season of 24, each candidate that has seasonal terms at lag 24 is slow, and the search tries dozens of them, so it takes far longer than the fits in this chapter.

On the book's pinned setup, two CPUs with statsforecast 2.1.1, one window for one client took about 75 seconds. That was one run on `MT_014` at the first standard cutoff. The subset needs 24 such windows, so one backtest of it would take about half an hour, against the 45 seconds the book allows an example. That is this book's own measurement, not an example in it, and it depends on the setup, so time it on your own machine before you plan a schedule around it. The fixed order costs one fit per window, which is the difference that matters here. Its weak point is a client whose pattern differs from the order's assumptions. The backtest is the check for that, and the [scores in the comparison section](#compare-models-on-the-same-cutoffs) are the place to look.

A fixed order is a judgement, not a finding. We chose (1,0,0)(1,1,0) with a season of 24 because the data has a daily pattern and the backtests in this chapter need a model that finishes. The order sits in the call, where a reader can see it and change it. Later chapters compare against it on the same windows, so a change to the order shows up in the scores.

**Working rule:** Do not run an automatic search with a season of 24 on hourly data in a schedule that refits every day. Time the search on one client before you commit to it. If it is too slow, fix the order and test that order in the backtest.

## What goes wrong in production {#what-goes-wrong-ch4}

- **A missing season.** A season length left at 1 gives a model with no daily pattern, and no error. The sign is that every client chooses a form with no season, such as ETS(A,N,N). A few clients doing so is a different case: a seasonal fit that fails, which the flat-form check in [the exponential smoothing section](#fit-exponential-smoothing) tests. Log the chosen forms at each refit.
- **A late starter's zeros.** A client that begins inside the history has its level pulled down until its first reading leaves the window. The sign is a forecast that runs low for days after the start. Count the zero share of each client's history before each refit, and apply the start rule to any client whose first reading falls inside the history.
- **Forms that change.** The chosen form can change between refits for one client. That is not wrong in itself, but a switch from a seasonal form to one with no season is worth checking against the readings. Keep the forms in the run log.
- **Costs that add up.** Fits scale with clients, times windows, times refits. Count them before you add clients to a schedule.
- **Intervals that miss.** The AutoETS intervals covered 0.910 of hours on this sample, short of the nominal 0.95. Chapter 8 shows how to calibrate bands on earlier windows, and until a band is calibrated that way, do not present it as a 95% range.
- **How you would notice.** Run seasonal naive beside every statistical model on each day's windows, and log the MASE gap. A gap that changes sign from day to day is the signal to look at the model, not a result to report.

## Decisions this chapter made {#decisions-ch4}

- **AutoETS with a season of 24.** The default `model="ZZZ"` and the forced-season run, `model="ZZA"`, are both reported, because a flat choice can be a failed seasonal fit. This chapter makes no choice between the two forms: the standard-cutoff comparison is reported, not used to choose, and the choice belongs on earlier windows. [Chapter 1](#forecast-with-a-statistical-model) forced `ZZA` on the lead client, and this chapter shows the forced run for all eight.
- **A fixed ARIMA(1,0,0)(1,1,0)[24], with no search.** This is a stated judgement, and the backtest is what tests it.
- **Eight clients, not thirty.** Chapter 6 fits all thirty clients with one model. A per-client fit of AutoETS for all thirty over three windows takes about 97 seconds on the pinned setup, against the 45 seconds the book allows an example, so this chapter uses the subset.
- **The subset, the cutoffs and the scale.** `SUBSET8`, the three standard cutoffs, Dec 29 to Dec 31, 2014, 1,344 hours of history, and seasonal naive as the bar in every comparison. The MASE scale reads each client from its first reading, with the clock-change hours left out.
- **The clock-change hours.** Every subset example masks them and fills them from history, as [Chapter 3](#mark-clock-change-hours-as-missing) sets out. None of the three windows holds a clock-change hour, so the masking changes no forecast and no scored hour. It does change the MASE scale, because the scale reads all history before Dec 29, 2014, and it counts `MT_115` from Jan 14, 2014.
- **Late starters: February cutoffs and the start rule.** The December cutoffs do not meet `MT_115`'s late start, so the February cutoffs do. Chapter 6 applies the rule to all thirty clients.
- **Coverage for the two AutoETS runs only.** The ARIMA backtest is scored without intervals here.
- **No fitted values quoted.** The ARIMA script prints parameter names only, and the flat-form check prints counts only. Errors are rounded to two decimals and coverage to three.
- **A statistical baseline for Chapter 5.** The per-client AutoETS and ARIMA scores on these windows are the bar the gradient boosting model must beat.
- **What to carry forward.** A flat form can be a failed seasonal fit, so read the chosen forms beside every score. MAE and MASE can rank models differently, so say which one the decision needs. Start each series at its first reading, and add windows before choosing between close models.

# Add calendar and lag features

After this chapter, you can turn one client's hourly history into a feature table, add lags and calendar and holiday features that a day-ahead forecast is allowed to see, measure what each feature adds on the same windows, and catch a lag that reads the day being forecast.

The statistical models in [Chapter 4](#fit-exponential-smoothing) see only a series. They know its level and its daily rhythm, and nothing else: they do not know that Dec 25, 2014 was a public holiday, that a Monday differs from a Sunday, or that demand moves with the season. [Chapter 1](#score-on-a-held-out-week) showed what that costs. Its seasonal naive baseline had its worst day on Dec 26, the day after Christmas, because it copied the holiday's readings forward. A gradient-boosting model can take such facts in as columns, and because the supplier buys before the day starts, only columns known at the cutoff may enter its forecast. This chapter adds calendar and holiday columns for the eight-client subset and checks each one before it counts. The holiday flag does not repair the Dec 26 failure, because Dec 26 is not a holiday. The feature that would repair it, a day-after-holiday flag, is also known in advance, but this chapter does not build it, and its standard cutoffs do not score a holiday.

## Turn history into features {#turn-history-into-features}

The table comes first, because every feature in this chapter is a column of it. A feature table has one row for each hour a model must predict. Each row of a feature table holds the value to predict, which the book calls the target, and the features: the readings and facts the model may use for the row's hour. A gradient-boosting model learns a mapping from the columns to the target, one row at a time, so the table is all the model can know.

A row's features must be known at the cutoff of the window that forecasts it, because the forecast is made there. The target is the reading at the hour. The model needs two kinds of row. Training rows are the hours of the history before the cutoff, each with its own features and its own reading as the target. Forecast rows are the 24 hours after the cutoff, with features but no target. The model is fitted on the first kind and predicts the second.

Take the cutoff at 00:00 on Dec 30, 2014. Its forecast rows are the 24 hours of Dec 30, from 00:00 to 23:00, and each has a blank target. Its training rows run back eight weeks, to Nov 4, 2014, one row per client and hour, with the reading as the target. Take the row for Dec 15 at 18:00: its target is the reading at that hour, and its features are readings from earlier hours, such as the reading 24 hours before. No row is a daily total.

Each window's training table is built from the 1,344 hours of history before its cutoff, eight weeks, for each client in `SUBSET8`. The eight clients are stacked in one table, and the same construction repeats for every decision window and for the three standard cutoffs, Dec 29 to Dec 31, 2014. The first week of history has no reading a week earlier, so it leaves the table: 1,344 hours less 168 hours leaves 1,176 rows per client in each window. One model is fitted per window on that window's history alone, so no training row comes from the days being forecast.

For the lags-only set the columns are the two lags and the target `y`; the hour of the day arrives with the other calendar columns later in the chapter, and the holiday column after them. `MLForecast` in `mlforecast` 1.1.0 takes the series and produces the training rows in `fit` and the forecast rows in `predict`. Two of its defaults matter here. A training row whose features are missing is dropped (`dropna=True`, in `mlforecast/core.py`), and the first hours of each history drop out because they have no reading a week earlier. Nothing in the library stops a feature from reading the hour it predicts. The rule that stops it is the one you write and check.

**Working rule:** Build every feature row as if the cutoff were the last reading you have. A feature for a forecast hour must be computable from readings at or before the cutoff of its window. Before any shift test, ask whether the feature is known at the cutoff. Then check it with the [shift test from Chapter 3](#check-a-backtest-for-leakage), and do not train on a table you have not checked.

## Add lags the forecast can see {#add-lags-the-forecast-can-see}

Lags are the first columns to add to that table: earlier readings of the same client. The lag of k hours for the hour t is the reading at t minus k. Lags are the cheapest way to give a model the rhythm of a series: a lag of 24 is the reading at the same hour yesterday, and a lag of 168 is the reading at the same hour a week ago. The model learns how much to trust each one. The lag of 168 also reads a whole week back, which is the same weekday, so it carries the weekly pattern without a weekday column.

Whether a lag is known depends on the horizon. A forecast made at 00:00 covers the 24 hours from the cutoff. Call the hours of that day 0 to 23, counted from the cutoff. A lag of k reads hour j minus k, so for the target hour j it reads a reading before the cutoff only when j minus k is negative. That must hold for every j from 0 to 23, which means k must be 24 or more. A lag of 1 is known for the first hour, because it reads the last reading before the cutoff. For the other 23 hours it reads a reading the forecast does not have.

The table below shows the three kinds of target hour in that day-ahead forecast, for a cutoff at 00:00. Hour 0 is the first hour after the cutoff, and hour 23 is the last.

| Target hour | Lag 1 reads | Lag 1 known at cutoff | Lag 24 reads | Lag 24 known at cutoff |
| --- | --- | --- | --- | --- |
| 00:00 (hour 0) | 23:00 the day before | yes | 00:00 the day before | yes |
| 01:00 (hour 1) | 00:00 of the forecast day | no | 01:00 the day before | yes |
| 23:00 (hour 23) | 22:00 of the forecast day | no | 23:00 the day before | yes |

Both of the supplier's lags, 24 and 168, are known at every hour of the day-ahead forecast, so the model never needs a reading that does not yet exist. [lgbm-feature-sets](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/lgbm-feature-sets.py) passes these two lags to `MLForecast`, and the library does the rest.

The [clock rule from Chapter 3](#mark-clock-change-hours-as-missing) reaches the lags too: the Mar 30, 2014 gap hour is set to missing, so any feature that reads it is missing as well. In the history of an Apr 1, 2014 cutoff, [lgbm-feature-sets](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/lgbm-feature-sets.py) finds 1 masked hour, and 1 row has a lag that reads it: the row 24 hours after the gap. Before each window's model is fitted, its masked hours are filled [as Chapter 3 fills them](#mark-clock-change-hours-as-missing), from that window's own history, and the fill reads nothing after the cutoff. Once filled, the hour is a reading, so the row that reads it keeps its value, and LightGBM never receives a missing value from a lag. The rule is a stated choice, not a measured optimum. The count printed beside it is what a reader should check.

**Working rule:** Use only lags at least as long as the horizon. With a 24-hour horizon that means 24 hours or more, and this book uses 24 and 168. A shorter lag needs a reason, and a model that uses one must pass the shift test before anyone trusts its score. Keep the clock rule on the lags, and count the rows whose lags read a masked hour, and the hours filled from history.

## Add calendar and holiday features {#add-calendar-and-holidays}

Lags are built from readings. Calendar features, unlike lags, are facts about the target hour itself: its hour of day, its day of the week and its month. They are known in advance, because the clock states them before any reading exists. They need no lag and no cutoff rule. Their value at a target hour cannot depend on any reading, so the shift test has nothing to catch.

The supplier's demand follows the day. The hour of day repeats what the lags already carry: the daily rhythm. The weekday adds what they do not. It tells the model that Saturday and Sunday differ from the working days. The month is the third column, and it is a weak one with this history. Eight weeks of history span two or three calendar months, so `month` can only tell a few weeks of the history from the rest, and on the standard cutoffs every forecast hour falls in December. The book uses three mlforecast date features, named by their attribute: `hour`, `dayofweek` and `month`. The weekday column already holds the weekend, so the book adds no separate weekend flag. In pandas, `dayofweek` counts Monday as 0 and Sunday as 6, so Saturday and Sunday are the two values a tree must separate from the rest. A tree can split on that in one step, which is why the weekday is cheap to test.

A public holiday does not follow the weekday. A model with only weekday features reads Dec 25, 2014 as an ordinary Thursday, and it reads a holiday on a Sunday as an ordinary Sunday. The holiday feature answers this. It is a function of the timestamp, `is_holiday` in the shared module `examples/lib/features.py`, which returns 1 on a Portuguese national holiday and 0 otherwise. mlforecast accepts such a function as a date feature and names its column after the function. The holidays come from the `holidays` package 0.110.dev20261118063242, under the MIT licence, through `holidays.country_holidays("PT", years=[2013, 2014])`. For the two years in the data, the package lists 18 dates. The book does not ship that list. Each run builds it from the package's calendar.

The standard cutoffs are Dec 29 to Dec 31, 2014, and none of their 72 hours per client is a holiday. The held-out week's Dec 25 is outside them, so the feature's effect on that day is not scored in this chapter. On the standard cutoffs, it reaches the scores only through the training history, which contains the holidays of Dec 8 and Dec 25, 2014.

The holiday list is a fact known in advance, like the weekday. It comes from a published calendar and not from the readings, so it cannot leak a reading from the target day. Calendar features answer the question in the [working rule](#turn-history-into-features) at every target hour, by construction, and a lag passes only when it is at least as long as the horizon.

**Working rule:** Add the calendar features the case's rhythm needs, and keep each one only if it helps on the same windows, as the next section measures. For holidays, use a calendar that covers every year in your data and every year you will forecast, and print how many holidays it holds for those years. A count of zero for a forecast year means the list is missing that year, not that the year had no holidays.

## Choose on decision windows {#choose-on-decision-windows}

Decision windows are the windows the feature set is chosen on, never the standard cutoffs, which are the windows the book reports. There are two kinds. The first is four ordinary cutoffs, Nov 25, Nov 26, Dec 2 and Dec 3, 2014. The second is seven holiday cutoffs: the Portuguese national holidays of 2014 before Dec 25 whose eight-week history is complete, Apr 18, Apr 20, Apr 25, May 1, Jun 10, Aug 15 and Dec 8. The ordinary windows hold no holiday, so a holiday feature could never win on them, and the holiday windows are where it can. The choice therefore uses all the decision windows, because one set holds a holiday feature. The standard cutoffs are scored after the choice and printed beside it. They choose nothing.

The holiday flag acts in two ways. On the decision windows it is on for scored hours, so it can change a forecast. On the standard cutoffs no holiday falls in a scored hour, so the flag changes those scores only through training.

**Working rule:** Choose a feature on windows that come before the held-out week and hold the kind of day the feature is for: a holiday feature needs holiday windows. The standard cutoffs are reported beside the choice and never used to make it.

## Measure what each feature adds {#measure-what-each-feature-adds}

A feature is worth keeping when it lowers the error on windows the model did not train on. To measure one, hold everything else fixed: the same windows, the same history, the same settings. Three feature sets are fitted, one at a time. The first has lags only. The second adds the calendar features. The third adds the holiday feature to them. Each set is fitted once per window, with one model for all eight clients of `SUBSET8`, and each is compared with seasonal naive on the same windows.

Each set gets two scores. The pooled MAE is the MAE over every client-hour in the windows, the single number for the whole supplier. The mean per-client MASE is MASE for each client, averaged over the eight clients. [MASE](#scale-errors-with-mase) divides each client's error by a scale: the mean absolute error that seasonal naive makes in sample on that client's own readings. The book uses one scale function everywhere, `scoring.book_scale`. The scale reads every reading before the first scored window of the set, Apr 18, 2014 at 00:00, with clock-change hours masked, so it does not depend on the windows scored and cannot leak. One client needs care. `MT_115` starts reading on Jan 14, 2014, and the zeros before its first reading are left out of its scale, because it is counted from its first reading. [Chapter 4](#meet-the-late-starters) teaches that rule for late starters.

Example `lgbm-feature-sets` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Compare three feature sets for the global LightGBM forecast of SUBSET8 on the decision windows (B-5, B-9, B-10): four
ordinary cutoffs and seven 2014 holidays before Dec 25, each fitted scaled and unscaled. The mean MASE lines carry the
choice; the standard-cutoff lines are reported beside it and choose nothing."""
import pandas as pd

from lib.backtest import history_before
from lib.electricity import load
from lib.features import CALENDAR, PT_HOLIDAYS, is_holiday, rows_reading_missing, score_feature_sets
from lib.panel import DECISION_CUTOFFS, HOLIDAY_DECISION_CUTOFFS
from lib.quirks import mask_clock_changes
from lib.settings import EVAL_CUTOFFS, LEAD, SUBSET8, show

LAGS = [24, 168]  # both at least the 24-hour horizon, so each is known at every forecast hour
# CALENDAR is hour, dayofweek and month, as mlforecast date features; the holiday set adds is_holiday to them
FEATURE_SETS = {"lags_only": [], "with_calendar": CALENDAR, "with_holidays": CALENDAR + [is_holiday]}
MODELS = ["SeasonalNaive", *FEATURE_SETS, *[f"unscaled_{name}" for name in FEATURE_SETS]]
DECISION = sorted([*DECISION_CUTOFFS, *HOLIDAY_DECISION_CUTOFFS])  # the decision windows; standard cutoffs are not here
raw = load(SUBSET8)

# DECISION[0] is the first decision window, Apr 18, 2014: the MASE scale reads only the readings before it
forecasts, scored, per_client = score_feature_sets(raw, DECISION, LAGS, FEATURE_SETS, DECISION[0])
show("clients", forecasts["unique_id"].nunique())
show("decision windows", len(DECISION))
show("scored hours", len(scored))
show("Portuguese holidays in 2013 and 2014", len(PT_HOLIDAYS))
show("holiday hours in the decision windows", int(is_holiday(scored["ds"]).sum()))
for model in MODELS:
    show(f"MAE {model.replace('unscaled_', 'unscaled ')}", (scored["y"] - scored[model]).abs().mean())
for model in MODELS:
    show(f"mean MASE {model.replace('unscaled_', 'unscaled ')}", per_client[model].mean(), digits=3)
for model in MODELS:
    # the prose does not quote this client, so this line is for reference only
    show(f"MASE MT_268 {model.replace('unscaled_', 'unscaled ')}", per_client.loc["MT_268", model], digits=3)

# The standard cutoffs, reported beside the choice above. Nothing is chosen from them.
# The scale here reads the readings before the first standard cutoff, Dec 29, 2014.
_, _, standard = score_feature_sets(raw, EVAL_CUTOFFS, LAGS, FEATURE_SETS, EVAL_CUTOFFS[0])
for model in MODELS:
    show(f"standard cutoffs, mean MASE {model.replace('unscaled_', 'unscaled ')}", standard[model].mean(), digits=3)

# Clock-change check: the history of a 1 April 2014 cutoff holds the 30 March gap, which the feature rules must reach.
march = history_before(mask_clock_changes(load([LEAD])), pd.Timestamp("2014-04-01"))  # the Chapter 3 mask, then the cut
show("masked hours in the history", int(march["y"].isna().sum()))
# a row counts when either lag, 24 or 168, reads a masked hour
show("rows whose lag reads a masked hour", rows_reading_missing(march.set_index("ds")["y"], LAGS))
```

Output:

```text
clients: 8
decision windows: 11
scored hours: 2112
Portuguese holidays in 2013 and 2014: 18
holiday hours in the decision windows: 1344
MAE SeasonalNaive: 305.21
MAE lags_only: 280.57
MAE with_calendar: 270.57
MAE with_holidays: 267.40
MAE unscaled lags_only: 304.03
MAE unscaled with_calendar: 271.63
MAE unscaled with_holidays: 280.75
mean MASE SeasonalNaive: 0.934
mean MASE lags_only: 0.886
mean MASE with_calendar: 0.894
mean MASE with_holidays: 0.893
mean MASE unscaled lags_only: 1.115
mean MASE unscaled with_calendar: 1.119
mean MASE unscaled with_holidays: 1.139
MASE MT_268 SeasonalNaive: 0.840
MASE MT_268 lags_only: 0.797
MASE MT_268 with_calendar: 0.758
MASE MT_268 with_holidays: 0.789
MASE MT_268 unscaled lags_only: 0.855
MASE MT_268 unscaled with_calendar: 0.871
MASE MT_268 unscaled with_holidays: 0.936
standard cutoffs, mean MASE SeasonalNaive: 1.162
standard cutoffs, mean MASE lags_only: 1.057
standard cutoffs, mean MASE with_calendar: 1.165
standard cutoffs, mean MASE with_holidays: 1.134
standard cutoffs, mean MASE unscaled lags_only: 1.357
standard cutoffs, mean MASE unscaled with_calendar: 1.494
standard cutoffs, mean MASE unscaled with_holidays: 1.486
masked hours in the history: 1
rows whose lag reads a masked hour: 1
```

Seasonal naive is the bar here. Its MAE, pooled over 2112 scored hours (eight clients in 11 windows of 24 hours each), is 305.21 kWh.

Without scaling, the lags-only model trails seasonal naive on the per-client view. Its mean per-client MASE is 1.115, worse than seasonal naive's 0.934. The clients differ a great deal in size, so the largest clients' readings are many times the smallest's. A [global model](#one-model-for-many-clients) fits one loss over every client's rows, and here that loss is squared error in kWh, so the largest clients' errors set the splits and the small clients carry little weight. We think the cause is the loss, not the range of any one client. The scaled runs below remove the difference, which supports that reading. It is a judgement from these numbers, not a proof.

The scaled runs standardise each client's target with mlforecast's `LocalStandardScaler` (`mlforecast/target_transforms.py`), which subtracts the client's mean and divides by its standard deviation. Both come from the window's history before the cutoff, fitted inside `fit`, and `predict` only inverts them. No reading after the cutoff sets them, so the scaling passes the same test as the features. `make_forecaster` takes `scaled=True` by default, and Chapter 6 reuses that default.

Scaled, the comparison changes, and lags alone beat seasonal naive on the per-client view: a mean MASE of 0.886, against 0.934. Its pooled MAE is 280.57 kWh.

The calendar set does not beat lags alone. It still beats seasonal naive: its mean MASE is 0.894, and its pooled MAE is 270.57 kWh.

The holiday flag, added to the calendar set, narrows the gap without closing it: the mean is 0.893, and the pooled MAE is 267.40 kWh.

[Chapter 4](#compare-models-on-the-same-cutoffs) set AutoETS and ARIMA as the bar a gradient-boosting model must beat. On the standard cutoffs, lags alone also come in under that bar: 1.057 against ARIMA's 1.09, on the same clients, cutoffs and scale.

By the working rule, only the lags stay, and the per-client view decides it, because it is the view that matters to a client's forecast. The calendar set's mean MASE is above lags alone's, so it is not kept. The pooled view points the other way here: the calendar set's pooled MAE is below lags alone's, and the holiday set's is lower still. The chapter reports both, as the rule asks, and does not tune the model to make them agree.

The rule keeps a feature only when it lowers the error of the set it extends. The holiday flag lowers the calendar set's mean by a hair, too little to count as a gain, and the set that holds it is still above lags alone. The flag is on for 1344 client-hours, a full day on each holiday window, for every client, so it had its chance, and it does not earn a place. The book therefore hands Chapter 6 the lags-only set. That result comes from 11 windows, which is a small test.

**Working rule:** Keep a feature set only when it lowers the mean per-client MASE against the set it extends, on the same windows. Report the pooled MAE beside it. Do not tune the model to make a disagreement between the two views go away.

## Catch a lag that leaks the future {#catch-a-lag-that-leaks}

The measurements above assume that each feature is known at its cutoff. A [leak](#keep-the-future-out-of-history) is a feature that uses a reading the forecast cannot have at its cutoff. In the book's view, a leak is a likely reason a backtest looks better than the model will be, and it can be built by accident. One pattern builds the feature table once from the whole series, so each target hour carries its lag, and then scores the table's forecast rows as if they were out of sample. Nothing fails. The score is too good, and nothing in it says so.

The library adds a second trap of its own. `MLForecast` predicts recursively by default. In recursive prediction, for a 24-hour horizon, `MLForecast` forecasts one hour, writes that forecast back as the reading for the next hour, and repeats (`_predict_recursive` in `mlforecast/core.py`). A lag of 1 therefore reads a forecast for every hour after the first, while training read actual readings for that lag. Calling that mismatch a leak is the book's judgement. It is one more reason to choose lags of 24 or more. With lags of 24 and 168, every step of the 24 reads actual readings, so the recursion never feeds this model its own forecasts. With `max_horizon` set, the library trains one model per horizon instead, and predicts with them in `_predict_multi`. The rule does not change with it: a feature must still be known at the cutoff of the window it forecasts.

The [shift test from Chapter 3](#check-a-backtest-for-leakage) carries over with one change: compare the lag at the target hours after the cutoff, not at the history rows. Compute the lag for each target hour twice: once from the history cut at the cutoff, and once from the whole series. Count the hours where the two differ. A lag of 24 agrees on every target hour, because each of the 24 hours reads a reading before the cutoff. A lag of 1 agrees only on the first hour.

Which do you expect to have the lower error, the model with a lag of 1 or the one without it? Both are fitted on the same training rows with the same settings, for each of the three standard cutoffs. The honest model uses lags 24 and 168. The leaky model adds a lag of 1, read from the whole series for the target hours, which is what the table built once would do.

Example `lag-leak-check` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Shows which lag leaks: a lag of 1 reads the hour before each target hour, which a 24-hour forecast does not have at its
cutoff. The rows-differing lines are the shift test, and the two MAE lines show how much the leak flatters the score."""
import numpy as np
import pandas as pd

from lib.electricity import load
from lib.features import fit_window_mae, lag_values
from lib.settings import EVAL_CUTOFFS, HORIZON, LEAD, show

y = load([LEAD]).set_index("ds")["y"].asfreq("h")  # asfreq gives every hour a row, so a missing hour is NaN, not skipped
HONEST, LEAKY = [24, 168], [1, 24, 168]  # HONEST lags are both at least the horizon; LEAKY adds lag 1
CUTOFFS = pd.to_datetime(EVAL_CUTOFFS)


def shift_test(lag):
    """Target hours where the lag differs between the history cut at each cutoff and the whole series, and how many."""
    compared = differing = 0
    for c in CUTOFFS:
        targets = pd.date_range(c, periods=HORIZON, freq="h")  # the 24 hours each forecast must predict
        honest = lag_values(y[y.index < c], lag, targets)  # readings before the cutoff only, as the forecast has them
        whole = lag_values(y, lag, targets)  # the same lag read from the whole series, as a table built once would read it
        same = (honest == whole) | (np.isnan(honest) & np.isnan(whole))  # NaN never equals NaN, so the second term counts two missing readings as the same
        compared += len(targets)
        differing += int((~same).sum())
    return compared, differing


show("cutoffs", len(CUTOFFS))
show("rows compared", shift_test(1)[0])  # the first value is the hours compared, the same for every lag
for lag in (1, 24, 168):
    show(f"rows differing for lag {lag}", shift_test(lag)[1])
show("MAE honest, lags 24 and 168", fit_window_mae(y, HONEST, CUTOFFS, leaky=False))  # the table holds the lags and the hour of day; leaky=False reads the target lags from the cut history
show("MAE leaky, lags 1, 24 and 168", fit_window_mae(y, LEAKY, CUTOFFS, leaky=True))  # leaky=True reads the whole series
```

Output:

```text
cutoffs: 3
rows compared: 72
rows differing for lag 1: 69
rows differing for lag 24: 0
rows differing for lag 168: 0
MAE honest, lags 24 and 168: 5.36
MAE leaky, lags 1, 24 and 168: 2.87
```

Only the lag of 1 fails the shift test. Of the 72 target hours compared, 69 read a different value from the cut history than from the whole series: every hour of each window but the first. The lags of 24 and 168 differ on 0 and 0 rows. The lag of 1 agrees only on the first hour of each window, because that hour reads the last reading before the cutoff, which the forecast does have.

The leaky model looks almost twice as good: its MAE is 2.87 kWh, against 5.36 kWh for the honest model. The gap is the leak. Its lag of 1 reads the hour before the target, which for most target hours is an hour of the forecast window itself. The honest model never sees that hour, so its error is the one that production will show.

**Working rule:** Run the shift test on every lag and every window feature before a model trains on it, and treat any differing row as a leak until you have shown otherwise. When an error falls sharply after you add a feature, check that feature first, before you credit it.

## What goes wrong in production {#what-goes-wrong-ch5}

A holiday list that stops at the end of the data reads zero for every later holiday. The flag then stays off for every day of the forecast year, and the model forecasts holidays as ordinary days. The first holiday shows up as a large miss on the dashboard. The holiday count printed by the example is the check: a list for a new year that prints zero holidays is missing that year. A later version of `holidays` can also change its calendar, so pin the version, as the book does, and rerun the count after any upgrade.

A lag shorter than the horizon, carried over from an hourly model used for a day-ahead forecast, scores well in the backtest, because the backtest was built from the whole series. In production, the later hours read the model's own forecasts. Compare the live error by hour of the day-ahead forecast with the backtest's own error by hour. A model without the leak should show about the error the backtest showed at each hour. Error that grows with the hour of the horizon, beyond that, is the sign of feedback from the model's own forecasts.

The clock rule can still go wrong in the pipeline. Masked hours make some forecast features missing, and a pipeline can drop those rows, or fill those hours, without saying so. Print the masked hours and the rows they affect every time the history changes.

A pooled score can hide the typical client. In [lgbm-feature-sets](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/lgbm-feature-sets.py), the unscaled lags-only model's pooled MAE, 304.03 kWh, is below seasonal naive's 305.21 kWh, but its mean per-client MASE is 1.115, against 0.934. The supplier buys power for each client, so the per-client figure is the one that decides a client's forecast. A supplier that reports only the pooled MAE will sell the wrong forecast to the clients the pooled figure hides. Report the per-client MASE next to the pooled figure, and look at the worst client first.

Results can also depend on the machine. LightGBM's [parameter reference](https://lightgbm.readthedocs.io/en/v4.7.0/Parameters.html) says `deterministic` gives stable results for the same data and the same parameters on the CPU device type, and warns that results can differ with other LightGBM versions, with binaries compiled by other compilers, or on other systems. The book sets `deterministic`, row-wise histograms, one thread and a fixed seed, under LightGBM 4.7.0. Those settings make the output repeatable on one machine with one thread count: the output was identical in three runs on the recording machine. They do not promise agreement across machines, and the book has not tested that. The scores are quoted rounded, and rounding does not make two machines agree. If a release machine prints different values, the recorded output needs a fresh recording, and the prose needs the same check.

**Working rule:** Recheck the holiday count for each forecast year, the masked hours and the rows they reach, and the lag lengths whenever the data or a library changes. Run the shift test on every lag, and read the per-client MASE beside the pooled MAE. A check that changes is a reason to stop and read the cause before the next forecast is published.

## Decisions this chapter made {#decisions-ch5}

The choices below are the ones a reader would have to make differently to change the results. Each one is the book's choice, open to revision when a later edition measures something else.

- The handed-on model uses lags 24 and 168 only. Lag 1 appears only in the leak check, never in a forecast.
- The calendar set is `hour`, `dayofweek` and `month`. It is measured and not kept: on the per-client view it is worse than lags alone. There is no separate weekend flag, because the weekday carries the weekend.
- The holiday flag is measured only on top of the calendar set, on all the decision windows. It is not kept, because the set that holds it is still worse than lags alone. Chapter 6 gets the lags-only set.
- Holidays come from the `holidays` package for Portugal, for the two years in the data. The `is_holiday` function lives in `examples/lib/features.py`.
- Each client's target is scaled by default, with `make_forecaster(scaled=True)`. The scaler is fitted on each window's history only. The unscaled run is printed once, to show the failure that scaling corrects.
- Each window's masked clock-change hours are filled from that window's history before the fit, so no row is dropped for them and LightGBM never receives a missing value.
- The LightGBM settings are `LGBM_PARAMS` in `examples/lib/features.py`: deterministic, row-wise histograms, one thread and seed 0. They sit there, not in `settings.py`, which this chapter leaves unchanged.
- The MASE scale is `scoring.book_scale`. It starts at each client's first reading and reads the readings before the first decision window, Apr 18, 2014.
- On the decision windows, the lags-only model beats seasonal naive on mean per-client MASE. The chapter does not tune the model to change that result.
- The feature sets are chosen on the decision windows, not on the standard cutoffs or the held-out week. The holiday windows end on Dec 8, so the holiday's effect on Dec 25 is not measured in this chapter. Whether calendar features pay on a longer history is also not measured here.
- The leak check uses `MT_014`, one client, because the leak is a property of the lag and not of the client mix, so its target is not scaled. The feature sets use `SUBSET8`.
- Chapter 6 reuses `make_forecaster`, `mlforecast_windows`, `fit_window_mae`, `lag_table` and `is_holiday` from `examples/lib/features.py`. `make_forecaster` scales by default.
- The lesson is an order of tests: the shift test runs before any score, and the per-client MASE decides which features stay.

# Forecast many clients with one model

You finish this chapter with one global model for all thirty clients, a start rule that keeps the late starters' leading zeros out of it, and a score that sets it beside seasonal naive for each client on the same cutoffs. The supplier's day-ahead purchase is a total that [Chapter 7](#why-the-total-needs-a-check) builds from thirty client forecasts. A per-client model needs thirty fits for every window, one for each client. The clients share a daily rhythm, so one model fitted on all of them can learn that rhythm once.

[Chapter 5](#measure-what-each-feature-adds) fitted its model on the eight clients of [`SUBSET8`](#the-case-and-its-data). Fitting one model on all thirty raises three choices that decide whether its score can be trusted: whether to give it a client identifier, how to handle the three late starters (clients whose meters begin partway through the data), and how to score thirty forecasts on one scale. Four examples carry the chapter. Each prints its counts first, so you can check that the model covered the clients it should before you read any error.

## One model for many clients {#one-model-for-many-clients}

[Montero-Manso and Hyndman (2021)](https://doi.org/10.1016/j.ijforecast.2021.03.004), in *Principles and algorithms for forecasting groups of time series: Locality and globality*, show that a global model is no more restrictive than a local one, because both can produce the same forecasts. They compare global methods, which fit one model across a group of series, with local methods, which fit each series on its own. A global model fits one set of parameters to many series in a single pass, so every client contributes to what it learns. Their abstract starts from the view that for groups of similar series, global methods outperform the more established local methods, and it notes that later results in that literature show good performance of global models even in heterogeneous datasets. It also reports that global linear models can give competitive accuracy with two orders of magnitude fewer parameters than local methods. The claims in this paragraph come from that abstract. Bojer and Meldgaard (2020) found that global ensemble models and gradient-boosted trees did well in six Kaggle forecasting competitions ([arXiv:2009.07701](https://arxiv.org/abs/2009.07701)). Those are not the supplier's data, so these sources are a reason to try the approach. They are not evidence about this case. The supplier's clients have the same daily shape and different sizes, so the test has to be run on them.

The data is held in long format: one row per client and hour, with three columns. `unique_id` names the client, `ds` is the hour, and `y` is the kilowatt hours used in that hour. Each row carries its own lags, the readings a day and a week earlier. [Chapter 5](#measure-what-each-feature-adds) found that calendar and holiday features did not lower the error on these eight weeks of history, so this chapter uses lags only, at 24 and 168 hours. Both lags are at least the 24-hour horizon, so neither reads the day being forecast.

The lags cost rows. A lag of 168 hours reads a week back, so the first week of each client's history has no row with every lag available. Chapter 5 showed that `MLForecast` drops such rows from training by default. A client's training rows are therefore fewer than the hours in its history. The forecast itself is unaffected, because each forecast hour reads only readings from before its cutoff.

Clients differ in size, and a squared-error loss over raw kilowatt hours lets the largest set the splits, while the small ones carry little weight. So each client's target is scaled before the fit. The scale is `LocalStandardScaler` from mlforecast 1.1.0. It gives each client its own mean and standard deviation, and the model learns the scaled values:

`z = (y - mean) / standard deviation`

The mean and deviation come from that client's history before the cutoff only, and the forecasts are converted back with the same two numbers. The library's source shows where this happens. `MLForecast.fit` reaches `TimeSeries._fit` in `mlforecast/core.py`, which fits the scaler on each client's history before the cutoff. That history includes the first week, whose rows the lags then drop from training. `TimeSeries.predict` in the same file inverts the scaling. The class is `_BaseLocalScaler` in `mlforecast/target_transforms.py`. [Chapter 5](#measure-what-each-feature-adds) compared the scaled and unscaled targets on `SUBSET8`, and the unscaled model did worse than seasonal naive. This chapter keeps the scaled target and does not repeat that comparison.

`make_forecaster` in the shared module `examples/lib/features.py` builds one `MLForecast` with one `LGBMRegressor`, set by `LGBM_PARAMS`. The model is deterministic, uses one thread and a fixed seed, and runs under LightGBM 4.7.0. The module `examples/lib/panel.py` fits that forecaster once per window, on the panel's history before the window's cutoff. It fills any clock-change hour from that same history, as [Chapter 3](#mark-clock-change-hours-as-missing) does, and returns a forecast for every client.

**Working rule:** Fit one global model when your series share a shape and you must forecast many of them on each window. Scale each series' target with statistics from that window's history only, use only lags at least as long as the horizon, and fit once per window, not once per series. Then score the model per series against a baseline on the same windows, as the sections below do, before you trust its pooled error.

## Identify each client {#identify-each-client}

Once the rows share one table, the model no longer knows which client a row belongs to, unless you tell it. Scaling removes each client's mean and spread, so what remains is mostly the shape of the day. A client identifier puts the client in the table as a static categorical feature. The trees can then split on the client and learn that client's own habits. The cost is one more split to learn for each client, and a client with no history has nothing to learn from. A new client therefore needs a refit before the model can forecast it.

The rule for keeping an identifier is the one [Chapter 5](#measure-what-each-feature-adds) sets for every feature: keep it only if it lowers the mean per-client MASE on decision windows that the standard cutoffs do not use. The held-out week runs from Dec 25 to Dec 31, 2014, and it has seven daily cutoffs. The three standard cutoffs, Dec 29 to Dec 31, 2014, are the reported scores, so the decision does not use them, and it does not use Dec 25 to Dec 28, 2014 either. The [decision windows](#choose-on-decision-windows) are four ordinary days before the held-out week: cutoffs on Nov 25, Nov 26, Dec 2 and Dec 3, all in 2014, each forecasting 24 hours. None of their scored hours falls on a Portuguese holiday. The one holiday of 2014 between these windows and the held-out week, Dec 8, 2014, is not among them. The calendar lists no holiday on Dec 1, 2014.

Before the identifier goes into the model, test it on windows the reported scores never touch. The check fits the model twice on each decision cutoff, with and without the identifier, and refuses to start if a scored hour is a holiday or in the held-out week.

Example `client-identifier-check` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Tests whether a client identifier lowers the global model's error on four ordinary decision windows, before the
held-out week, on the book's MASE scale. The point rests on the mean per-client MASE lines and on `identifier kept`."""
from lib.electricity import load
from lib.panel import ALL30, DECISION_CUTOFFS, check_decision_hours, mae_change, panel_windows
from lib.quirks import mask_clock_changes
from lib.scoring import actual_hours, book_scale, report_scores, score_windows
from lib.settings import show

raw = load(ALL30)                                             # all thirty clients, not the eight of SUBSET8
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]        # clock-change hours are set to missing, so the scores leave them out
cutoffs = DECISION_CUTOFFS
actual = actual_hours(df, cutoffs)
holiday_hours = check_decision_hours(actual)  # raises before any fit if a scored hour is a holiday or in the held-out week

plain = panel_windows(df, cutoffs, client_id=False, name="plain")               # the model without the identifier
identified = panel_windows(df, cutoffs, client_id=True, name="identifier")      # the same model, with the client as a feature
models = ["plain", "identifier"]
# the MASE scale reads only the readings before the first decision cutoff, so the scored hours cannot leak into it
scored, per_client = score_windows(actual, plain.merge(identified, on=["unique_id", "ds"], validate="one_to_one"),
                                   models, book_scale(raw, cutoffs[0]))
lowers = per_client["identifier"] < per_client["plain"]
size = raw.groupby("unique_id")["y"].mean().sort_values(ascending=False)   # mean kWh per client, largest first
contrib = mae_change(scored, "plain", "identifier")           # each client's share of the pooled MAE change, kWh; the shares sum to the total

show("first decision cutoff", f"{cutoffs[0]:%Y-%m-%d}")
show("last decision cutoff", f"{cutoffs[-1]:%Y-%m-%d}")
show("clients", scored["unique_id"].nunique())
show("windows", len(cutoffs))
show("scored hours", len(scored))
show("decision hours on a holiday", holiday_hours)
report_scores(scored, per_client, models)
show("clients where the identifier lowers MASE", int(lowers.sum()))
show("of the 15 largest clients, where the identifier lowers MASE", int(lowers[size.index[:15]].sum()))
show("of the 15 smallest clients, where the identifier lowers MASE", int(lowers[size.index[-15:]].sum()))
show("largest client", size.index[0])
show("pooled MAE change from the largest client, kWh", contrib[size.index[0]])
show("pooled MAE change, all clients, kWh", contrib.sum())
# the decision rule keeps the identifier only if the mean per-client MASE is lower, not the pooled MAE
show("identifier kept", "yes" if per_client["identifier"].mean() < per_client["plain"].mean() else "no")
```

Output:

```text
first decision cutoff: 2014-11-25
last decision cutoff: 2014-12-03
clients: 30
windows: 4
scored hours: 2880
decision hours on a holiday: 0
MAE plain: 70.72
MAE identifier: 66.81
RMSE plain: 316.50
RMSE identifier: 303.68
mean MASE plain: 0.77
mean MASE identifier: 0.75
clients where the identifier lowers MASE: 23
of the 15 largest clients, where the identifier lowers MASE: 10
of the 15 smallest clients, where the identifier lowers MASE: 13
largest client: MT_370
pooled MAE change from the largest client, kWh: -3.80
pooled MAE change, all clients, kWh: -3.92
identifier kept: yes
```

The identifier lowers the mean per-client MASE, from 0.77 to 0.75, and it lowers the MASE of 23 of the thirty clients.

On these windows the pooled MAE falls too, from 70.72 kWh to 66.81 kWh. The two errors weigh clients differently, because the pooled MAE is weighted toward the largest.

The pooled change of -3.92 kWh is almost all from one client, MT_370, which accounts for -3.80 kWh of it.

The per-client MASE gives every client the same weight, and the identifier helps across sizes. It lowers the MASE of 10 of the fifteen largest clients and of 13 of the fifteen smallest. It does not help only the large clients.

The printed verdict is yes, so the identifier is kept and the final model uses it. The margin is small. Other windows could give the opposite sign by a similar small amount, so the verdict is close and depends on the windows. A reader with other windows should measure it again before trusting it.

The identifier has a cost at the standard cutoffs. There, the final model's mean per-client MASE is 0.91 with the identifier, against 0.98 without it, but its pooled MAE is 122.81 kWh, against 119.03 kWh without it. Chapter 7 has to check its total against that. The rest of the chapter uses the model with the identifier.

**Working rule:** Treat a client identifier as one more feature under [Chapter 5's rule](#measure-what-each-feature-adds): keep it only if it lowers the mean per-client MASE on the decision windows, with the pooled MAE beside it. If the decision is close, say so, and measure it again on other windows. If the identifier wins, write down its cost: a new client has no learned split until it has history, and the model must be refitted before it can forecast that client.

## Apply the start rule to every client {#apply-the-start-rule-to-every-client}

A series begins at its first reading. Before that, the meter reports zeros, and those zeros are not low use. [Chapter 4](#meet-the-late-starters) met this for one client. With thirty clients, three are late starters. `MT_039` first reads on 2013-10-03, `MT_179` on 2013-12-20, and `MT_115` on 2014-01-14. A model fitted on their leading zeros learns a level near zero for them. The [start rule](#start-each-series-at-its-first-reading) drops every reading before a client's first reading. It finds that reading in the rows before the cutoff only.

The rule refuses a cutoff in two cases only: a client has no non-zero reading before the cutoff, or it has fewer than 48 hours of history since its first reading. It does not skip the client, because a skipped client leaves the total short. The rule does not catch a gap or a run of zeros in the middle of a series, and a meter replaced under the same id passes too, because its zeros are not leading. The scale catches a gap, because it refuses a series that is not one unbroken hourly series. Nothing in the forecast checks for one, so a gap there shows only as fewer training rows.

The leak check asks whether the rule reads only the past and refuses the right cutoffs. It runs nine cutoffs, refused or not, and poisons each from the cutoff on, once with zeros and once with a value no meter could record. The history must not change, and every accepted cutoff must hold all thirty clients.

Example `start-rule-leak-check` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Checks that the start rule reads only rows before each cutoff, refuses the cutoffs the data's flag predicts, and
removes only leading zeros, on nine cutoffs. The same checks run on a deliberately broken history function, and each
must fail there; the point rests on the refused, changed and missing counts. The checks are start_rule_checks in lib/panel.py."""
import pandas as pd

from lib.electricity import load
from lib.panel import ALL30, FEBRUARY_CUTOFFS, leaky_history, panel_history, start_rule_checks
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
# nine cutoffs: three around MT_179's and MT_115's first readings (refused), the February cutoffs and the standard cutoffs
CHECKED = pd.DatetimeIndex(["2013-12-15", "2014-01-01", "2014-01-15", *FEBRUARY_CUTOFFS, *EVAL_CUTOFFS])


def report(prefix, r):
    show(f"{prefix}cutoffs checked", len(CHECKED))
    show(f"{prefix}cutoffs refused by the start rule", r["refused"])
    show(f"{prefix}cutoffs expected to be refused, from the flag", r["expected"])
    show(f"{prefix}cutoffs where the refusal differs from the flag", r["mismatch"])
    show(f"{prefix}cutoffs accepted", r["accepted"])
    show(f"{prefix}accepted cutoffs with a client missing from the history", r["missing"])
    show(f"{prefix}cutoffs whose history changes when the future is poisoned", r["changed"])


# the book's history function, with the count of rows the rule removes switched on
real = start_rule_checks(raw, CHECKED, panel_history, removed_rows=True)
report("", real)
show("rows the start rule removes, accepted cutoffs", real["removed"])
show("removed rows that are not zero", real["removed_nonzero"])
# the deliberately broken history, which reads each first reading from the whole series, must fail every check
report("broken version, ", start_rule_checks(raw, CHECKED, leaky_history))
```

Output:

```text
cutoffs checked: 9
cutoffs refused by the start rule: 3
cutoffs expected to be refused, from the flag: 3
cutoffs where the refusal differs from the flag: 0
cutoffs accepted: 6
accepted cutoffs with a client missing from the history: 0
cutoffs whose history changes when the future is poisoned: 0
rows the start rule removes, accepted cutoffs: 2064
removed rows that are not zero: 0
broken version, cutoffs checked: 9
broken version, cutoffs refused by the start rule: 1
broken version, cutoffs expected to be refused, from the flag: 3
broken version, cutoffs where the refusal differs from the flag: 2
broken version, cutoffs accepted: 8
broken version, accepted cutoffs with a client missing from the history: 2
broken version, cutoffs whose history changes when the future is poisoned: 2
```

The real rule passes all three checks. Its refusals match the data's own flag, which uses each first reading and the 48-hour rule: the rule refuses 3 of the nine cutoffs, the flag predicts 3, and 0 cutoffs differ from it.

Poisoning the future changes no history. Its history changes at 0 cutoffs, and 0 accepted cutoffs are missing a client.

The rule removes leading zeros and nothing else. Across the accepted cutoffs it drops 2064 rows, and 0 of them are not zero.

The same checks run on a deliberately broken version, which reads each first reading from the whole series, the leak the checks are written for. It fails each of them. Its refusals differ from the flag at 2 cutoffs, its history changes at 2, and 2 accepted cutoffs drop a client without a message. So a pass on the real rule means something.

The model runs with the start rule and without it, on the standard cutoffs and then the February cutoffs. The counts that differ, and each run's error, show what the rule does.

Example `global-model-raw-starts` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Runs the global model with the start rule and without it, on the February cutoffs, where a late starter's zeros enter
the history, and on the standard cutoffs, where the rule changes no row. The point rests on the February mean MASE lines
and the count of forecast hours that differ; the scale lines compare the book's scale with a labelled counterfactual."""
import numpy as np
import pandas as pd

from lib.electricity import load
from lib.panel import (ALL30, FEBRUARY_CUTOFFS, IDENTIFIER_KEPT, LATE_STARTERS, late_starter_facts, mae_by_group,
                       mase_by_run, scale_facts, with_and_without_start_rule)
from lib.quirks import mask_clock_changes
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]        # clock-change hours are set to missing, so the scores leave them out
for label, value in late_starter_facts(raw, df).items():
    show(label, value)

standard = pd.to_datetime(EVAL_CUTOFFS)                       # the three standard cutoffs
# two runs of the same model on the same cutoffs, differing only in their histories
scored = with_and_without_start_rule(df, standard, client_id=IDENTIFIER_KEPT)
# compared with isclose, so a rounding difference between two runs does not count as a changed forecast
show("forecast hours that differ, standard", int((~np.isclose(scored["start"], scored["raw"])).sum()))

# the February cutoffs, where the zeros of MT_115 and MT_179 enter the history
feb = with_and_without_start_rule(df, FEBRUARY_CUTOFFS, client_id=IDENTIFIER_KEPT)
show("scored hours, february", len(feb))
show("forecast hours that differ, february", int((~np.isclose(feb["start"], feb["raw"])).sum()))
for model in ["start", "raw"]:
    for group, value in mae_by_group(feb, model).items():
        show(f"MAE {group}, {model}, february", value)
# both runs on one scale, read from the first February cutoff
feb_mase = mase_by_run(raw, feb, ["start", "raw"])
show("clients in MASE, february", int(feb_mase["start"].notna().sum()))
for uid in LATE_STARTERS:
    show(f"MASE {uid}, start, february", feb_mase.loc[uid, "start"])
    show(f"MASE {uid}, raw, february", feb_mase.loc[uid, "raw"])
for run in ["start", "raw"]:
    show(f"mean MASE, {run}, february", feb_mase[run].mean())

# 'counterfactual raw' keeps the zeros before each first reading, a scale the book does not use; 'start rule' is the book's
for label, value in scale_facts(raw, standard, IDENTIFIER_KEPT).items():
    show(label, value)
```

Output:

```text
first reading MT_039: 2013-10-03
first reading MT_115: 2014-01-14
first reading MT_179: 2013-12-20
zero share of history MT_115, 2014-02-01: 68
zero share of history MT_179, 2014-02-01: 23
zero share of history MT_115, 2014-02-15: 43
zero share of history MT_179, 2014-02-15: 0
zero share of history MT_115, 2014-02-28: 20
zero share of history MT_179, 2014-02-28: 0
forecast hours that differ, standard: 0
scored hours, february: 2160
forecast hours that differ, february: 2160
MAE late starters, start, february: 4.60
MAE other clients, start, february: 77.53
MAE MT_039, start, february: 4.64
MAE MT_115, start, february: 4.62
MAE MT_179, start, february: 4.54
MAE late starters, raw, february: 4.60
MAE other clients, raw, february: 77.95
MAE MT_039, raw, february: 4.57
MAE MT_115, raw, february: 4.68
MAE MT_179, raw, february: 4.53
clients in MASE, february: 30
MASE MT_039, start, february: 0.78
MASE MT_039, raw, february: 0.77
MASE MT_115, start, february: 0.90
MASE MT_115, raw, february: 0.91
MASE MT_179, start, february: 0.54
MASE MT_179, raw, february: 0.54
mean MASE, start, february: 0.70
mean MASE, raw, february: 0.71
MASE scale MT_039, start rule: 5.21
MASE scale MT_039, counterfactual raw: 3.28
MASE scale MT_115, start rule: 4.58
MASE scale MT_115, counterfactual raw: 2.22
MASE scale MT_179, start rule: 5.91
MASE scale MT_179, counterfactual raw: 3.04
mean MASE global, counterfactual raw scale, standard cutoffs: 1.01
mean MASE global, book scale, standard cutoffs: 0.91
```

At the standard cutoffs the rule matters less than the late starters might suggest. The three standard cutoffs have 8 weeks of history, reaching back to early November, and no client's history in that window holds a zero from before its first reading. So the rule changes no training row there, and 0 of the 2160 forecast hours change. The rule still changes the score through the MASE scale.

The February cutoffs, which Chapter 4 used for `MT_115`, are where the zeros enter the history. `MT_115`'s history is 68 percent zeros on Feb 1, 2014, 43 percent on Feb 15 and 20 percent on Feb 28. `MT_179`'s history is 23 percent zeros on Feb 1 and none after.

Here the forecasts change, because one model learns from all thirty clients at once: 2160 of the 2160 forecast hours in these three windows differ. Both runs are scored on the one book scale, read from Feb 1, 2014, so only the histories differ. The mean MASE is 0.70 with the rule and 0.71 without it.

Each late starter's MASE moves very little, and the direction is mixed. `MT_039` scores 0.78 with the rule and 0.77 without it, and `MT_115` scores 0.90 with the rule and 0.91 without.

`MT_179` scores 0.54 with the rule and 0.54 without it, the same to two decimals.

The other 27 clients' MAE moves from 77.95 kWh without the rule to 77.53 kWh with it. The effect is small and mixed, and three windows are too few to call it. We keep the rule because the zeros are not demand, and because the scale depends on it.

You could instead mark the zeros as missing, as the clock rule does for the two clock-change hours in [Chapter 3](#mark-clock-change-hours-as-missing). That would be the wrong fix for this data. A zero before a first reading is the meter being silent, not a reading that is missing, and a missing value would still be a row the model learns from. The start rule removes the rows instead, and its check is simple to state: it removes only rows before the first reading, and every one of them is zero.

**Working rule:** Apply the start rule to every series that has leading zeros, and find each first reading from the rows before the cutoff. When the rule refuses a cutoff, stop the run and name the client and the cutoff. Do not fill the zeros, and do not publish a panel with a client missing. Judge the rule by the scale and by the windows where zeros enter the history, and expect the MAE effect to be small and mixed.

## Score thirty clients on the same cutoffs {#score-thirty-clients-on-the-same-cutoffs}

The three standard cutoffs are Dec 29 to Dec 31, 2014, each at 00:00 on that day, with 24 hours forecast from each. For thirty clients that is 72 scored hours per client. Seasonal naive is the baseline [Chapter 2](#three-baselines-to-beat) set. It takes each cutoff's own history, the 8 weeks before it, and copies the last day's readings forward. At the standard cutoffs no client's history holds a leading zero, so the global model and seasonal naive read the same readings. The comparison is fair because the inputs match, not because the methods are alike.

The headline is the mean per-client MASE, the measure [Chapter 2](#scale-errors-with-mase) defines. The book uses one scale per client per backtest, the rule [Chapter 2](#take-the-mase-scale-from-history) sets, and the function `book_scale` in `examples/lib/scoring.py` builds it. The [scale](#scale-errors-with-mase) is the mean absolute change between each reading and the reading one day earlier, taken over every reading before the first scored cutoff. A late starter is counted from its first reading, and the clock-change readings are masked out. A client with no non-zero reading before the cutoff has no scale, so it is left out of MASE and reported, not treated as an error. Because the scale reads only readings before the scored windows, it is known at every cutoff and cannot leak. Each client counts once in the mean, so one large client cannot set the headline. The pooled MAE and RMSE follow as second lines. They show the error in kilowatt hours that the supplier pays, and they are weighted toward the large clients.

The scale is part of the score. `MT_115`'s scale is 2.22 kWh if the zeros before its first reading are kept, and 4.58 kWh with the book's rule, which counts it from its first reading.

`MT_039` and `MT_179` move the same way. `MT_039`'s scale rises from 3.28 to 5.21 kWh, and `MT_179`'s from 3.04 to 5.91 kWh.

On the standard cutoffs the same forecasts score a mean MASE of 1.01 with the scale that keeps the zeros, and 0.91 with the book's scale. That difference is the scale, not the model. A score computed with a different scale rule is not comparable with one computed with the book's rule, so every score should record the rule it used.

The score below sets the global model beside seasonal naive on the three standard cutoffs.

Example `global-model-thirty-clients` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Scores one global LightGBM model, with the identifier that client-identifier-check kept, against seasonal naive on the
three standard cutoffs, per client on the book's MASE scale. The point rests on the mean per-client MASE lines; the model
without the identifier is printed for reference only."""
import pandas as pd
from statsforecast import StatsForecast
from statsforecast.models import SeasonalNaive

from lib.backtest import forecast_windows
from lib.baselines import SEASON
from lib.electricity import load
from lib.panel import ALL30, IDENTIFIER_KEPT, panel_windows
from lib.quirks import mask_clock_changes
from lib.scoring import actual_hours, book_scale, left_out_facts, report_scores, score_windows
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]        # clock-change hours are set to missing, so the scores leave them out
cutoffs = pd.to_datetime(EVAL_CUTOFFS)                        # the three standard cutoffs, Dec 29 to Dec 31, 2014
actual = actual_hours(df, cutoffs)

# seasonal naive from each cutoff's own history: the baseline of Chapter 2
naive = forecast_windows(StatsForecast(models=[SeasonalNaive(season_length=SEASON)], freq="h", n_jobs=1), df, cutoffs)
global_fc = panel_windows(df, cutoffs, client_id=IDENTIFIER_KEPT)   # the identifier setting that client-identifier-check kept
plain_fc = panel_windows(df, cutoffs, client_id=False, name="plain")  # the same model without the identifier, for reference
forecasts = naive.merge(global_fc, on=["unique_id", "ds"], validate="one_to_one").merge(
    plain_fc, on=["unique_id", "ds"], validate="one_to_one")
models = ["SeasonalNaive", "global", "plain"]
# the book's scale, read from the readings before the first cutoff only
scored, per_client = score_windows(actual, forecasts, models, book_scale(raw, cutoffs[0]))
beats = per_client["global"] < per_client["SeasonalNaive"]    # per client: True where the global model's MASE is lower

show("clients scored", scored["unique_id"].nunique())
show("windows", len(cutoffs))
show("scored hours", len(scored))
# a client with no non-zero reading before the first cutoff is left out of MASE, and is not an error
for label, value in left_out_facts(raw, df, cutoffs[0]).items():
    show(label, value)
show("clients in MASE", per_client["global"].notna().sum())
show("clients where global beats seasonal naive", int(beats.sum()))
show("clients where seasonal naive wins", ", ".join(sorted(per_client.index[~beats])))
report_scores(scored, per_client, models)
```

Output:

```text
clients scored: 30
windows: 3
scored hours: 2160
clients left out of MASE: 0
clients with no reading before 2013-12-15, check: 2
clients with any row before 2013-12-15, broken count: 30
clients in MASE: 30
clients where global beats seasonal naive: 23
clients where seasonal naive wins: MT_001, MT_128, MT_141, MT_179, MT_217, MT_268, MT_294
MAE SeasonalNaive: 148.26
MAE global: 122.81
MAE plain: 119.03
RMSE SeasonalNaive: 780.48
RMSE global: 678.81
RMSE plain: 657.63
mean MASE SeasonalNaive: 1.18
mean MASE global: 0.91
mean MASE plain: 0.98
```

The example scores 30 clients on 3 windows, 2160 scored hours, and it counts 0 clients left out of MASE, which is zero: every client has a non-zero reading before the first cutoff, so every client is in the MASE table.

The count is only as good as its rule. At a cutoff of Dec 15, 2013, 2 clients have no reading yet, and counting any row before the cutoff would give 30, which reports no gap.

One global model beats seasonal naive on these cutoffs, for most clients and in total: its mean per-client MASE is 0.91 against 1.18, and it wins for 23 of the thirty clients. It loses for MT_001, MT_128, MT_141, MT_179, MT_217, MT_268, MT_294, and those clients are the first place to look when the model is refreshed.

The pooled figures point the same way. The MAE is 122.81 kWh against 148.26 kWh, and the RMSE is 678.81 kWh against 780.48.

The model without the identifier, printed for reference, has a mean MASE of 0.98 and an MAE of 119.03 kWh, which is the trade-off the [identifier section](#identify-each-client) describes.

The printed figures are means and counts, and a mean can hide one client that is far worse than the rest. When a figure changes, open the per-client table first.

The chapter does not explain why seasonal naive wins for those clients, and that is a reason not to report only the pooled figure. The result covers three windows in one week of one season, so it describes these cutoffs, not every day of the year. The longer test is the seven-window held-out week, which the rolling example in [Chapter 10](#log-every-forecast-before-actuals) uses.

**Working rule:** Score a global model per client on the same cutoffs as the baseline, with the MASE scale fixed before the first scored cutoff. Report the mean per-client MASE beside seasonal naive, and print the counts of clients and scored hours so a reader can check them. State any client left out of MASE and why. Read the pooled MAE and RMSE as the cost in kilowatt hours, not as the measure of model quality.

## Budget the run time {#budget-the-run-time}

A backtest costs fits, and the count is what to check before you run one. A [per-client model](#why-a-statistical-model-comes-next) is fitted once for each client, each window and each method, so the count is clients times windows times methods. Thirty clients over three windows is ninety fits for one per-client method. A global model is fitted once for each window and method, however many clients it covers, so the same three windows cost three fits. More clients make each global fit larger, but they do not add fits.

The book sets a subset rule for the expensive per-client methods. [`SUBSET8`](#the-case-and-its-data) is the eight-client subset that [Chapter 4](#fit-exponential-smoothing) runs its per-client statistical models on. A global model covers all thirty in one fit, so it runs on all thirty clients, as `ALL30`. The subsets are fixed for the whole book, so a number in one chapter can be set beside the next. The window count follows the same logic. Expensive methods use three windows, and cheap methods can use more. Each window of the identifier check costs two global fits, one with the identifier and one without, so its four windows cost eight fits.

The two kinds of cost move differently when the number of clients grows. A per-client method gets more fits, one for each new client. A global model gets bigger fits, because each one reads more rows, and it keeps the same number of fits.

The book does not quote timings. The examples run with one thread, so their output repeats on the machine that recorded the book's outputs, and that choice costs run time. A time depends on the machine, and a reader's machine may differ. Count the fits, not the seconds.

**Working rule:** Before you run a backtest, count its fits: clients times windows times methods for a per-client method, and windows times methods for a global one. If the count is over budget, cut the subset or the window count, never the scored windows, and write the cut down beside the score. Then a reader can compare like with like.

## What goes wrong in production {#what-goes-wrong-ch6}

A run can stop. A client with no non-zero reading before a cutoff, or with under 48 hours of history since its first reading, makes the start rule refuse that cutoff. The rule raises an error instead of skipping the client, because a skipped client leaves the total short, and [Chapter 7](#sum-the-client-forecasts) sums the clients to make the total. In production the run should stop, name the client and the cutoff, and wait for a person. The leak check shows the same refusals in the book's data.

One model moves every client. At the February cutoffs, the late starters' zeros changed 2160 forecast hours, and the other 27 clients' MAE moved as well, as the [start-rule section](#apply-the-start-rule-to-every-client) shows. Any change to one client's history, to a rule, or to a feature changes forecasts across the panel. After any change, check every client, not only the one you changed.

The scale is part of the score. The same forecasts score a mean MASE of 1.01 or 0.91, depending on whether the scale keeps the zeros before a first reading. The [scoring section](#score-thirty-clients-on-the-same-cutoffs) gives the scales. Record the rule with every score, and treat a change to the rule as a change to the score.

The decisions can drift. The identifier decision rests on four ordinary windows that lie outside the held-out week, and its verdict is close. When clients join, leave or change size, the decision should be made again.

A missing client does not appear as an error in the table. A panel of twenty-nine clients is still a valid table, and a forecast of it still looks like a forecast. A run that scores fewer clients or hours than expected, or a count of clients where the global model beats seasonal naive that changes sharply between runs, is a reason to stop. So is a per-client MASE that jumps for one client while the pooled MAE stays flat.

**Working rule:** Print and check the counts (clients, windows, scored hours, refused cutoffs, clients left out of MASE) on every refresh, before any score is read. Then report the mean per-client MASE beside seasonal naive. Stop the run on any refusal, and do not publish a total built from a short panel.

## Decisions this chapter made {#decisions-ch6}

[Chapter 7](#why-the-total-needs-a-check) starts from what this chapter hands over: thirty client forecasts for each standard cutoff, made by one model with the client identifier and the start rule applied, and scored on the book's one MASE scale. The identifier is kept by the decision in `client-identifier-check`, and `examples/lib/panel.py` holds that verdict in `IDENTIFIER_KEPT`. The pooled error at these cutoffs is worse with the identifier, so Chapter 7 should check the total before it relies on the client forecasts.

- One global LightGBM model covers all thirty clients, with lags 24 and 168, no calendar or holiday features, and each client's target scaled from that window's history only, the scaling [Chapter 5](#measure-what-each-feature-adds) settles on.
- A client identifier is kept. The decision used four ordinary windows, from Nov 25 to Dec 3, 2014, outside the held-out week and without a Portuguese holiday, as the book's rule for choosing a setting requires. Mean MASE on those windows moves from 0.77 to 0.75. The verdict is close, as [the identifier section](#identify-each-client) shows.
- The start rule applies to every client. It refuses a cutoff it cannot start, instead of skipping the client.
- Clock-change hours in each panel history are filled from that history only, so a rolling window across a clock change is fitted, not refused. Chapter 10's rolling windows depend on this.
- The headline is the mean per-client MASE over all thirty clients, on the book's one scale (set in [Chapter 2](#take-the-mase-scale-from-history) and computed by `book_scale`), beside seasonal naive on the same cutoffs. A client with no non-zero reading before the cutoff is left out of MASE and reported. The pooled MAE and RMSE follow as second lines. No client is left out at the standard cutoffs, because each has a reading before the first cutoff.
- The model runs on one thread with a fixed seed, so its output repeats on the machine that recorded the book's outputs. The book has not tested other machines.

# Make the total add up

You'll forecast the total demand of thirty clients two ways, choose between the two on decision windows, and split that total into client shares using only the history before each cutoff.

The supplier sets its one day-ahead purchase of power against the total demand of its clients, not against any one of them. [Chapter 6](#one-model-for-many-clients) forecast each client, and the total can be built from those forecasts or forecast on its own. This chapter asks whether the two agree, and which one to trust. Both are scored on the same windows as before, so the comparison is fair. The examples use Chapter 6's clients, windows and scale, so each number here is printed by an example you can run.

## Why the total needs a check {#why-the-total-needs-a-check}

Two forecasts of one quantity can disagree, because they are made by different routes. Here the quantity is the total demand of the thirty clients (`ALL30`) in each hour. One route adds up thirty client forecasts. The other fits one model to the total series. Both read the same readings, but each fails in its own way. The sum carries every client's error into the total. The total model sees only the total's own pattern, and the clients' separate patterns can disappear inside it.

The check matters because the supplier buys against the total, not against each client, and a total can miss even when every client is forecast well, if the clients' errors lean the same way in the same hour. The chapter therefore asks two questions, in order. First, do the parts add up to the whole in the data? Second, is a total forecast better than the sum of the parts on the windows that matter?

Forecasts that add up are called coherent. Hyndman and Athanasopoulos's [*Forecasting: Principles and Practice*, 3rd ed., chapter 11](https://otexts.com/fpp3/hierarchical.html) names several ways to make them so, and two matter here. *Bottom-up* forecasts the bottom level, here the clients, and sums the results. *Top-down* forecasts the top, the total, and splits it down by proportions. This chapter uses the two names only in those senses.

Coherence is a constraint, not a measure of accuracy. A total that adds up can still be worse than one that does not. So this chapter scores the total and each client separately, and it never accepts a forecast because its parts agree.

The book builds both totals by hand, in pandas, rather than with `hierarchicalforecast`. The hierarchy has one level, so each route is one sum or one split, and the runtime stays small. A method that combines the two routes optimally does exist. MinT chooses the coherent forecasts that minimise the mean squared error of the coherent forecasts, under an unbiasedness assumption, and it uses the covariance matrix of the forecast errors ([Wickramasuriya, Athanasopoulos and Hyndman, 2019](https://doi.org/10.1080/01621459.2018.1448825); an open working paper is at [robjhyndman.com/papers/mint.pdf](https://robjhyndman.com/papers/mint.pdf)). Estimating that matrix is a project of its own, and this chapter does not take it on.

**Working rule:** Forecast the total both ways whenever a decision depends on it. Score both on the same windows, against the total and against each client. Keep the method that wins on the decision windows, and never accept a total because its parts add up.

## Sum the client forecasts {#sum-the-client-forecasts}

The bottom-up total is the sum, hour by hour, of the thirty client forecasts that [Chapter 6](#one-model-for-many-clients) made. For each cutoff, one global model forecasts every client for the next 24 hours, with lags 24 and 168 and the client identifier that Chapter 6 kept. The total's forecast for an hour is the sum of the thirty values for that hour. With `c` a client and `t` an hour, the rule is `total(t) = sum over c of forecast(c, t)`.

The sum adds up by construction, so the check that matters is on the data. The total series has to equal the sum of the thirty readings in every hour, or every score is measured against the wrong number. The exception is the clock-change hours. A clock-change flag sets a client's reading to missing, so the total is missing at 01:00 on Mar 31, 2013 and Mar 30, 2014, the gap hours, and at 01:00 on Oct 27, 2013 and Oct 26, 2014, the repeat hours.

Every score below is measured against the total, so the total has to match its parts first. The first example builds both totals and scores them, and its last lines check that match in each clock-free hour.

Example `total-bottom-up-and-direct` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Forecasts the total of the thirty clients two ways, bottom-up (the sum of the client forecasts) and direct (one model on the
total), and chooses between them on the four decision windows by MAE on the total. The choice rests on the two decision-window
MAE lines; the standard cutoffs then report both beside seasonal naive, and the last two lines check the total and the scale."""
import pandas as pd

from lib.baselines import seasonal_scale
from lib.electricity import load
from lib.panel import ALL30, DECISION_CUTOFFS, check_decision_hours
from lib.scoring import actual_hours, score_windows
from lib.settings import EVAL_CUTOFFS, show
from lib.totals import (DIRECT_SCALED, METHODS, TOTAL, direct_windows, plain_mae, pooled_mae, total_checks,
                        total_forecasts, total_frame, total_scale_rows)

raw = load(ALL30)
total = total_frame(raw)  # the hourly sum of the thirty clients, after the start rule; missing in a clock-change hour
dec, cut = pd.to_datetime(DECISION_CUTOFFS), pd.to_datetime(EVAL_CUTOFFS)

dec_actual = actual_hours(total, dec)
# the direct model here uses the target setting the book chose: DIRECT_SCALED is False, so the direct total is unscaled
dec_mae = pooled_mae(dec_actual, total_forecasts(raw, dec), ["bottom-up", "direct"])
# the direct model's MAE with its target scaled (True) and unscaled (False)
direct_setting = {s: pooled_mae(dec_actual, direct_windows(raw, dec, scaled=s), ["direct"])["direct"]
                  for s in (True, False)}
# stops the run if the constant no longer names the setting with the lower MAE
if direct_setting[DIRECT_SCALED] != min(direct_setting.values()):
    raise ValueError("DIRECT_SCALED is not the target setting with the lower MAE on the decision windows")
show("decision cutoffs", len(dec))
# raises if a decision hour is a Portuguese holiday or reaches the held-out week; the count is printed for the record
show("decision holiday hours", check_decision_hours(dec_actual))
show("MAE bottom-up, decision windows", dec_mae["bottom-up"])
show("MAE direct, decision windows", dec_mae["direct"])
show("MAE direct with a scaled target, decision windows", direct_setting[True])
show("chosen direct target", "unscaled" if not DIRECT_SCALED else "scaled")
# the bottom-up total without the client identifier, which Chapter 6 also tried
show("MAE bottom-up without identifier, decision windows", plain_mae(raw, dec))
show("chosen total method", dec_mae.idxmin())

# the scale reads the readings before the first standard cutoff, and no later ones
scale = total_scale_rows(raw, cut[0])
scored, per_client = score_windows(actual_hours(total, cut), total_forecasts(raw, cut), METHODS, scale)
show("scored hours", len(scored))
# seasonal_scale returns one value per unique_id, so [TOTAL] picks the total's
show("MASE scale of the total", seasonal_scale(scale)[TOTAL])
for model in METHODS:
    show(f"MAE {model}", (scored["y"] - scored[model]).abs().mean())
for model in METHODS:
    show(f"MASE {model}", per_client.loc[TOTAL, model])
show("MAE bottom-up without identifier", plain_mae(raw, cut))
# the first cutoff is passed as first_cutoff; the scale check then runs on every cutoff in cut
for label, value in total_checks(raw, cut[0], cut).items():
    show(label, value)
```

Output:

```text
decision cutoffs: 4
decision holiday hours: 0
MAE bottom-up, decision windows: 1267.47
MAE direct, decision windows: 1678.62
MAE direct with a scaled target, decision windows: 1705.71
chosen direct target: unscaled
MAE bottom-up without identifier, decision windows: 1383.29
chosen total method: bottom-up
scored hours: 72
MASE scale of the total: 2689.26
MAE SeasonalNaive: 3639.41
MAE bottom-up: 2985.88
MAE direct: 3423.11
MASE SeasonalNaive: 1.35
MASE bottom-up: 1.11
MASE direct: 1.27
MAE bottom-up without identifier: 2837.66
clock-free hours where the total differs from the sum of the thirty clients: 0
scale changes after poisoning readings from each scored cutoff: 0
```

In every hour without a clock-change flag, the total equals the sum of the thirty clients. The check counts 0 hours that differ.

The clock-change hours need one more rule. The total is missing in those hours, but the forecast is not. Each forecast window still produces a value for every labelled hour, so the output grid has no holes that a downstream table would have to fill. The scoring refuses a forecast hour that has no reading, so a masked hour cannot be scored as a zero. No scored window in this chapter holds one.

The start rule applies to each client before the sum. A late starter's zeros before its first reading are not demand, so the rule drops them, as [Chapter 4](#meet-the-late-starters) showed for one client. The rule does not change the total, because a zero adds nothing to a sum, and a client that has not started contributes nothing either. The check above shows it. The rule does change each client's own forecast, because a model fitted on those zeros would learn them as low use.

The bottom-up total inherits the clients' errors, good and bad. Chapter 6 found that the model beats seasonal naive for 23 of the thirty clients on the standard cutoffs, and does worse for the rest. The comparison later in the chapter measures how much of that error reaches the total.

Only means add. The mean of a sum is the sum of the means, because expectation is linear. The median of a sum is not, in general, the sum of the medians, and the same holds for other quantiles. They add only in special cases, such as errors that move together perfectly. So the bottom-up total is a point forecast. A range for the total cannot be built by adding the clients' ranges. Chapter 8 returns to this for the intervals.

**Working rule:** Sum the client forecasts hour by hour, after checking that the total series equals the sum of its readings in every hour without a clock-change flag. Apply the start rule to each client before the sum, so no client's model is fitted to its zeros. Treat the sum as a point forecast, and do not take the sum of the clients' medians as the median of the total.

## Forecast the total directly {#forecast-the-total-directly}

The direct total is one model fitted to the total series itself. Its history is the total's history before the cutoff: the readings of the 1,344 hours before it, which is 8 weeks, the same length as the clients' histories. Each client's history is cut by the start rule, its clock-change hours are filled from its own neighbours, as [Chapter 3](#mark-clock-change-hours-as-missing) does, and the readings are summed by hour. The total's history therefore has no missing hour.

The model uses the features the clients use. Its inputs are the total's readings 24 and 168 hours earlier, so no input comes from the day being forecast. Its target is the total as it is, not scaled. [Chapter 6](#score-thirty-clients-on-the-same-cutoffs) scales each client's target, because one model over clients of different sizes would otherwise fit the large ones. A single total has no such mix, so that reason does not carry over. On the decision windows, the unscaled target has the lower error, 1678.62 kWh against 1705.71 kWh with the target scaled, as [the first example](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/total-bottom-up-and-direct.py) shows, so the direct model uses it. The model is fitted once per cutoff, on that cutoff's history only, and it forecasts the next 24 hours. It is a LightGBM regressor inside mlforecast 1.1.0, with the settings of [Chapter 5](#measure-what-each-feature-adds).

The two totals differ in how they are built. The bottom-up total comes from one global model that forecasts every client, and the direct total from one model that forecasts the sum. The lags and the other settings are the same. The target scaling differs, and the decision windows choose it for the direct model alone.

The total is usually smoother than any client, because the clients' peaks partly cancel in the sum. A model fitted to a smoother series has less of its own noise to learn, which can help the direct total. The comparison later in the chapter tests whether it does on these windows, and the answer is not the one the smoothness suggests.

The model sees the total only through its history. It does not see which client a reading came from, or which client moved the total. That is the trade the direct method makes. It gets one smooth series and loses the clients' separate shapes. The bottom-up method keeps those shapes and must add thirty errors together.

Seasonal naive on the total is the baseline that both must beat. It forecasts each hour as the same hour the day before, as in [Chapter 2](#three-baselines-to-beat), applied to the total's history. It is scored on the same windows and against the same scale.

**Working rule:** Forecast the total directly with the clients' features, from the total's own history before each cutoff, and choose the target's scaling on the decision windows. Use the same windows for both methods, and score each against seasonal naive on the total. Report both methods, and not only the one you built.

## Compare the two totals {#compare-the-two-totals}

On the four decision windows, the [bottom-up total](#sum-the-client-forecasts) has the lower error, so it is the method chosen. Those windows are [Chapter 5's](#choose-on-decision-windows) four ordinary days before the held-out week, used only to choose: Nov 25, Nov 26, Dec 2 and Dec 3, 2014. [Chapter 6](#identify-each-client) chose the client identifier on the same days. Each method is scored by MAE on the total, and bottom-up's MAE there is 1267.47 kWh, against 1678.62 kWh for the [direct total](#forecast-the-total-directly). The example refuses to run if a decision hour falls on a Portuguese holiday, and it counts 0 such hours.

The standard cutoffs, Dec 29 to Dec 31, 2014, are the report. They are not used to choose. On these three windows the total has 72 scored hours. Its [MASE scale](#scale-errors-with-mase) is the mean absolute change between readings a day apart, taken over every reading before Dec 29, 2014, with the clock-change hours masked. That is the rule for a MASE scale from [Chapter 2](#take-the-mase-scale-from-history), as [Chapter 6](#score-thirty-clients-on-the-same-cutoffs) applies it to each client, here applied to the total, and it comes to 2689.26 kWh.

| Method | MAE (kWh) | MASE |
| --- | --- | --- |
| Seasonal naive | 3639.41 | 1.35 |
| Bottom-up | 2985.88 | 1.11 |
| Direct | 3423.11 | 1.27 |

MAE and MASE rank the methods alike here. MAE is in kilowatt hours, so it reads as the average cost of an hour's error. [MASE](#scale-errors-with-mase) divides each error by the same scale, so on one series it is only a constant multiple of MAE. Its use is across series. [Chapter 6](#score-thirty-clients-on-the-same-cutoffs) needs it to compare clients of very different sizes, and the total sits beside them on the same footing.

On the standard cutoffs the direct total beats seasonal naive on the total, but by a smaller margin than the bottom-up total does, as the MASE column of the table shows.

A method wins on the decision windows when its error is lower there, and the standard cutoffs then either confirm that or contradict it. For the total, the standard cutoffs confirm the choice, since bottom-up is lower on both window sets. That agreement is why the chapter recommends bottom-up for the total, with some confidence. Agreement counts only when the two sets are separate, and they are. The four decision windows all fall before Dec 25, 2014, the first day of the held-out week, and the standard cutoffs are the last three days of that week. No decision reads a standard cutoff.

The identifier is where the two window sets disagree. On the decision windows, the bottom-up total without the identifier has an MAE of 1383.29 kWh, against 1267.47 kWh with it, so the identifier lowers the total's error there, as the per-client MASE in [Chapter 6](#identify-each-client) does on the same windows. On the standard cutoffs the plain total has an MAE of 2837.66 kWh, and the total with the identifier has 2985.88 kWh. The standard cutoffs therefore favour the plain total. The book reports that and does not act on it, because Chapter 6 kept the identifier on per-client MASE, and the decision windows are the ones that choose. A supplier that doubts four decision windows should add decision windows before Dec 25, 2014, and decide again on those, not on the standard cutoffs.

**Working rule:** Choose a method on the decision windows, by the measure you will report. Report the same measure on the standard cutoffs beside seasonal naive, with the scored hours and the scale. When two window sets disagree, report both, say how far apart they are, and do not pick the one that reads better.

## Split a total by share {#split-a-total-by-share}

The top-down split starts from the [direct total](#forecast-the-total-directly) and gives each client a share of it. A client's share is its readings over its own history, the 1,344 hours before the cutoff, divided by the total's readings over the same hours. Each client's top-down forecast for an hour is the direct total's forecast for that hour, times its share. A client that supplied a tenth of the total over the past 8 weeks is expected to take a tenth of the total the next day.

The eight weeks are a choice, not a tuned value. They are the history every other model in this chapter reads, so the split uses the same memory as the clients. A shorter window would follow a change in a client's share sooner, and it would be noisier. This chapter does not measure that trade-off. A supplier that wants it should test it on the decision windows.

The split's weak point is the shape of the day. It gives every client the total's daily profile, so a client that peaks at a different hour gets the wrong profile in every hour, however good the total forecast is. On these clients the split loses even with the true total in hand. A check this chapter calls the oracle check applies the shares to the actual total, which no forecast can know, and its mean per-client MAE is 171.80 kWh. That is a reference, not a method: the error the split would have with a perfect total. The bottom-up forecast, which has to forecast each client, has a mean per-client MAE of 122.81 kWh.

The shares are also uneven. On the first cutoff, Dec 29, 2014, the largest client's share of the total is 0.56.

The working paper that defines MinT reports, citing Hyndman et al. (2011), that a top-down split biases the reconciled forecasts at each level, even when the base forecasts are unbiased ([Wickramasuriya, Athanasopoulos and Hyndman, 2017](https://robjhyndman.com/papers/mint.pdf)). This chapter does not re-derive that result. It measures the split on these clients, and the oracle check gives a reference for the split's own error.

Scoring the split client by client, against bottom-up on the same windows, shows whether it wins for any single client, which a mean cannot show. The second example does that, and runs the leak check on the shares. It makes the choice on the decision windows first, then reports the standard cutoffs.

Example `total-split-by-share` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Splits the direct total forecast into client forecasts, each client taking its share of the history before each cutoff
(top-down), and compares them with bottom-up on the same windows. The point rests on the mean per-client MASE and MAE lines;
the last line is a leak check that no share reads a reading after its cutoff."""
import pandas as pd

from lib.electricity import load
from lib.panel import ALL30, DECISION_CUTOFFS, check_decision_hours
from lib.settings import EVAL_CUTOFFS, show
from lib.totals import SPLIT_METHODS, client_mae, oracle_split_mae, shares, split_checks, split_scored

raw = load(ALL30)
dec, cut = pd.to_datetime(DECISION_CUTOFFS), pd.to_datetime(EVAL_CUTOFFS)

# the third value, the hours the top-down clients don't add up to the direct total (none, by construction), is unused
dec_scored, dec_per, _ = split_scored(raw, dec)
show("decision holiday hours", check_decision_hours(dec_scored))
show("mean per-client MASE bottom-up, decision windows", dec_per["bottom-up"].mean())
show("mean per-client MASE top-down, decision windows", dec_per["top-down"].mean())
show("chosen client method", dec_per.mean().idxmin())

# the third value is discarded, as above
scored, per_client, _ = split_scored(raw, cut)
show("scored hours", len(scored))
mae = pd.DataFrame({m: client_mae(scored, m) for m in SPLIT_METHODS})
for model in SPLIT_METHODS:
    show(f"mean per-client MAE {model}", mae[model].mean())
for model in SPLIT_METHODS:
    show(f"mean per-client MASE {model}", per_client[model].mean())
show("clients where top-down has the lower MAE", int((mae["top-down"] < mae["bottom-up"]).sum()))
# the split applied to the actual total: a reference, since no forecast can know the total in advance
show("mean per-client MAE with the true total split by share", oracle_split_mae(raw, cut))
# the first standard cutoff, Dec 29, 2014; its shares read only the history before it
# shares read the 1,344 hours (8 weeks) before the cutoff
show("largest client share of the total, first cutoff", shares(raw, cut[0]).max())
# each cutoff's readings from the cutoff on replaced by zero, then by a very large value
for label, value in split_checks(raw, cut).items():
    show(label, value)
```

Output:

```text
decision holiday hours: 0
mean per-client MASE bottom-up, decision windows: 0.75
mean per-client MASE top-down, decision windows: 2.37
chosen client method: bottom-up
scored hours: 2160
mean per-client MAE bottom-up: 122.81
mean per-client MAE top-down: 218.95
mean per-client MASE bottom-up: 0.91
mean per-client MASE top-down: 2.59
clients where top-down has the lower MAE: 0
mean per-client MAE with the true total split by share: 171.80
largest client share of the total, first cutoff: 0.56
shares changed after poisoning readings from their cutoff: 0
```

On the decision windows bottom-up wins for the clients, with a mean per-client MASE of 0.75 against 2.37 for top-down, so bottom-up is chosen.

On the standard cutoffs the split is worse on error in kilowatt hours as well. Over 2160 client hours, its mean per-client MAE is 218.95 kWh, against 122.81 kWh for bottom-up, and top-down has the lower MAE for 0 of the thirty clients.

The mean per-client MASE is 2.59 for top-down, against 0.91 for bottom-up. Chapter 6's pooled MAE for the same forecasts is 122.81 kWh, the same as the bottom-up mean of 122.81 kWh. That is no accident, because every client has the same 72 scored hours, so the mean of the client errors equals the pooled error.

No share moves. A share must read only the history before its cutoff, so the leak check changes every reading from each cutoff on, once to zero and once to a very large value, and recomputes the shares. Across the six comparisons (three cutoffs by two values), 0 of them change a share. If any share had read a reading after its cutoff, the large value would have entered it. The shares add to 1, and the top-down clients add up to the direct total in every hour, by construction. An identity like that cannot fail, so the chapter counts only the leak check, the one test here that can.

**Working rule:** Split a total by share only as a baseline, or when the clients' daily profiles are alike. Compute each share from the history before its cutoff, check that no share moves when readings after its cutoff change, and score each client against bottom-up on the same scale. Keep the split for a client only when it wins on the decision windows.

## What goes wrong in production {#what-goes-wrong-ch7}

Most failures in this chapter are silent.

The first is a quietly short total. If one client drops out of the sum, the total is smaller than the demand, and its scores still look reasonable. The bottom-up total in this book raises an error when an hour has fewer than thirty client forecasts. The start rule refuses a client with too little history instead of skipping it, as [Chapter 6](#apply-the-start-rule-to-every-client) explains. In production, a run that builds fewer forecasts than clients should stop and name the missing client.

The second is the raw sum. The October repeat hour holds the energy of two ([Chapter 3](#the-october-hour-that-holds-two)), so a total built from raw readings can count that hour at roughly twice its neighbours. The result is a demand spike that is not there. The book's total masks the hour instead, and the check in [the first example](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/total-bottom-up-and-direct.py) excludes the clock-change hours for the same reason.

The third is a split that reads the future. A split whose shares use the window's own readings scores better in a backtest than it will in production. The [poisoning check](#what-goes-wrong-ch10) is the test. A share must not move when readings after its cutoff change. A production run should keep both checks, the share check in [the second example](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/total-split-by-share.py) and the scale check in [the first](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/total-bottom-up-and-direct.py), as tests that fail loudly.

The fourth is a decision and a report that disagree. The identifier lowers the total's error on the decision windows and raises it on the standard cutoffs. A monitor that watches only one set of windows will miss the other's failure. On every refit, log the total's error with and without the identifier on the same windows, beside the per-client MASE that [Chapter 6](#identify-each-client) uses to decide.

The fifth is a share that goes stale. A share is a fact about the past 8 weeks, so when a client's habits change, its share lags the change for as long as the window is long. A top-down forecast of that client is then wrong in a steady direction, not at random, and the error does not show in the total at all, because the shares still add to one. A monitor should therefore watch each client's share over time, not only the total's error. The per-client MASE in [the second example](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/total-split-by-share.py) is the measure that sees it.

The sixth is a scale that moves with the window. The total's MASE scale is read once, from the history before the first scored cutoff. A scale refreshed for each window reads that window's own readings, and every score on the total then changes for reasons that have nothing to do with the forecast. [The first example](https://github.com/rollingedition/book-time-series-forecasting/blob/c81194a3b125d35d83886136e2a40dc799ed2876/examples/total-bottom-up-and-direct.py) checks, for each scored cutoff, that the scale for that window does not move when readings from the cutoff on are replaced. No scale moves: 0 of the six comparisons (three cutoffs by two values) changes it.

**Working rule:** In production, check the total against the sum of its readings on every refit, stop when a client is missing from a total, and run the poisoning check on each share and on the scale for each window. Log the total's error with and without the identifier on the same windows. When the windows disagree, report both and do not choose the one that flatters the model.

## Decisions this chapter made {#decisions-ch7}

- **The total is the hourly sum of `ALL30`.** Each client is cut by the start rule before the sum, and an hour is missing when any client's reading in it is masked as a clock-change hour. The sum does not change with the start rule, and the check in the first example shows it.
- **Every forecast hour has a value, and scoring refuses a masked hour.** The output grid is complete through the clock-change hours, and a missing reading is never scored as zero.
- **Both totals are built by hand, in pandas.** The hierarchy has one level, so each route is one sum or one split. `hierarchicalforecast` and MinT are named, and neither is used.
- **The total is forecast bottom-up.** It has the lower MAE on the four decision windows and on the standard cutoffs. The direct total is reported beside it.
- **The direct total's target is unscaled.** On the decision windows it has the lower MAE, so the direct model does not scale its target, unlike the clients' model. The first example prints both settings.
- **The identifier stays in the bottom-up total, as [Chapter 6](#identify-each-client) decided for the clients.** Its cost to the total on the standard cutoffs is reported above, not reversed.
- **The split is not the client method.** Bottom-up has the lower per-client MASE on the decision windows, and on these clients the split loses even with the true total.
- **Shares come from the 1,344 hours before each cutoff, the same history as the clients.** The poisoning check shows that no share reads a later reading.
- **The total's MASE scale is the rule from Chapter 2, applied to the total:** the seasonal naive error over every reading before the first scored cutoff, with clock-change hours masked and the start rule applied to each client first.
- **The windows are Chapter 6's.** The decision windows and the standard cutoffs are the same days, so the two chapters' numbers can be compared line by line.
- **Chapter 8 receives point forecasts:** the bottom-up forecast for each client and for the total, with the direct total as the alternative. A range for the total cannot be built from the clients' ranges.

# Quantify forecast uncertainty

This chapter puts a range around each day-ahead forecast, so you can say whether a purchase can rely on it. You'll calibrate prediction intervals on days that come before the days you score, measure how often they hold, and choose the level to buy at from the two costs of an error.

## Why one number is not enough {#why-one-number-is-not-enough}

The forecasts so far give one number for each client and hour, and a single number is not enough to buy on. The supplier cannot buy an expected value. It buys an amount of power a day ahead for each hour, so it needs a range around each forecast. Each kWh it buys is either used or not. Buying short means buying the rest later at a premium, and buying long means paying for power that nobody uses. The two errors cost different amounts, so the right purchase is usually not the forecast itself. Take one hour. Suppose the forecast is 100 kWh. If the reading turns out 10 kWh higher, the supplier is short and pays the short cost on those 10 kWh. If it turns out 10 kWh lower, the supplier has 10 kWh it pays for and does not use. When the short cost is three times the unused cost, a purchase above the forecast costs less on average, because it makes the expensive error less likely. How far above the forecast the purchase should sit depends on that ratio, and on how wide the errors are.

This chapter states two costs as assumptions. A kWh bought short costs 3.00 cost units, and a kWh bought and not used costs 1.00. These numbers are illustrative. They are not prices, and the book does not claim that any real supplier faces them. [A later section](#choose-the-level-from-the-cost-of-error) shows how they set the purchase.

A prediction interval gives a lower and an upper bound for each hour. A prediction interval's level, such as 80 or 95, is the share of readings the bounds should contain. The share of readings the bounds contain is the coverage, and coverage belongs to many forecasts made the same way, not to one. One day's band either contains the readings or it does not. Across the 72 hours of three days, the share inside the band is a measurement. Across a year of days, it becomes a track record. A 95 percent band whose hours fall inside it 90 percent of the time has undercovered, but no single day shows it. Only the average over many hours shows it, and that average is what we measure.

Quantiles connect the bands to the costs. The lower bound of a central 80 percent interval is the 0.1 quantile of the forecast's distribution, the value that the reading falls below one time in ten. The upper bound is the 0.9 quantile. The purchase that minimises the expected cost is also a quantile, and [Choose the level from the cost of error](#choose-the-level-from-the-cost-of-error) derives which one.

In the case, the range that matters is the total's, because the total is what the supplier buys. [Chapter 6](#one-model-for-many-clients) forecast the thirty clients with one global model, and the total is the hourly sum of those forecasts. Each client has its own range, but the total's range cannot be read off the clients' ranges by adding them. [Choose the level from the cost of error](#choose-the-level-from-the-cost-of-error) compares the two.

**Working rule:** Ask for a range wherever the costs of the two errors differ, and choose its level from those costs rather than from a number that looks reassuring. Report coverage as the share of hours inside the band over many windows, never as the chance that one day falls inside.

## Build intervals from residuals {#build-intervals-from-residuals}

A range needs bounds, and the simplest bounds come from past errors. A residual is the error of an earlier forecast: the reading minus the forecast, `e = y - f`. Conformal prediction builds a band from the residuals of forecasts made the same way on earlier days. Take the residuals of a set of calibration windows, group them by horizon step, and read off the quantiles you need. A day-ahead forecast made at 00:00 has one horizon step for each hour of the day, so the group is the hour alone. For a level `L` in percent, the bounds for a forecast `f` made at a given step are:

```text
lower = f + Q((1 - L/100) / 2)
upper = f + Q((1 + L/100) / 2)
```

where `Q(p)` is the empirical p-quantile of the calibration residuals at that step. The method assumes no model of the errors. It only assumes that the calibration residuals resemble the residuals to come, which [a later section](#calibrate-on-earlier-windows) tests. A calibration window is a cutoff whose residuals size the bands. It always comes before the cutoffs being forecast, and it is never one of the windows that are scored.

The library that does this for the book's model is mlforecast, at version 1.1.0. Its class `PredictionIntervals` takes a number of windows, a horizon and a method, and it has four conformal methods: `conformal_error`, `conformal_distribution`, and their weighted versions. The two unweighted methods score the absolute residuals, `|y - f|`, so their bounds are symmetric around the forecast (mlforecast 1.1.0, `mlforecast/conformal_prediction.py`, function `compute_conformity_scores`). The default, `conformal_distribution`, works one client and one horizon step at a time. For each client and step it takes the calibration windows' absolute residuals, makes the paths of the forecast plus and minus each one, and reads the quantiles of those paths. Nothing is pooled across clients. The statistical library, statsforecast at version 2.1.1, names its interval class `ConformalIntervals`, not `PredictionIntervals`.

To see what the library does with a short calibration history, run its class with three calibration windows. It scores the three standard cutoffs, Dec 29 to Dec 31, 2014, each at 00:00. Those are the held-out week's last three days, and the days this chapter scores. mlforecast refits the global model at each cutoff and calibrates each one on the three before it.

Example `conformal-three-windows` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""mlforecast's own conformal intervals, with three calibration windows, refitted at each of the three standard cutoffs.
Compare the coverage at 80 and 95 with their labels, and with the highest coverage three windows allow."""
import pandas as pd

from lib.electricity import load
from lib.intervals import builtin_intervals, design_coverage, symmetric_ceiling
from lib.panel import ALL30
from lib.quirks import mask_clock_changes
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN. builtin_intervals keeps only rows from Oct 27, 2014 on, so no masked hour is in a window.
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]
# One row per client and hour. The bounds are columns named by model and level, such as LGBMRegressor-lo-80.
cv = builtin_intervals(df, pd.to_datetime(EVAL_CUTOFFS))

show("scored hours", len(cv))
show("calibration windows per cutoff", 3)
show("residuals per client and hour", 3)
# Ceiling of a band of plus and minus n absolute residuals, n / (n + 1), with n = 3: at most 3 of the 4 places a new one can take.
show("highest coverage three windows allow, any level", symmetric_ceiling(3), 3)
for level in (80, 95):
    # Simulated under exchangeable normal errors, with mlforecast's construction, to set against the measured coverage.
    show(f"simulated coverage at {level}, exchangeable errors", design_coverage(level, 3, "builtin"), 3)
for level in (80, 95):
    lo, hi = cv[f"LGBMRegressor-lo-{level}"], cv[f"LGBMRegressor-hi-{level}"]
    inside = (cv["y"] >= lo) & (cv["y"] <= hi)
    show(f"coverage at {level}", inside.mean(), 3)
    show(f"mean width at {level}", (hi - lo).mean())
    if level == 80:
        # The by-day lines are printed for the 80 level only. Each hour is grouped by its calendar day.
        day = cv["ds"].dt.floor("D")
        for d, share in inside.groupby(day).mean().items():
            show(f"coverage at 80 on {d:%Y-%m-%d}", share, 3)
        for d, err in (cv["y"] - cv["LGBMRegressor"]).abs().groupby(day).mean().items():
            show(f"mean absolute error on {d:%Y-%m-%d}", err)
```

Output:

```text
scored hours: 2160
calibration windows per cutoff: 3
residuals per client and hour: 3
highest coverage three windows allow, any level: 0.750
simulated coverage at 80, exchangeable errors: 0.656
simulated coverage at 95, exchangeable errors: 0.732
coverage at 80: 0.690
mean width at 80: 281.07
coverage at 80 on 2014-12-29: 0.756
coverage at 80 on 2014-12-30: 0.676
coverage at 80 on 2014-12-31: 0.637
mean absolute error on 2014-12-29: 114.62
mean absolute error on 2014-12-30: 94.42
mean absolute error on 2014-12-31: 163.37
coverage at 95: 0.750
mean width at 95: 355.39
```

The output shows the failure before any fix: across the 2160 client-hours, the bands at the 80 level cover 0.690 and at the 95 level 0.750. Both fall short of their labels, and the bands at 80 are 281.07 kWh wide on average.

The design sets a ceiling, and the 95 result sits on it. With three windows, each client and hour has three absolute residuals, and the band reaches no further than the largest of them. For exchangeable errors, the test error is the largest of four absolute errors one time in four, so the band misses it that often. The highest coverage three windows allow is 0.750, at any level. Under the same exchangeable assumption, the band's expected coverage is 0.656 at 80 and 0.732 at 95. The simulation draws normal errors, with the same construction. The measured figures are a little above those expected values, so the calibration windows did not have smaller errors than the scored days. Calm calibration windows would push coverage below the expected value, not above it. The shortfall is the construction's.

The coverage by day shows where the shortfall sits. At the 80 level it is 0.756 on Dec 29, 0.676 on Dec 30 and 0.637 on Dec 31. The forecast errors on those days were 114.62, 94.42 and 163.37 kWh on average. Dec 31 has the largest errors and the lowest coverage, and three days cannot separate day-to-day variation from the design.

A second problem comes from the calibration. Each cutoff calibrates only on days before it: the bands for Dec 31 use Dec 28, 29 and 30. But Dec 29 and Dec 30 are scored days too. So the three scored windows are not independent, because one window's calibration contains the readings of another scored window. No band uses its own day's readings, and the rule is right for operation, where those readings are known. For a test it weakens the result. Fixing it would need a separate calibration fit for each cutoff, which is more than a one-argument change, so the chapter's own bands use a fixed calibration set instead, as [Calibrate on earlier windows](#calibrate-on-earlier-windows) shows.

**Working rule:** Build each band from the residuals of windows that come before its cutoff, and check the coverage it reaches on windows it did not use before you trust it. Read the level a design can reach before you read the coverage. Three residuals per client and hour cannot reach either label, so the fix is a larger calibration set, not a different reading of the output.

## Calibrate on earlier windows {#calibrate-on-earlier-windows}

The calibration windows must come before the windows you score, and they must not share a target hour with them. The book adds one more rule: a calibration window must not fall on a Portuguese public holiday, because a holiday's load follows the calendar rather than the week before it. The rule applies to the decision windows as well. The held-out evaluation calibrates on twelve ordinary days, Dec 9 to 20, 2014, which end four days before the held-out week. Both examples in this section print the holiday hours and the shared hours in their calibration, and both counts are zero.

Twelve windows with thirty clients give 360 residuals for each hour of the day. They come from twelve days, though, and the thirty clients share each day's weather and calendar. So a day-level error is still a sample of twelve. The pooled residuals have a second problem. The clients differ enormously in size. In the scored week the largest client, MT_370, averages 17499.81 kWh an hour over the three scored days. Pooled residuals in kWh are dominated by such clients, so the bands fit the large clients and misfit the small ones. The fix is to divide each client's residuals by that client's scale before pooling, and multiply the bound back by the scale of the client being forecast. The scale is the seasonal naive error on the history before the first calibration cutoff, the same kind of scale the book uses for MASE. With the scale, the thirty clients count alike.

Two choices in this design were made on the decision windows, not on the held-out evaluation. The [decision windows](#choose-on-decision-windows) are the four cutoffs from Nov 25 to Dec 3, 2014. Their calibration is Nov 9 to 20, 2014, twelve holiday-free days that end four days before the first decision cutoff, the same shape as the held-out design. The windows are scored on their own, and the held-out evaluation is not read. The choices are judged by the interval score and the lowest client's coverage. The interval score adds a band's width to a penalty for each reading that misses the band, the penalty growing with the distance of the miss and scaled by 2 over the miss rate ([Gneiting and Raftery (2007)](https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf), eq. 43). Those windows settle both choices, as the measurements below show.

Example `conformal-decision-windows` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""The two calibration choices, scaling each client's residuals and the finite-sample levels, decided on the four decision
windows (Nov 25 to Dec 3, 2014), with calibration on Nov 9 to 20. Compare the lowest client's coverage and the interval scores printed for each variant;
the standard cutoffs are not read."""
from lib.baselines import seasonal_scale
from lib.electricity import load
from lib.intervals import DECISION_CAL, bounds, calibration_facts, coverage, interval_score, paired, total_of
from lib.panel import ALL30, DECISION_CUTOFFS
from lib.quirks import mask_clock_changes
from lib.scoring import book_scale
from lib.settings import show

raw = load(ALL30)
# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN, as in every example of this chapter.
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]
# One row per client and hour, with the forecast f, the reading y and the residual e.
cal, sc = paired(df, DECISION_CAL), paired(df, DECISION_CUTOFFS)
# Each client's MASE-style scale, from the history before the first calibration cutoff.
scale = seasonal_scale(book_scale(raw, DECISION_CAL[0]))

for label, value in calibration_facts(df, DECISION_CUTOFFS, DECISION_CAL).items():
    show(label, value)
show("residuals per hour of the day", len(cal) // 24)
variants = (("unscaled", None, False), ("scaled", scale, False),
            ("unscaled, finite", None, True), ("scaled, finite", scale, True))
for name, s, fin in variants:
    for level in (80, 95):
        # scale=None pools the raw kWh residuals; finite=True applies the (n + 1) quantile correction.
        lo, hi = bounds(sc, cal, level, scale=s, finite=fin)
        show(f"{name}, coverage at {level}", coverage(sc["y"], lo, hi), 3)
        show(f"{name}, interval score at {level}", interval_score(sc["y"], lo, hi, level))
    lo, hi = bounds(sc, cal, 80, scale=s, finite=fin)
    # Coverage per client, then the lowest of the thirty clients.
    show(f"{name}, lowest client coverage at 80", ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean().min(), 3)
# One series, so no scale: the scale exists only to pool clients of different size.
tcal, tsc = total_of(cal), total_of(sc)
for name, fin in (("total", False), ("total, finite", True)):
    lo, hi = bounds(tsc, tcal, 80, finite=fin)
    show(f"{name}, coverage at 80", coverage(tsc["y"], lo, hi), 3)
    show(f"{name}, interval score at 80", interval_score(tsc["y"], lo, hi, 80))
```

Output:

```text
calibration windows: 12
holiday hours in calibration: 0
calibration hours shared with scored: 0
residuals per hour of the day: 360
unscaled, coverage at 80: 0.809
unscaled, interval score at 80: 603.88
unscaled, coverage at 95: 0.957
unscaled, interval score at 95: 1950.32
unscaled, lowest client coverage at 80: 0.021
scaled, coverage at 80: 0.834
scaled, interval score at 80: 336.72
scaled, coverage at 95: 0.954
scaled, interval score at 95: 581.28
scaled, lowest client coverage at 80: 0.365
unscaled, finite, coverage at 80: 0.811
unscaled, finite, interval score at 80: 604.19
unscaled, finite, coverage at 95: 0.959
unscaled, finite, interval score at 95: 1961.48
unscaled, finite, lowest client coverage at 80: 0.021
scaled, finite, coverage at 80: 0.835
scaled, finite, interval score at 80: 337.31
scaled, finite, coverage at 95: 0.956
scaled, finite, interval score at 95: 592.35
scaled, finite, lowest client coverage at 80: 0.375
total, coverage at 80: 0.948
total, interval score at 80: 9268.36
total, finite, coverage at 80: 0.990
total, finite, interval score at 80: 10724.99
```

Scaling wins on the two measures the variants were judged by, and loses only on the pooled coverage at 80. The worst-covered client's coverage at 80 rises from 0.021 to 0.365, and the interval score at 80 falls from 603.88 to 336.72. It falls at 95 as well, as the output shows.

We judged the variants by these two measures because the pooled figure hides clients. We chose them after seeing the decision numbers, so the pooled figure is reported beside them. At 80 the pooled coverage is closer to its label without the scale, at 0.809 against 0.834. The scale stays.

The finite-sample levels stay out of the bands. Each new residual is one more among n + 1, so the finite-sample levels move outward a little: each quantile level moves to `floor((n + 1) * a) / n` for the lower bound and `ceil((n + 1) * (1 - a)) / n` for the upper, where `n` is the number of residuals and `a` the tail share. Their effect on the scaled coverage at 80 is small, from 0.834 to 0.835, and the interval score at 80 gets worse, from 336.72 to 337.31. It gets worse at 95 too, as the output shows. The correction widens the bands and does not help, so the bands do not use it.

The held-out evaluation applies the same design to the standard cutoffs, which are the days the rest of the chapter scores.

Example `conformal-earlier-windows` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Conformal bands for every client, calibrated on twelve holiday-free days before the scored week, with each client's
residuals scaled first. Read the coverage and interval score at 80 and 95 first; the unscaled, three-day and
finite-sample lines are printed for comparison."""
import pandas as pd

from lib.baselines import seasonal_scale
from lib.electricity import load
from lib.intervals import CAL_CUTOFFS, bounds, calibration_facts, coverage, interval_score, paired
from lib.panel import ALL30
from lib.quirks import mask_clock_changes
from lib.scoring import book_scale
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]
scored_cutoffs = pd.to_datetime(EVAL_CUTOFFS)
cal, sc = paired(df, CAL_CUTOFFS), paired(df, scored_cutoffs)
# The scale is read from the history before the first calibration cutoff, as the book's MASE scale is.
scale = seasonal_scale(book_scale(raw, CAL_CUTOFFS[0]))
# The largest client is the one with the highest mean reading over the scored windows.
big = sc.groupby("unique_id")["y"].mean().idxmax()

for label, value in calibration_facts(df, scored_cutoffs).items():
    show(label, value)
show("calibration residuals per hour of the day", len(cal) // 24)
show("largest client", big)
show("mean hourly load of the largest client", sc.groupby("unique_id")["y"].mean().max())
for level in (80, 95):
    lo, hi = bounds(sc, cal, level, scale=scale)
    show(f"coverage at {level}", coverage(sc["y"], lo, hi), 3)
    show(f"mean width at {level}", (hi - lo).mean())
    show(f"interval score at {level}", interval_score(sc["y"], lo, hi, level))
lo, hi = bounds(sc, cal, 80, scale=scale)
per_client = ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean()
show("lowest client coverage at 80", per_client.min(), 3)
# No scale here: the pooled residuals stay in kWh, which is what lets the large clients set the bands.
lo, hi = bounds(sc, cal, 80)
unscaled = ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean()
show("largest client coverage at 80, unscaled residuals", unscaled[big], 3)
# The three calibration days nearest the scored week: Dec 18, 19 and 20, 2014.
recent = cal[cal["cutoff"] >= "2014-12-18"]
lo, hi = bounds(sc, recent, 80, scale=scale)
show("coverage at 80 from three calibration days", coverage(sc["y"], lo, hi), 3)
lo, hi = bounds(sc, cal, 80, scale=scale, finite=True)
show("coverage at 80 with finite-sample levels", coverage(sc["y"], lo, hi), 3)
```

Output:

```text
calibration windows: 12
holiday hours in calibration: 0
calibration hours shared with scored: 0
calibration residuals per hour of the day: 360
largest client: MT_370
mean hourly load of the largest client: 17499.81
coverage at 80: 0.739
mean width at 80: 267.62
interval score at 80: 673.89
coverage at 95: 0.902
mean width at 95: 532.00
interval score at 95: 1326.80
lowest client coverage at 80: 0.472
largest client coverage at 80, unscaled residuals: 0.000
coverage at 80 from three calibration days: 0.711
coverage at 80 with finite-sample levels: 0.741
```

The scaled bands still fall short of both labels on the standard cutoffs: they cover 0.739 of client-hours at 80 and 0.902 at 95. Their mean widths are 267.62 kWh and 532.00 kWh.

The scores are 673.89 at 80 and 1326.80 at 95. They are in kWh, so the large clients dominate them. Lower is better, and the score rewards a band that is both narrow and right.

The pooled bands still have to be read per client. The lowest client coverage at 80 is 0.472, for the client the scaled bands fit least well. With unscaled residuals, the largest client is covered in 0.000 of its hours at 80. The pooled kWh bands were set by the large clients, and they miss the largest client entirely.

Two more checks on the standard cutoffs test the design. Calibrating on only the three days nearest the scored week, Dec 18 to 20, gives 0.711 at 80, against 0.739 with all twelve. Three days are a smaller sample, and the coverage is a little lower. The finite-sample levels give 0.741 at 80, a small change that agrees with the decision windows: the correction does little for the client bands.

**Working rule:** Calibrate on windows that come before the scored ones, with no holiday in them, and scale each client before pooling its residuals. Make each design choice on windows that are not the ones you report. Report how many windows and residuals went into the bands and the coverage they gave on the scored days, even when that coverage falls short of the level.

## Measure coverage on held-out days {#measure-coverage-on-held-out-days}

Coverage is measured on days that the calibration never used. The standard cutoffs give three days, and each day gives 24 hours for each client, so the 2160 client-hours are not that many independent observations. The thirty clients share the calendar and the weather, so the hours of one day agree with one another. For day-level errors the sample is three days, which cannot measure a coverage level. Three days can show that a band fell short on those days, but they cannot say by how much a band falls short over a longer period.

The supplier buys the total, so the total's coverage matters most. Its bands are built from its own residuals over the same twelve days, with the quantiles read directly, and are offered at 80 only, because twelve residuals cannot support 95. With signed bounds, the band from the smallest residual to the largest covers at most 0.846 of the time, if the residuals are exchangeable, and a 95 band, or a 90 band, asks for more.

The total's bands cover 0.667 of its 72 scored hours at 80. For exchangeable errors, the same construction covers 0.682, so most of the gap to 80 comes from the twelve-residual sample. [The section on cost](#choose-the-level-from-the-cost-of-error) puts these bands to use.

The total's errors vary by day. The mean absolute error of the total was 2415.51 kWh on the calibration windows and 2985.88 kWh on the scored days. Dec 31 accounts for most of the gap between the scored days and the calibration windows.

| Scored day | Error of the total, kWh | Coverage at 80 |
| --- | --- | --- |
| Dec 29, 2014 | 2448.53 | 0.625 |
| Dec 30, 2014 | 2149.78 | 0.708 |
| Dec 31, 2014 | 4359.32 | 0.667 |

The coverage does not follow the errors. Dec 29 has the lowest coverage with a middle-sized error, and Dec 31 has the largest error with a middle-sized coverage. Each day has 24 hours, and three days are too few to tell the two apart.

The finite-sample levels would change the total's figure. On the scored days they give 0.736 at 80. The choice was made on the decision windows, where the correction widened the total's bands and worsened their interval score, so the bands do not use it. The decision windows also show that the direction of the gap is not fixed. There the total's 80 bands covered 0.948 of hours, more than their label, and the finite levels pushed that to 0.990.

[Angelopoulos and Bates (2022)](https://arxiv.org/pdf/2107.07511) show that split conformal prediction covers the test point with probability at least `1 - alpha` when the calibration and test points are independent and identically distributed, or more generally exchangeable. They also note that adjacent points in a time series are not exchangeable, using a temperature series as the example. The same holds for hourly load: each hour carries the one before it, and a day's hours share the weather and the calendar. [Barber and co-authors (2023)](https://arxiv.org/abs/2202.13415) call the loss in coverage when exchangeability fails a coverage gap. Their bound does not assume a particular joint distribution. It is small when recent calibration points are weighted in a way that matches the test point's distribution, as in slow drift. That result is theoretical, and this chapter does not test it. A 2026 preprint, [a survey of conformal methods for dependent series](https://arxiv.org/abs/2601.18509), states that temporal dependence violates the exchangeability assumption. It is not peer reviewed, and we cite it only for that point.

So the book promises nothing about coverage that it does not measure. The coverage in this chapter is what three days showed, under the calibration described in [Calibrate on earlier windows](#calibrate-on-earlier-windows).

**Working rule:** Report coverage on windows that the calibration never used, with the number of windows and the coverage of each. Keep the same measure in a running log once the forecast runs every day, as [Chapter 10](#keep-the-forecast-running) does. Read three scored days as a warning, not as a measurement of the level you were promised.

## Choose the level from the cost of error {#choose-the-level-from-the-cost-of-error}

The supplier buys one amount for each hour, and the costs of the two errors set that amount. Let `cu` be the cost of one kWh bought short and `co` the cost of one kWh bought and not used. Let `F` be the distribution of demand. Differentiating the expected cost with respect to the purchase `q` gives `co F(q) - cu (1 - F(q))`, which is zero where `F(q) = cu / (cu + co)`, the smallest expected cost. The level `cu / (cu + co)` is the critical fractile of the newsvendor model. The [HKUST IEEM 101 notes on the newsvendor model](https://ieda.ust.hk/dfaculty/ajay/courses/ieem101/lecs/inventory/newsvendor-proof.pdf) give the same proof, writing the cost of excess as `c0`. This chapter writes it `co`. The purchase is therefore the quantile at level `tau = cu / (cu + co)`.

With the illustrative costs from [Why one number is not enough](#why-one-number-is-not-enough), `tau` is 0.75. The rule says to buy the quantile of the demand at that level, for each hour. The quantile at level `tau` is the purchase that minimises the quantile loss at that level, as [Gneiting and Raftery (2007)](https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf) show. The quantile loss is `tau * (y - q)` when the reading `y` is at or above the purchase `q`, and `(1 - tau) * (q - y)` when it is below. The quantile loss is not a cost in kWh or in money. It is the rule that the quantile minimises, and it has the same shape as the supplier's cost. Multiplying it by `cu + co` gives the cost exactly, which the example prints beside the mean cost.

Does buying at the critical fractile cost less than buying at the median or higher, on days the bands never saw? Below, the purchase is set for each hour from these costs, and the scored days answer the question. The level 0.95 is not offered, as [Measure coverage on held-out days](#measure-coverage-on-held-out-days) explains.

Example `buy-ahead-level-by-cost` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""The buy level for the total, set from the two costs: the critical fractile, cu / (cu + co), is the quantile that minimises
the expected cost. Compare the mean cost and the hours short at each level; the costs are illustrative, not prices."""
import pandas as pd

from lib.electricity import load
from lib.intervals import CAL_CUTOFFS, COST_EXCESS, COST_SHORT, buy_report, critical_fractile, paired
from lib.panel import ALL30
from lib.quirks import mask_clock_changes
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN, as in every example of this chapter.
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]
# Calibration on Dec 9 to 20, 2014; scored on the three standard cutoffs.
pairs_cal, pairs_sc = paired(df, CAL_CUTOFFS), paired(df, pd.to_datetime(EVAL_CUTOFFS))

show("cost of a kWh short", COST_SHORT)
show("cost of a kWh unused", COST_EXCESS)
# The quantile level cu / (cu + co), from the two costs printed above.
show("critical fractile", critical_fractile())
# buy_report returns the printed rows as (label, value, decimal places).
for label, value, digits in buy_report(pairs_cal, pairs_sc):
    show(label, value, digits)
```

Output:

```text
cost of a kWh short: 3.00
cost of a kWh unused: 1.00
critical fractile: 0.75
scored hours: 72
residuals per hour for the total: 12
highest coverage the total's residuals allow: 0.846
simulated coverage at 80, exchangeable errors: 0.682
mean absolute error of the total, calibration days: 2415.51
mean absolute error of the total, scored days: 2985.88
total coverage at 80: 0.667
total coverage at 80 with finite-sample levels, not used: 0.736
total coverage at 80 on 2014-12-29: 0.625
total coverage at 80 on 2014-12-30: 0.708
total coverage at 80 on 2014-12-31: 0.667
total mean absolute error on 2014-12-29: 2448.53
total mean absolute error on 2014-12-30: 2149.78
total mean absolute error on 2014-12-31: 4359.32
mean cost at 0.50: 5087.95
hours short at 0.50: 0.528
mean cost at 0.75: 3843.61
hours short at 0.75: 0.403
mean cost at 0.90: 3667.43
hours short at 0.90: 0.222
quantile loss at critical fractile, times both costs: 3843.61
mean hourly total bound from the total: 31983.95
mean hourly total bound from the sum of client bounds: 32388.77
hours short at critical fractile, sum of client bounds: 0.306
mean cost at critical fractile, sum of client bounds: 3729.17
```

The identity holds: multiplied by the sum of the two costs, the quantile loss at the critical fractile is 3843.61, the same as the mean cost at that level, 3843.61. The rule is correct as arithmetic.

| Level | Mean cost | Hours short |
| --- | --- | --- |
| 0.50 | 5087.95 | 0.528 |
| 0.75, the critical fractile | 3843.61 | 0.403 |
| 0.90 | 3667.43 | 0.222 |

The result on the scored days is less comfortable. The rule chose the critical fractile, and on these three days 0.90 cost less, at 3667.43 against 3843.61. At the critical fractile the purchase was short in 0.403 of the hours, where a calibrated level would fall short in one hour in four. That is the coverage shortfall from [Measure coverage on held-out days](#measure-coverage-on-held-out-days) seen as money. We do not move the level to 0.90. The three scored days are the test, and a level chosen because it wins on them has been chosen on the test. The honest reading is that the bands were too narrow here, so the purchase was too low, and that the level is right only when the bands are.

Should the purchase follow the clients' bounds instead? Summing each client's bound at the critical fractile gives a bound of 32388.77 kWh an hour, against 31983.95 from the total's own residuals. Summed quantiles equal the total's quantile when the errors move together perfectly. For an upper quantile, with errors that partly cancel, the sum usually sits above it, as it does here.

On the scored days the summed bound did better, with hours short at 0.306 against 0.403, and a mean cost of 3729.17 against 3843.61. That advantage comes from its higher level, not from a better calibration, and the scored days cannot show that it holds in general. We keep the total's own bound, because the purchase is made against the total's residuals, and we do not switch to the bound that did better here.

**Working rule:** Set the buy level from the two costs, as `tau = cu / (cu + co)`, fix it before you score the days, and calibrate the total's bands on the total's own residuals. Show the realised cost at other levels as a check, but never move the level to whichever one would have won on the days you score.

## What goes wrong in production {#what-goes-wrong-ch8}

- **The calibration season is not the scored season.** Holidays are kept out of the calibration, which keeps the test honest, but the total's errors vary from one window to another, and they can be larger in the scored days than in the calibration windows. A band calibrated in a calm period misses a harder one. Notice it by comparing coverage with the nominal level every day and watching the running average. A coverage that falls below its label for a week is the sign, and the errors of that week are the first place to look.
- **Pooled bands hide clients.** Bands pooled in kWh are set by the largest clients. With unscaled residuals, the largest client's bands cover 0.000 of its hours at 80. With scaled residuals the lowest client coverage at 80 is 0.472, so even the scaled pool hides a client that misses. Read coverage per client, not only the pooled figure.
- **Three days say little.** A coverage figure from three scored days can look like a finding, but it is three observations of the day-level error. Use the coverage over many windows, and treat a short run as a reason to look, not as a result.
- **Dependence breaks the simple guarantee.** Hourly load is autocorrelated, and the calibration residuals of neighbouring hours are not independent. The bands carry no guarantee from the textbook theorem. Their coverage can drift when the weather or the calendar changes, and only the measured coverage tells you when that happens. A run of days whose errors share a sign is the pattern to watch for, and the running log is where it shows up.
- **The total has few residuals per hour.** A calibration window gives one residual for the total in each hour, so twelve windows give twelve. With signed residuals, the highest coverage twelve residuals allow is 0.846, so the total's bands stop at 80, and a level check should come before any band is offered. More calibration windows would give the total a larger sample, but each must be an ordinary day before the scored week.
- **Rolling calibration can use the days you score.** The built-in rule in [Build intervals from residuals](#build-intervals-from-residuals) calibrates each cutoff on the three cutoffs before it. In operation that is fine, because those readings are known. In a test, it makes the scored windows depend on each other. Keep the calibration set fixed and earlier than every scored window.

## Decisions this chapter made {#decisions-ch8}

- **Calibration windows.** Twelve ordinary days, Dec 9 to 20, 2014, before the held-out week. Holidays are excluded, and no calibration hour is scored. The decision windows use twelve days too, Nov 9 to 20, 2014. The length is a choice, not a result: the data do not show that twelve days is the best length.
- **Scale before pooling.** Each client's residuals are divided by its seasonal naive error on the history before the first calibration cutoff, the same scale the book uses for MASE. The decision windows settled it, and the total needs no scale.
- **No finite-sample correction in the bands.** The correction was tested on the decision windows, for the clients and for the total, and it did not improve the interval score. It is used only to read off which levels a set of residuals can support.
- **Signed residuals.** The bands and the purchase quantile use signed residuals, so the lower and upper bounds need not be symmetric. mlforecast's conformal methods use absolute residuals and give symmetric bands. The built-in run in [Build intervals from residuals](#build-intervals-from-residuals) is there for comparison.
- **The purchase is for the total.** The buy level and its quantile are set on the total's residuals. The clients' bands are for reading the forecasts, not for buying.
- **The costs are assumptions.** The costs of 3.00 and 1.00 are stated in [Why one number is not enough](#why-one-number-is-not-enough), as judgement, and not measured.
- **The statistical library's interval class.** statsforecast's class is `ConformalIntervals`. Its name differs from mlforecast's `PredictionIntervals`, and this chapter uses each name for its own library.
- **A third example.** The purchase decision needed its own measurement, so `buy-ahead-level-by-cost` was added, along with `conformal-decision-windows`, which makes the design choices on windows the held-out evaluation does not use.
- **Holiday coverage is not claimed.** Dec 25, 2014 is in the held-out week, but it is outside both the calibration and the scored windows. The book reports nothing about how the bands cover the holiday itself.
- **Bands before levels.** A purchase level is only as good as the band under it, and a band only as good as the windows that set it.

# Compare Prophet and a foundation model

Once you finish this chapter, you can fit Prophet on the book's three standard cutoffs, load a recorded foundation model (Chronos-2) and a recorded neural model (NHITS), and score all of them on one MASE scale against the same readings.

Readers meet Prophet and neural forecasting models outside this book, each with its own claims. This chapter asks what each of them does on the supplier's clients, on the standard cutoffs, Dec 29 to 31, 2014. The supplier needs something narrower from any model: a day-ahead forecast for each client, and a range it can buy against. [Chapter 8](#calibrate-on-earlier-windows) calibrated such ranges on earlier windows. The ranges here are not calibrated that way, so this chapter checks their coverage and does not fix it. Every model is scored against seasonal naive from [Chapter 2](#three-baselines-to-beat) and the global model from [Chapter 6](#one-model-for-many-clients), on the same hours and one MASE scale. Two of the models are recorded rather than run.

## Where Prophet fits {#where-prophet-fits}

Prophet is the first model a reader tries beyond the baselines, so we test it first, and its assumptions decide where it can help.

Prophet writes a series as a trend, plus Fourier seasonal terms, plus holiday effects, plus an error term. The authors describe it as a modular regression model whose parameters an analyst can read and adjust ([Taylor and Letham (2018)](https://doi.org/10.7287/peerj.preprints.3190v2)). By default Prophet's fit is a maximum a posteriori estimate, an optimisation rather than a set of sampled draws. The code of version 1.5.0 sets that default in its optimisation call (`models.py`, lines 165 and 172), and its `mcmc_samples` argument defaults to 0 (`forecaster.py`, line 144). Both are in the [Prophet source distribution for the pinned version](https://files.pythonhosted.org/packages/e3/5b/8f9e2387cefc8fab4071f9ece67d89793f3c13566d6349dd43e9d8a5a3e3/prophet-1.5.0.tar.gz). Each client gets its own fit, so Prophet does not learn from the other clients.

Prophet does several things well. It needs no interpolation of missing observations: the code drops rows whose `y` is missing before the fit (`forecaster.py`, line 1335). The clock-change hours are missing in the history by [the rule in Chapter 3](#mark-clock-change-hours-as-missing), so Prophet leaves them out without any extra code. Prophet takes holidays as a table of dates and effects, and the table this chapter tests holds the Portuguese national holidays for 2013 and 2014. The authors report that, on their own series with default settings, Prophet had lower error than auto.arima, ETS, seasonal naive and TBATS ([Taylor and Letham (2018)](https://doi.org/10.7287/peerj.preprints.3190v2)). That result rests on their data, so we cite it as the authors' result and do not use it as evidence about this case.

Several defaults get in the way here. Yearly seasonality turns on only with at least 730 days of history (`forecaster.py`, line 1187), so it stays off with eight weeks. The default interval is 80%, not 95% (`forecaster.py`, line 145). It covers trend uncertainty and observation noise, not the seasonal pattern, unless the model is fitted with full sampling ([Prophet documentation](https://meta.github.io/prophet/docs/uncertainty_intervals.html)). Its coverage therefore makes a narrower claim than the word "interval" suggests. Prophet also assumes that the future trend changes as often, and by as much, as it did in the history ([Prophet documentation](https://meta.github.io/prophet/docs/uncertainty_intervals.html)). With eight weeks of history there is little to learn trend changes from, so in our reading that assumption carries most of the forecast.

**Working rule:** Use Prophet per client when a reader needs its trend and seasonal settings as parameters they can read, and when its scored forecasts beat seasonal naive on the decision windows you chose before scoring. Set its interval width in code, and read its default 80% interval as an 80% interval, never as a 95% range. Check its coverage against the level it claims.

## Run Prophet on the same cutoffs {#run-prophet-on-the-same-cutoffs}

The last section named Prophet's assumptions. This one fixes its settings and runs it on the same hours as every other model.

The comparison uses the three standard cutoffs, at 00:00 on Dec 29, Dec 30 and Dec 31, 2014. These are the windows [Chapters 4 to 8](#meet-the-late-starters) score. Each window's history is the 1,344 hours, eight weeks, before its cutoff, with the start rule applied, so a late starter's history begins at its first reading. Prophet is fitted once for each client and cutoff, which makes 90 fits.

We chose the holiday table on decision windows, not on the standard cutoffs. The ordinary decision windows hold no holiday in their scored hours, so they cannot test a holiday table at all, while the holiday decision windows can. So the choice uses both kinds. A run with the table and one without it shows what the table does on those windows for `SUBSET8`: the [four ordinary days](#choose-on-decision-windows) and seven holidays, Apr 18, Apr 20, Apr 25, May 1, Jun 10, Aug 15 and Dec 8, 2014. All of them come before the held-out week, and none has Christmas Day in its history.

Example `prophet-holiday-decision` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · prophet 1.5.0 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Prophet with and without the Portuguese holiday table, on the eleven decision windows for SUBSET8 (B-9, B-10),
before any standard cutoff is scored. The point that the table stays off rests on the two mean MASE lines and on
holiday table kept, which says whether the table lowers the mean per-client MASE."""
from lib.compare import prophet_windows, seasonal_naive, score
from lib.electricity import load
from lib.features import is_holiday
from lib.panel import DECISION_CUTOFFS, HOLIDAY_DECISION_CUTOFFS
from lib.quirks import mask_clock_changes
from lib.settings import SUBSET8, show

# the four ordinary and seven holiday windows, in date order
DECISION = sorted([*DECISION_CUTOFFS, *HOLIDAY_DECISION_CUTOFFS])
raw = load(SUBSET8)
# clock-change hours set to missing, so Prophet fits without them
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]
keys = ["unique_id", "ds"]
# one fit per client and window, with the Portuguese holidays as Prophet's holiday table; two processes, and the same
# fits at any count
with_holidays = prophet_windows(df, DECISION, use_holidays=True, workers=2)[keys + ["Prophet"]].rename(
    columns={"Prophet": "WithHolidays"})
without = prophet_windows(df, DECISION, use_holidays=False, workers=2)[keys + ["Prophet"]].rename(
    columns={"Prophet": "NoHolidays"})
forecasts = with_holidays.merge(without, on=keys, validate="one_to_one").merge(
    seasonal_naive(df, DECISION), on=keys, validate="one_to_one")
models = ["WithHolidays", "NoHolidays", "SeasonalNaive"]
# per_client: one MASE per client and model; scored: forecasts beside readings
per_client, scored = score(raw, forecasts, models, SUBSET8, DECISION)

show("decision windows", len(DECISION))
show("holiday decision windows", len(HOLIDAY_DECISION_CUTOFFS))
show("clients", len(SUBSET8))
show("scored hours", len(scored))
# 1 for each hour on a Portuguese national holiday
show("holiday hours in the decision windows", int(is_holiday(scored["ds"]).sum()))
for model in models:
    # the mean of the per-client MASEs, not a pooled error
    show(f"mean MASE {model}", per_client[model].mean(), digits=3)
show("hours where the forecasts differ", int((scored["WithHolidays"] != scored["NoHolidays"]).sum()))
show("clients where holidays help", int((per_client["WithHolidays"] < per_client["NoHolidays"]).sum()))
# the table is kept only on a strictly lower mean; a tie keeps the simpler model, with the table off
show("holiday table kept", "yes" if per_client["WithHolidays"].mean() < per_client["NoHolidays"].mean() else "no")
```

Output:

```text
decision windows: 11
holiday decision windows: 7
clients: 8
scored hours: 2112
holiday hours in the decision windows: 1344
mean MASE WithHolidays: 1.070
mean MASE NoHolidays: 1.061
mean MASE SeasonalNaive: 0.934
hours where the forecasts differ: 768
clients where holidays help: 2
holiday table kept: no
```

On the decision windows the table makes Prophet worse. Over the eleven of them, its mean per-client MASE is 1.070 with the table and 1.061 without it, against 0.934 for seasonal naive.

A setting is kept only if it lowers the decision-window MASE, and a tie keeps the simpler model. The example's verdict on keeping the table is no, so the table stays off.

The holiday windows hold 1344 client-hours of holiday, and they are the only place the table can change a forecast. 768 of the 2112 scored hours differ between the two variants.

The standard cutoffs tell a different story, which we report without using it to choose. Their histories include Christmas Day, and the holiday windows' histories do not. That is a likely reason for the gap, but we did not test which holiday causes it. With the table on, Prophet's mean per-client MASE on the standard cutoffs would be 1.27, against 1.35 with it off. A team that wants the table should test it on its own decision windows before it scores anything.

Daily and weekly seasonality are on, and yearly is off. These are the choices Prophet makes for itself on eight weeks of hourly data: yearly needs at least 730 days, weekly at least two weeks, and daily at least two days of sub-daily data (`forecaster.py`, lines 1187, 1209 and 1223). We set them explicitly and did not tune them. The interval is 95%, set in code and printed as `Prophet interval level`, so coverage sits beside the recordings' 95% ranges. Prophet draws its interval samples from numpy's global generator, so the code sets the seed (`SEED`) before each predict. The fit itself is deterministic.

Every model is scored on [the book's one MASE scale](#take-the-mase-scale-from-history): the seasonal naive error in sample, over every reading before Dec 29, 2014, with the clock-change hours masked. A [MASE](#scale-errors-with-mase) below 1 means the model's error on the scored hours is smaller than seasonal naive's in-sample error. Each client has its own scale, so a large client does not outweigh a small one. The next run fits Prophet for all thirty clients and scores it beside seasonal naive and the global model on this scale, which is the comparison every later result in the chapter uses.

Example `prophet-thirty-clients-backtest` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · prophet 1.5.0 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Prophet for each of the thirty clients on the three standard cutoffs, with a 95% interval, scored on the book's MASE
scale beside seasonal naive and the Chapter 6 global model. The point that Prophet's mean MASE is higher rests on
the mean MASE lines, and the coverage line says whether its 95% interval holds."""
import pandas as pd

from lib.compare import (PROPHET_HOLIDAYS, PROPHET_LEVEL, global_forecasts, prophet_windows, report_range,
                         report_subset, score, seasonal_naive)
from lib.electricity import load
from lib.panel import ALL30
from lib.quirks import mask_clock_changes
from lib.scoring import report_scores
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
# clock-change hours set to missing, as chapter 3 sets them
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]
cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)
# Prophet at the book's settings from lib.compare: holiday table off, 95% interval, one fit per client and cutoff
forecasts = prophet_windows(df, cutoffs).merge(
    seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(
    global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")
models = ["Prophet", "SeasonalNaive", "Global"]
# per_client: one MASE per client and model; scored: forecasts beside readings
per_client, scored = score(raw, forecasts, models, ALL30)

show("Prophet interval level", PROPHET_LEVEL)
show("Prophet holiday table", "on" if PROPHET_HOLIDAYS else "off")
show("clients scored", scored["unique_id"].nunique())
show("scored hours", len(scored))
show("clients where Prophet beats seasonal naive", int((per_client["Prophet"] < per_client["SeasonalNaive"]).sum()))
show("clients where Prophet beats the global model", int((per_client["Prophet"] < per_client["Global"]).sum()))
report_scores(scored, per_client, models)
report_subset(per_client, models)
# the 95% range from Prophet's lower and upper interval columns
report_range(scored, "Prophet", "ProphetLo", "ProphetHi", PROPHET_LEVEL)
# Sensitivity only, not used to choose: the holiday table on the same cutoffs, which the decision windows
# did not favour.
# two processes; each fit is the same at any worker count
on = prophet_windows(df, cutoffs, use_holidays=True, workers=2)[["unique_id", "ds", "Prophet"]].rename(
    columns={"Prophet": "ProphetHolidays"})
per_holiday, _ = score(raw, forecasts.merge(on, on=["unique_id", "ds"], validate="one_to_one"),
                       ["ProphetHolidays"], ALL30)
show("mean MASE Prophet with holidays, not chosen", per_holiday["ProphetHolidays"].mean())
```

Output:

```text
Prophet interval level: 95
Prophet holiday table: off
clients scored: 30
scored hours: 2160
clients where Prophet beats seasonal naive: 9
clients where Prophet beats the global model: 6
MAE Prophet: 182.44
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE Prophet: 764.88
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE Prophet: 1.35
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE Prophet SUBSET8: 1.39
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
coverage 95 Prophet: 0.918
interval score 95 Prophet: 932.66
mean MASE Prophet with holidays, not chosen: 1.27
```

On these cutoffs Prophet has a higher mean per-client MASE than seasonal naive: 1.35 over the thirty clients, against 1.18 for seasonal naive and 0.91 for the global model.

The loss is on most clients too, not only on average: Prophet beats seasonal naive on only 9 of the thirty clients, and the global model on 6.

Pooled over every scored hour, Prophet's RMSE is 764.88 kWh, against 780.48 for seasonal naive, so it makes fewer very large misses. It loses to seasonal naive on MAE and MASE, and to the global model on every measure. Its 95% interval covers 0.918 of the readings, and its interval score is 932.66. Coverage below 0.95 means the interval is too narrow for its stated level on these hours.

**Working rule:** Set Prophet's seasonality, holidays and interval width in code before you score it, and choose any of them on decision windows, never on the standard cutoffs. Keep the holiday table only if it lowers the MASE on the holiday decision windows. Compare Prophet with seasonal naive and the global model on the same windows and the same scale, and list the clients it loses. Set the seed before each predict, and check that a second run prints the same lines.

## Load a recorded foundation model's forecasts {#load-a-recorded-foundation-model}

Prophet fits each client from its own eight weeks. A foundation model brings what it learned from other series, and this section shows how the book holds its forecasts.

A foundation model is trained once on many series and then applied to a context, with no fitting on that series. Chronos-2 is one such model. Chronos-2 is an encoder-only transformer that takes a context of past values and outputs quantiles for the next steps ([Ansari et al. (2025)](https://arxiv.org/abs/2510.15821); the earlier Chronos model is described in [Ansari et al. (2024)](https://arxiv.org/abs/2403.07815)). Its weights are published as `amazon/chronos-2` on Hugging Face under the Apache-2.0 licence, not gated ([model card](https://huggingface.co/amazon/chronos-2)).

Its pretraining covers this dataset, which limits how far its score can be read. The Chronos-2 report lists the electricity dataset in its pretraining corpus. Appendix A, Table 6 gives "Electricity", with 370 series in the energy domain ([Ansari et al. (2025)](https://arxiv.org/abs/2510.15821)). Those series match the UCI ElectricityLoadDiagrams20112014 dataset the book reads, and the book's thirty clients are drawn from them. The report notes partial overlap with some GIFT-Eval training portions. For strictly zero-shot results it points to a variant trained only on synthetic data, in its Section 5.4. We have not run that variant. The report does not show which readings of these clients the model saw, so the scores below are upper bounds, not out-of-sample results.

We do not run Chronos-2 in an example. Its forecasts are recorded: the model ran once, outside the examples, on a CPU, with the revision pinned to `29ec3766d36d6f73f0696f85560a422f50e8498c`, and the examples read the saved output. The recording is dated Nov 10, 2026. Examples must run offline and give the same output every time, and a model call does neither. A downloaded model can change, and a reader's machine may not hold the files at all.

The input for each client and cutoff is the 1,344 hours before the cutoff, with the start rule applied. None of the standard cutoffs needs that step. Their histories begin on Nov 3, 2014 or later, after the autumn clock change, and the three late starters had their first readings before Nov 3, 2014. The manifest counts no missing values in the contexts, so the clock-change handling changes no scored number here. The package builds a mask from missing values and scales around them ([chronos-forecasting source](https://files.pythonhosted.org/packages/79/0e/f205d4a1841e3740045c40e9e8caeff584937eb561831144ca7b93a29d49/chronos_forecasting-2.3.2.tar.gz), `chronos2/model.py`, line 398). The 90 contexts went in one batch, with each series kept separate. Cross-learning, which would let one series inform another, is off by default, and the recording does not turn it on (`chronos2/pipeline.py`, line 833). So the Dec 30 context does not inform the Dec 29 forecast.

The output is, for each client, cutoff and hour, five quantiles for the next 24 hours: 0.025, 0.1, 0.5, 0.9 and 0.975. That is 2160 rows in all. The manifest lists the SHA-256 of the output and of `inputs.csv.gz`, which keeps the exact context each window was given, along with the package versions and the machine, described only as a 4-vCPU Linux container with no GPU.

Not every requested quantile is trained. The package is trained on 21 levels, from 0.01 to 0.99 ([Ansari et al. (2025)](https://arxiv.org/abs/2510.15821)). The levels 0.1, 0.5 and 0.9 are among them, so they are read directly. The levels 0.025 and 0.975 are not, so the package interpolates each one linearly between the two nearest trained levels: 0.01 and 0.05 for the lower, and 0.95 and 0.99 for the upper. The code is `interpolate_quantiles` in `chronos/utils.py`, called from `predict_quantiles` in `chronos2/pipeline.py` ([chronos-forecasting source](https://files.pythonhosted.org/packages/79/0e/f205d4a1841e3740045c40e9e8caeff584937eb561831144ca7b93a29d49/chronos_forecasting-2.3.2.tar.gz)). The manifest names the interpolated levels, and a 95% range built from them is partly the package's interpolation.

Before any score reads the recording, run the check. It hashes each output and compares it with `manifest.json`, counts the rows, clients and cutoffs in the forecast file, and looks for quantiles that run out of order.

Example `recorded-chronos2-check` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · prophet 1.5.0 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Checks the recorded Chronos-2 outputs before any score reads them: each file's SHA-256 against manifest.json, the
shape of the forecast file, and any quantiles out of order, without running the model. The point that the recording
is intact rests on the hash mismatches line and the rows with quantiles out of order line."""
from lib.compare import check_recording
from lib.settings import show

for label, value in check_recording().items():   # the default name is the chronos-2 folder in examples/data/recorded/
    show(label, value)
```

Output:

```text
outputs checked: 2
hash mismatches: 0
rows: 2160
clients: 30
windows: 3
interpolated levels: 2
rows with quantiles out of order: 0
largest quantile crossing, kWh: 0.00
inverted 80 or 95 ranges: 0
```

The recording is intact: 0 of its 2 outputs fail their hashes, and 0 rows have quantiles out of order. The check verifies the output files, not the prompt hash, which covers the input and the configuration.

The licence decides which weights the book can run or record: this book uses only Apache-2.0 or MIT weights. Chronos-2's weights and the `chronos-forecasting` package are Apache-2.0 ([model card](https://huggingface.co/amazon/chronos-2); [package metadata](https://pypi.org/pypi/chronos-forecasting/2.3.2/json)). Two other families were considered and are not used. The TimesFM 3.0 weights carry a non-commercial licence, which allows non-commercial and non-production use only ([TimesFM 3.0 licence](https://huggingface.co/google/timesfm-3.0-pytorch/blob/main/LICENSE)). TimesFM 2.5 is Apache-2.0 ([TimesFM 2.5 card](https://huggingface.co/google/timesfm-2.5-200m-pytorch)), so the licence depends on the version named. The Moirai R-series checkpoints are tagged CC-BY-NC-4.0, which is also non-commercial ([Moirai 2.0 card](https://huggingface.co/Salesforce/moirai-2.0-R-small)).

A later edition can run Chronos-2 live once its weights and package fit the release service's limits. Until then the recording stands in for the live run.

**Working rule:** Record a foundation model only with its script, its pinned revision, its package versions and a manifest that hashes every output. Read the recorded outputs in every example, and check the hashes before any score reads them. Check the licence of the weights, not only the licence of the package, and name the model version in any licence sentence.

## Score the recordings on the same cutoffs {#score-the-recording-on-the-same-cutoffs}

The recordings mean something only when they are scored on the same hours as Prophet and the baselines.

A forecast with five quantiles can be scored in two ways. The median, the 0.5 quantile, is a point forecast, scored with MASE on the book's scale. The 0.1 and 0.9 quantiles form an 80% range, and the 0.025 and 0.975 quantiles form a 95% range. Each range is scored by [coverage](#why-one-number-is-not-enough), the share of readings inside it, and by the [interval score](#calibrate-on-earlier-windows), which adds the width to a penalty for each reading outside it. The penalty is 2 divided by the nominal miss rate, 0.05 for a 95% range, times the distance outside. [Chapter 8](#calibrate-on-earlier-windows) defines the interval score with its source. Each quantile is also scored by its [quantile loss](#choose-the-level-from-the-cost-of-error), which charges tau for each kWh a reading falls above the quantile and one minus tau for each kWh below it ([Chapter 8](#choose-the-level-from-the-cost-of-error)).

Two examples score the recordings, one for each model, on the same thirty clients, cutoffs and MASE scale as [the Prophet run](#run-prophet-on-the-same-cutoffs). Each rebuilds the global model and seasonal naive instead of copying them from [Chapter 6](#score-thirty-clients-on-the-same-cutoffs), so every model goes through one scoring path.

Example `recorded-chronos2-score` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · prophet 1.5.0 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Scores the recorded Chronos-2 forecasts on the three standard cutoffs and the MASE scale of the other models: the
median is the point forecast, the 0.1 and 0.9 quantiles the 80% range, and the 0.025 and 0.975 quantiles the 95%
range. The point about the lead rests on the mean MASE lines, and the code reads the recording only, never the
model."""
import pandas as pd

from lib.compare import (global_forecasts, recorded_forecasts, report_pinball, report_range, report_subset, score,
                         seasonal_naive)
from lib.electricity import load
from lib.panel import ALL30
from lib.quirks import mask_clock_changes
from lib.scoring import report_scores
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
# clock-change hours set to missing, as chapter 3 sets them
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]
cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)
# the recording, read only after each output matches its manifest hash; its median is the Chronos column
forecasts = recorded_forecasts().merge(
    seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(
    global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")
models = ["Chronos", "SeasonalNaive", "Global"]
# per_client: one MASE per client and model; scored: forecasts beside readings
per_client, scored = score(raw, forecasts, models, ALL30)

show("clients scored", scored["unique_id"].nunique())
show("scored hours", len(scored))
show("clients where Chronos beats seasonal naive", int((per_client["Chronos"] < per_client["SeasonalNaive"]).sum()))
report_scores(scored, per_client, models)
report_subset(per_client, models)
report_pinball(scored, "Chronos", "q10", 0.1)                       # quantile loss at tau 0.1: the 0.1 quantile column
report_pinball(scored, "Chronos", "q50", 0.5)                       # at tau 0.5 the loss is half the absolute error
report_pinball(scored, "Chronos", "q90", 0.9)
report_range(scored, "Chronos", "q10", "q90", 80)
report_range(scored, "Chronos", "q025", "q975", 95)
```

Output:

```text
clients scored: 30
scored hours: 2160
clients where Chronos beats seasonal naive: 27
MAE Chronos: 100.13
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE Chronos: 655.39
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE Chronos: 0.81
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE Chronos SUBSET8: 0.87
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
quantile loss 0.1 Chronos: 39.44
quantile loss 0.5 Chronos: 50.07
quantile loss 0.9 Chronos: 17.85
coverage 80 Chronos: 0.749
interval score 80 Chronos: 572.89
coverage 95 Chronos: 0.944
interval score 95 Chronos: 958.19
```

Example `recorded-nhits-score` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · prophet 1.5.0 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Scores the recorded NHITS forecasts on the same three cutoffs and MASE scale as Chronos-2 and the other models: the
median is the point forecast, and each range comes from levels trained directly, so none is interpolated. The point
about its ranges rests on the coverage and interval score lines, and the code reads the recording only, never the
model."""
import pandas as pd

from lib.compare import (global_forecasts, recorded_forecasts, report_pinball, report_range, report_subset, score,
                         seasonal_naive)
from lib.electricity import load
from lib.panel import ALL30
from lib.quirks import mask_clock_changes
from lib.scoring import report_scores
from lib.settings import EVAL_CUTOFFS, show

raw = load(ALL30)
# clock-change hours set to missing, as chapter 3 sets them
df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]
cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)
# the NHITS recording, read only after each output matches its manifest hash; its median is the NHITS column
forecasts = recorded_forecasts("nhits", "NHITS").merge(
    seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(
    global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")
models = ["NHITS", "SeasonalNaive", "Global"]
# per_client: one MASE per client and model; scored: forecasts beside readings
per_client, scored = score(raw, forecasts, models, ALL30)

show("clients scored", scored["unique_id"].nunique())
show("scored hours", len(scored))
show("clients where NHITS beats seasonal naive", int((per_client["NHITS"] < per_client["SeasonalNaive"]).sum()))
report_scores(scored, per_client, models)
report_subset(per_client, models)
report_pinball(scored, "NHITS", "q10", 0.1)                         # quantile loss at tau 0.1: the 0.1 quantile column
report_pinball(scored, "NHITS", "q50", 0.5)                         # at tau 0.5 the loss is half the absolute error
report_pinball(scored, "NHITS", "q90", 0.9)
report_range(scored, "NHITS", "q10", "q90", 80)
report_range(scored, "NHITS", "q025", "q975", 95)
```

Output:

```text
clients scored: 30
scored hours: 2160
clients where NHITS beats seasonal naive: 24
MAE NHITS: 136.61
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE NHITS: 649.41
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE NHITS: 1.03
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE NHITS SUBSET8: 1.17
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
quantile loss 0.1 NHITS: 27.66
quantile loss 0.5 NHITS: 68.31
quantile loss 0.9 NHITS: 40.58
coverage 80 NHITS: 0.700
interval score 80 NHITS: 682.37
coverage 95 NHITS: 0.877
interval score 95 NHITS: 1250.94
```

| Model | Mean MASE, ALL30 | Mean MASE, SUBSET8 | MAE, ALL30 (kWh) |
| --- | --- | --- | --- |
| Seasonal naive | 1.18 | 1.16 | 148.26 |
| Global model (Chapter 6) | 0.91 | 1.03 | 122.81 |
| Chronos-2, median | 0.81 | 0.87 | 100.13 |
| NHITS, median | 1.03 | 1.17 | 136.61 |
| Prophet, point forecast | 1.35 | 1.39 | 182.44 |

Chronos-2 has the lowest mean per-client MASE of the five models on both sets of clients: 0.81 over the thirty clients and 0.87 on `SUBSET8`. It beats seasonal naive on 27 of the thirty clients. The global model from [Chapter 6](#score-thirty-clients-on-the-same-cutoffs) is second on both sets. [Chapter 4's fixed seasonal ARIMA](#fit-a-fixed-seasonal-arima), on the same eight clients, cutoffs and scale, scores 1.09 on `SUBSET8`, between the global model and seasonal naive.

NHITS is third on the thirty clients and fourth on `SUBSET8`, where it is narrowly behind seasonal naive, 1.17 against 1.16. Prophet has the highest mean on both sets.

The lead needs its caveat. Chronos-2's pretraining corpus includes this dataset ([the recording section](#load-a-recorded-foundation-model)), so these readings may have been part of its training, and its score here is an upper bound on what to expect on readings it has not seen. NHITS has no such advantage, because each of its cutoffs was trained only on that cutoff's own context. NHITS trails Chronos-2 on the same hours, but the two are not on equal footing, and this chapter does not rank them on this comparison.

The median is also a check on the quantile arithmetic. The pinball loss at 0.5 is half the absolute error, so the median's loss should be half its MAE. For Chronos-2 the loss at 0.5 is 50.07, against the MAE of 100.13 kWh divided by two. For NHITS the loss at 0.5 is 68.31, against half its MAE of 136.61 kWh.

Chronos-2's 95% range covers 0.944 of the readings, close to its level, and its 80% range covers 0.749, a little under its level of 0.80. The Chronos-2 95% range is partly interpolated, so its score is for the model and the package together.

Prophet's 95% range has an interval score a little lower than Chronos-2's, 932.66 against 958.19, but its coverage of 0.918 sits further from its level.

NHITS's ranges are too narrow for their levels on these hours: its 80% range covers 0.700 of the readings, and its 95% range covers 0.877.

Its 95% interval score is the highest of the three 95% ranges, and its quantiles are trained directly, so no range is interpolated. The recording does not put its quantiles in order, though. The check counts 22 rows with an adjacent pair out of order, by at most 21.11 kWh, and 0 inverted 80% or 95% ranges. The scores use the values as recorded.

Three limits apply. These are three cutoffs in one week of one dataset, so the results are not a general claim about any model. Neither range was calibrated on the calibration windows that [Chapter 8](#calibrate-on-earlier-windows) used, so the coverage here is a check, not a calibration. The comparison makes no choice, because decisions in this book are made on decision windows, and the recordings have no forecasts there, so nothing is picked with these scores.

**Working rule:** Score a recorded forecast on the same windows, the same scale and the same readings as every other model, and name any interpolated quantile wherever its range is reported. Before you read a lead, check what the model was trained on, because a model that saw the scored readings cannot show skill on them. Do not choose a model or a setting on these windows. A recorded forecast with no forecasts on the decision windows cannot be chosen for production until it has them.

## Why the neural model is recorded {#why-this-edition-has-no-neural-model}

This edition runs no neural model. Its two neural forecasts are recorded, and the reason is the cost of the runtime, not a verdict on neural models.

NHITS is a neural forecaster from the neuralforecast library, which packages neural forecasters for panels of series. The library is Apache-2.0 ([neuralforecast metadata](https://pypi.org/pypi/neuralforecast/3.3.0/json)). We record NHITS the way Chronos-2 is recorded, with its own script, pinned version and manifest, and score it beside Chronos-2 on the same contexts, cutoffs, scale and metrics. Its recording trains once for each cutoff on the same thirty 1,344-hour contexts that Chronos-2 was given. The input window is 168 hours, one week, and the horizon is 24 hours. The five levels are trained directly under the MQLoss loss, with one output for each level. The scaler is neuralforecast's standard scaler, the seed is fixed, and the run is on a CPU. Each cutoff's model sees only that cutoff's context, so no reading after the cutoff reaches it. The weights are not shipped.

NHITS gets the same check, against the hashes in its own manifest.

Example `recorded-nhits-check` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · prophet 1.5.0 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Checks the recorded NHITS outputs before any score reads them: each file's SHA-256 against manifest.json, the shape
of the forecast file, and any quantiles out of order, without running the model. The point that the recording is
intact rests on the hash mismatches line; the interpolated levels line shows that every level is trained directly, and
the rows with quantiles out of order line is the one the chapter reports."""
from lib.compare import check_recording
from lib.settings import show

for label, value in check_recording("nhits").items():   # "nhits" is the NHITS folder in examples/data/recorded/
    show(label, value)
```

Output:

```text
outputs checked: 1
hash mismatches: 0
rows: 2160
clients: 30
windows: 3
interpolated levels: 0
rows with quantiles out of order: 22
largest quantile crossing, kWh: 21.11
inverted 80 or 95 ranges: 0
```

The NHITS files match their manifest, and none of its levels is interpolated, because each one is trained directly.

Two costs keep NHITS out of the examples. Its library brings torch, ray and pytorch-lightning, which take the examples' environment from 687 MB to 1.7 GB, and a CPU training run takes longer than an example should on the release service's 2 vCPU and 4 GiB of memory. No version conflict is involved. The NHITS requirements file, `recordings/requirements-nhits.txt`, installs neuralforecast with the same pandas version the examples pin (2.3.3).

Chronos-2 has its own stack. Its requirements file, `recordings/requirements.txt`, installs `chronos-forecasting` with transformers and torch, and it pins pandas 3 for the recording. The package does not need pandas 3. Its metadata accepts pandas from 2.0 up to, but not including, 4 ([chronos-forecasting source](https://files.pythonhosted.org/packages/79/0e/f205d4a1841e3740045c40e9e8caeff584937eb561831144ca7b93a29d49/chronos_forecasting-2.3.2.tar.gz), `pyproject.toml`). The examples never import either package. They read the saved outputs.

NHITS can run live in a later edition, given a runtime that fits the release service's limits or one of its own. Either way the windows, the scale and the scoring in this chapter stay the same, so a later model is scored beside the others on the same hours.

**Working rule:** Keep a model out of the examples' runtime when its dependencies would break the runtime budget, and record it instead, with its pins in a separate requirements file. Before you add a dependency, install it in a clean environment and measure what it adds in size and in packages. A resolve that moves a pinned package is a reason to record the model, not a reason to change the pin.

## What goes wrong in production {#what-goes-wrong-ch9}

Prophet's holiday table covers 2013 and 2014 only. The book's runs leave it off, because the holiday decision windows gave no reason to turn it on, but the standard cutoffs show that it moves the result. A forecast made in 2015 with the table unchanged gets no holiday effect on later holidays, and the run reports nothing wrong. A team that turns the table on must extend it to the forecast's years. Holidays are known in advance, so the fix is to extend the table. Count the holiday hours in each window as well. A window that contains a holiday should show a count above zero, and a count of zero where a holiday falls is the sign to look for.

Interval width fails differently: a Prophet interval left at its default of 80% gets read as a 95% range by a reader who sees only the word "interval" in a report. The coverage line is the check. Printed beside its level, a coverage near 0.8 is correct for an 80% interval and a failure for a 95% one. Print the level with the coverage every time.

A recorded forecast goes stale in a different way. The recording covers three cutoffs. A day-ahead forecast in production needs one cutoff every day, and the release service does not run the model. Covering another cutoff means running the model, saving its output and hashing it, which is work outside the examples. The sign that the recording is out of date is a window with no rows in the file, which the check's window count shows.

A later model revision changes every number that depends on the recording, even when the code does not change. Treat the revision as an update trigger. Record it, score it on the same windows as the old one, and compare the two before the replacement recording is used. A package that changes its interpolation between versions changes the 95% range without touching the median, so the manifest's list of interpolated levels should be compared as well.

A recorded forecast can also break the order of its own quantiles. The NHITS recording does in 22 of its 2160 rows, by at most 21.11 kWh. A range built from a crossed pair would be read as inverted. The check counts the crossings, and a consumer should sort or reject such rows before it uses them.

Prophet's trend assumption fails when a client's demand moves to a different level, as it does after a change of contract or the loss of a large load. Prophet expects the trend to change as often as it did in the history, and its default interval does not cover the seasonal pattern. The sign is residuals that keep one sign across successive windows, and a coverage line that falls over them. The fix is a refit on a shorter history for that client, not a wider interval.

**Working rule:** Before any Prophet or recorded forecast is reported, check that the holiday table covers the forecast's years when the table is on, that the interval level is printed beside its coverage, that the recording has a window for every cutoff in use, that its quantiles are in order, and that the model revision matches the one scored. A failed check stops the report, and the report says which check failed.

## Decisions this chapter made {#decisions-ch9}

- **Prophet runs on all thirty clients, not only on `SUBSET8`.** Running all thirty puts Prophet beside the global model on the same clients. The `SUBSET8` means are printed from the same run.
- **Prophet's interval is 95%, set in code and printed.** Its coverage and interval score then sit beside the recordings' 95% ranges. The default of 80% is not used.
- **Prophet's seasonality flags are fixed in code.** Daily and weekly seasonality are on and yearly is off. These are the choices Prophet makes for eight weeks of hourly data, set explicitly.
- **The holiday table is off.** On the holiday decision windows the table raised the mean MASE, so the simpler model was kept. On the standard cutoffs the table would have lowered Prophet's mean MASE, and we report that figure without using it to choose.
- **Chronos-2 and NHITS are recorded, not run.** The examples read the saved outputs, check their hashes, and never call a model, because examples must run offline and give the same output every time.
- **NHITS is recorded for its runtime cost.** Its dependencies would take the examples' runtime from 687 MB to 1.7 GB, past the release service's 2 vCPU and 4 GiB, and a CPU training run takes longer than an example should.
- **Chronos-2's 95% range is scored with its two interpolated quantiles, and the chapter says so.** NHITS's quantiles are trained directly, so none of its ranges is interpolated.
- **The Chronos-2 lead is read as an upper bound.** Its pretraining corpus includes this dataset, so its score here is not out-of-sample.
- **Choices come only from the decision windows.** The recordings have no forecasts there, so this chapter makes no choice with them.

# Keep the forecast running

Once a forecast runs every day, the question changes from whether it scored well to whether it still holds on the day it's used. This chapter builds the loop that answers that question. You'll refit a forecast on a schedule, log each day's forecast before its readings exist, check that the log used only what had arrived, score the readings as they come in, flag a day whose error is out of line with the backtest, and say when a model should be retired.

The stake is the next buying round. The supplier commits to power for the next day from the forecast, so a forecast that has gone wrong without anyone noticing costs money in that purchase. This chapter does not reduce that risk to zero. It makes the risk visible on the day it appears, using only what the supplier knew at the time.

The earlier chapters checked the supplier's forecast on past data. Each method was scored after the fact on the last days of 2014, with every reading in hand. In operation the order runs the other way. The supplier buys power for Dec 29, 2014 from a forecast made at 00:00 on Dec 29, the day's cutoff, using only what it knows at that time, and it learns what happened a day later. A forecast that held up on the backtest can drift. The clients change, a holiday falls inside the window, or a meter feed stops for a few hours. The supplier needs to see that before the next buying round, not in a review at the end of the quarter.

This chapter runs the loop for the last seven days of 2014, from Dec 25 to Dec 31. Each day gets one forecast for each of the thirty clients, made from its own midnight cutoff. Every time in this chapter is a data time. A [cutoff](#the-case-and-its-data) is 00:00 of its day, and a reading arrives at the end of its hour. The decision and calibration runs start on Nov 25 and Dec 9, 2014, and the refit cycle is anchored to Monday, Nov 24, 2014. All of these start after Jan 16, 2014. That is the first midnight at which the start rule admits all thirty clients into one model: `MT_115`'s first reading is at 00:00 on Jan 14, 2014, and the rule needs 48 hours of history from it, as [Chapter 6](#apply-the-start-rule-to-every-client) sets out.

A forecast says what the readings will do. It does not say why they do it. The loop can tell the supplier that demand has moved away from the forecast. It cannot tell the supplier which habit, price or weather caused the move, and this chapter does not try. A forecast that holds up is not a causal claim, and no flag in this chapter turns into one.

## Refit on a schedule {#refit-on-a-schedule}

A refit re-estimates the model's parameters on the history known at a cutoff. Between refits, the model keeps its trees. Each forecast still reads the most recent readings to build its lag inputs, so the forecast reflects the last hours even when the trees do not.

One detail is often missed. mlforecast 1.1.0 fits each client's scaling again whenever it predicts from a new history: `predict` with `new_df` fits the target transforms on that history (`forecast.py` lines 1374 to 1412, and `core.py` lines 536 to 540 of the pinned release). Each client's mean and spread therefore come from the history known at every cutoff, and only the trees wait for a refit. That is not leakage, because that history is known at the cutoff.

The word needs one caution. In Hyndman and Athanasopoulos's *Forecasting: Principles and Practice*, 3rd ed. (FPP3), `refit()` applies a fitted model without re-estimating it ([FPP3, Chapter 13](https://otexts.com/fpp3/practical.html)). Here, "refit" means re-estimating the trees.

A refit must not use the readings it is about to forecast. A refit at cutoff `c` may read only readings that arrived by `c`. The loop enforces this by construction. The history for a refit, and for every forecast, holds only readings that had arrived by the cutoff, so the actuals of the day being forecast are never in it.

The cadence is the choice that remains. We considered two cadences, a refit every day or a refit every seven days. Both forecast every day from its own cutoff, with the same global model from [Chapter 6](#one-model-for-many-clients): one scaled LightGBM over the thirty clients, with lags of 24 and 168 hours and the client identifier. The global model runs here because it is the best choosable model on the decision windows: the recorded models in Chapter 9 have no forecasts there, so they cannot be chosen. They differ only in how often the trees are re-estimated. The weekly cycle is anchored to Monday, Nov 24, 2014, so its refits fall on Mondays: Nov 24, Dec 1, Dec 8 and so on. A run's refit dates therefore do not depend on the day the run starts. Each forecast uses the model fitted on the Monday on or before its cutoff.

We choose between the cadences on the four ordinary [decision windows](#choose-on-decision-windows), Nov 25, Nov 26, Dec 2 and Dec 3, 2014. None of them is a Portuguese holiday, and the held-out week is not used. The rule is the lower mean [MASE](#scale-errors-with-mase) on those windows. Every run in this chapter uses one scale per client, read from readings before Nov 25, 2014 under the rule in [Chapter 2](#take-the-mase-scale-from-history). The decision windows, the calibration windows and the held-out week share that yardstick. The decision and calibration histories reach back over the clock change of Oct 26, 2014. The loop fills that 01:00 hour from the hours either side, as [Chapter 3](#the-october-hour-that-holds-two) does.

Both cadences run on the four decision windows. Because the weekly model is one or two days old there, a model of each age from 0 to 6 days also forecasts those days.

Example `refit-cadence-decision` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Choose the refit cadence on the four decision days, Nov 25 to Dec 3, 2014: a refit every day or every seven days on
Mondays, with the global model from Chapter 6. The cadence with the lower `mean MASE` line is the one the rule takes, and
the `model aged` lines show how far a model's age alone moves that score. The held-out week is not used."""
import pandas as pd

from lib.electricity import load
from lib.panel import ALL30
from lib.running import DECISION_DAYS, daily_mase, feed, fit_at, forecast_at, prepare, run_days, scale_from, with_actuals
from lib.settings import show

raw = load(ALL30)
df = prepare(raw)
scale = scale_from(raw)  # one MASE scale, read before Nov 25, used by every run in the chapter
show("decision days", len(DECISION_DAYS))
# feed returns (train, known, pending count, read_by); [0] is train
show("rows read by one refit", len(feed(df, DECISION_DAYS[0])[0]))
names = {1: "daily", 7: "weekly"}  # the keys are the refit interval in days
means = {}
for every in names:
    scored = with_actuals(run_days(df, DECISION_DAYS, refit_every=every), df)  # refit_every: a refit every this many days
    means[every] = daily_mase(scored, scale).mean()  # the mean over the four decision days of each day's MASE
    show(f"refits, {names[every]}", scored["refit"].nunique())  # the distinct refit days the decision days use
    show(f"MAE, {names[every]}", scored["ae"].mean())
    show(f"mean MASE, {names[every]}", means[every], digits=3)
show("mean MASE, daily minus weekly", means[1] - means[7], digits=3)  # positive when the weekly cadence has the lower mean
show("chosen cadence", names[min(means, key=means.get)])  # the lower mean wins; the close-call tie-break is made in the prose
models = {}
for age in range(7):
    parts = []
    for day in DECISION_DAYS:
        fit_day = day - pd.Timedelta(days=age)  # a model aged `age` days, fitted that many days before the decision day
        if fit_day not in models:  # a model is fitted once and reused by every decision day it serves
            models[fit_day] = fit_at(df, fit_day)[0]  # fit_at returns (model, read_by); [0] is the model
        # forecast_at returns (forecast, pending count, read_by); the first is the forecast
        part, _, _ = forecast_at(models[fit_day], df, day)
        parts.append(part.assign(cutoff=day))  # daily_mase groups each forecast by its cutoff
    show(f"mean MASE, model aged {age} days", daily_mase(with_actuals(pd.concat(parts), df), scale).mean(), digits=3)
```

Output:

```text
decision days: 4
rows read by one refit: 40320
refits, daily: 4
MAE, daily: 66.81
mean MASE, daily: 0.754
refits, weekly: 2
MAE, weekly: 67.64
mean MASE, weekly: 0.746
mean MASE, daily minus weekly: 0.008
chosen cadence: weekly
mean MASE, model aged 0 days: 0.754
mean MASE, model aged 1 days: 0.747
mean MASE, model aged 2 days: 0.748
mean MASE, model aged 3 days: 0.741
mean MASE, model aged 4 days: 0.748
mean MASE, model aged 5 days: 0.758
mean MASE, model aged 6 days: 0.750
```

On the decision windows the weekly cadence has the lower mean MASE, by 0.008, and it needs half as many fits on those four days: the daily cadence runs 4 refits, and the weekly cadence 2. Each refit reads 40320 rows, which is thirty clients times 1,344 hours of history. In operation a weekly cadence needs one seventh of the fits.

A model up to six days old scores near a fresh one on the decision windows, and that is all the age check claims. Its mean MASE runs from 0.741 at three days, the lowest of ages 0 to 6, to 0.758 at five, the highest, with a fresh model at 0.754. Age shows no trend, and the spread across all seven ages is wider than the gap between the two cadences. That gap is small, and four days cannot separate the cadences, so the example does not show that weekly is better. A reader with a larger model or a longer horizon should run the same check on their own decision windows.

The rule picks weekly, because its mean is lower. That gap sits inside the spread that model age alone produces, so the two cadences are close. When they are, the working rule takes the one with fewer refits, and the book says so here: weekly.

**Working rule:** Choose the refit cadence on ordinary days that come before the held-out week, scored on one scale that was read before the first of those days. Take the cadence with the lower mean MASE, and when the two are close, take the one with fewer refits and say so. Never refit on readings from the day you are forecasting.

## Log every forecast before its actuals arrive {#log-every-forecast-before-actuals}

A forecast log is a table with one row for each client and hour. Each row holds the cutoff the forecast was made at, the time the row was written, the client, the hour, the forecast, its lower and upper bounds, the refit day of the model, two arrival times described below, and the count of readings filled for that forecast. The log has one job. It proves what the forecast said at the time, and what the loop could see when it said it. Without it, nobody can tell whether a later report used the forecast that was made, or a forecast rebuilt afterwards from a newer model and the actuals. A rebuilt log is leakage in disguise, because it knows what the forecast should have been.

The band in each row is built the way [Chapter 8](#build-intervals-from-residuals) builds one. It is the forecast plus the scaled residuals of the same hour, taken from the calibration windows, Dec 9 to Dec 20, 2014. They come before every held-out day and are fixed before the week begins. Each hour's residuals are pooled over the thirty clients, so each hour has 360 of them, not twelve. Chapter 8's client bands pool the same residuals and are offered at 95% as well. This loop logs the 80% level only, because the retirement trigger in the last section reads the 80% band, and one level keeps the log and the trigger in step. The band is computed from the logged forecast and the fixed calibration residuals, so it needs no reading from the day it covers.

A log proves less than it appears to unless its checks can fail. The rule is that a model and a forecast read only readings that had arrived by the time they were made. The log therefore records two arrival times for each row. `fit_read_by` is the last arrival among the readings the model was fitted on, and it must be no later than the model's refit day. `read_by` is the last arrival among the readings the forecast read, and it must be no later than the cutoff. A comparison of `written_at` with the arrival of a reading would not test this. The replay writes `written_at` as the cutoff, so that comparison passes for any code. In production, `written_at` would come from the append-only store's clock. A replay can only label it. Both checks run on every row of the held-out week, and the model's check runs again on a deliberately leaky week, which it must catch.

Example `rolling-refit-seven-days` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Runs the held-out week, Dec 25 to Dec 31, 2014: one forecast a day from its own midnight cutoff, with the weekly refit
cycle and the late-data fill, each logged when it is made. The `log rows` lines show both arrival checks pass, and the
leaky run fails the first. The `MAE` and `coverage 80` lines give each day's mean absolute error and how often its 80%
band held. The 80% bands use calibration days Dec 9 to Dec 20, 2014, earlier than every scored day."""
import pandas as pd

from lib.electricity import load
from lib.panel import ALL30
from lib.running import CAL_DAYS, HELD_OUT_DAYS, REFIT_EVERY, band, coverage_by_day, prepare, run_days, scale_from, with_actuals
from lib.settings import show

raw = load(ALL30)
df = prepare(raw)
scale = scale_from(raw)
cal = with_actuals(run_days(df, CAL_DAYS), df)  # the residuals that set the bands, from days earlier than the week
bands = band(with_actuals(run_days(df, HELD_OUT_DAYS), df), cal, scale)
# leak=True refits each day's model on the next day's readings
leaky = with_actuals(run_days(df, HELD_OUT_DAYS, leak=True), df)
# refit dates in the week; the Dec 22 refit is dated before it
in_week = sorted({d for d in bands["refit"] if d >= HELD_OUT_DAYS[0]})
show("refit every (days)", REFIT_EVERY)
show("refit days in the week", ", ".join(f"{d:%Y-%m-%d}" for d in in_week))
show("models used in the week", bands["refit"].nunique())
show("log rows", len(bands))
# hour 0 holds 12 calibration days times 30 clients
show("calibration residuals per hour, pooled over clients", cal.dropna(subset=["e"]).groupby("hour").size().iloc[0])
# a model may read only readings that had arrived by its refit day
show("log rows whose model read only readings in by its refit day", int((bands["fit_read_by"] <= bands["refit"]).sum()))
# a forecast may read only readings that had arrived by its cutoff
show("log rows whose forecast read only readings in by its cutoff", int((bands["read_by"] <= bands["cutoff"]).sum()))
show("leaky log rows whose model read only readings in by its refit day", int((leaky["fit_read_by"] <= leaky["refit"]).sum()))
# a reading arrives at the end of its hour, so any other arrival is late
show("readings that arrived late", int((bands["arrives"] != bands["ds"] + pd.Timedelta(hours=1)).sum()))
# every row of a cutoff has the same count, so the first row gives it
show("forecast inputs filled, Dec 29", int(bands.loc[bands["cutoff"] == "2014-12-29", "filled"].iloc[0]))
# the last of Dec 28's readings to arrive
done = bands.loc[bands["cutoff"] == "2014-12-28", "arrives"].max()
show("Dec 28 complete at", f"{done:%H:%M on %b %d, %Y}")
mae = bands.groupby("cutoff")["ae"].mean()
cov = coverage_by_day(bands)
for day in mae.index:
    show(f"MAE {day:%Y-%m-%d}", mae[day])
for day in cov.index:
    show(f"coverage 80 {day:%Y-%m-%d}", cov[day], digits=3)
show("leaky MAE 2014-12-25", leaky.loc[leaky["cutoff"] == "2014-12-25", "ae"].mean())
```

Output:

```text
refit every (days): 7
refit days in the week: 2014-12-29
models used in the week: 2
log rows: 5040
calibration residuals per hour, pooled over clients: 360
log rows whose model read only readings in by its refit day: 5040
log rows whose forecast read only readings in by its cutoff: 5040
leaky log rows whose model read only readings in by its refit day: 0
readings that arrived late: 6
forecast inputs filled, Dec 29: 6
Dec 28 complete at: 09:00 on Dec 29, 2014
MAE 2014-12-25: 447.59
MAE 2014-12-26: 203.43
MAE 2014-12-27: 54.14
MAE 2014-12-28: 71.67
MAE 2014-12-29: 111.64
MAE 2014-12-30: 87.37
MAE 2014-12-31: 167.15
coverage 80 2014-12-25: 0.232
coverage 80 2014-12-26: 0.431
coverage 80 2014-12-27: 0.790
coverage 80 2014-12-28: 0.764
coverage 80 2014-12-29: 0.713
coverage 80 2014-12-30: 0.782
coverage 80 2014-12-31: 0.701
leaky MAE 2014-12-25: 351.79
```

The log has 5040 rows, one for each of thirty clients' 24 hours on seven days, and 5040 pass the model's arrival check, and 5040 pass the forecast's. The week uses two models. The one fitted on Monday, Dec 22, serves Dec 25 to Dec 28, and the one refitted on Monday, Dec 29, serves Dec 29 to Dec 31, so the refit falls inside the week.

A check that never fails tells you little, so the week also runs with a leak. In that run, each day's model is refitted on the next day's readings, so it has seen the day it forecasts, and the model's check fails for every row: 0 rows pass. The leak also flatters the forecast. Its MAE on Dec 25 is 351.79 kWh, against 447.59 kWh for the honest run. The check is the thing to keep. An edit that moves a refit onto the next day's readings turns the printed count to zero, and the recorded output then fails.

**Working rule:** Write each forecast's rows at its cutoff, from the readings that have arrived by then, with the refit day and both arrival times beside them. Never rebuild a past log from a newer model or from the actuals. If a day's rows are lost, mark that day as missing in the log, and do not fill it in afterwards.

## Score the log as actuals arrive {#score-the-log-as-actuals-arrive}

A reading scores the forecast logged for its hour when it arrives. The day's score is its daily MASE. The daily MASE takes each client's mean absolute error over the day's 24 hours, divides it by the client's scale, and averages the results over the thirty clients:

`daily MASE = mean over clients of (MAE over the day's 24 hours / scale)`

The scale is the one set in the [refit section](#refit-on-a-schedule), read from before Nov 25, 2014, and it does not change from day to day. A reading enters the score for its own day, and only after it has arrived, so the score never uses a value the supplier did not have at the time. That is the second half of the leakage rule in the [log section](#log-every-forecast-before-actuals). A day is scored once its last reading is in. For Dec 29, that is 00:00 on Dec 30.

Most days arrive whole. The held-out week has one day that does not. The lead client's last six readings of Dec 28, 2014, from 18:00 to 23:00, arrive at 09:00 on Dec 29. That is nine hours after the Dec 29 cutoff, so the Dec 29 forecast has to be made without them. The rule for this case is the book's own choice, and no source supports it:

- A reading that has not arrived at a cutoff is left out of that day's refit.
- For the forecast at that cutoff, the missing value is filled with the reading of the same hour a week earlier, which had arrived. The log counts how many values were filled.
- The forecast is not redone when the reading arrives.
- The reading is scored when it arrives, and its day is scored once its last reading is in.

The first option tried was to leave the missing values as they were. The library refused them. At the pinned version, 1.1.0 raised this error when the forecast was made:

```
ValueError: y column contains null values.
```

There are two ways to fill the gap. Forward fill repeats the last reading that arrived. That flattens the evening, because the lead client's 18:00 to 23:00 values all become the 17:00 value, and lag 24 carries that flat evening into the next day's evening forecast. Filling from the same hour a week earlier keeps the evening's shape. We picked the fill on the decision windows, not the held-out week. There the same six readings are made late before each decision cutoff, and both fills forecast from them.

Example `late-fill-decision` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Chooses how a late reading is filled on the decision windows, Nov 25 to Dec 3, 2014: the lead client's last six readings
of the day before arrive at 09:00 on each cutoff day, as the Dec 28 readings do in the held-out week. The
`MAE, lead client hours 18 to 23` lines decide between forward fill and a fill from the same hour a week earlier; only
those hours change, because lag 24 reads them, and the held-out week is not used."""
import pandas as pd

from lib.electricity import load
from lib.panel import ALL30
from lib.running import DECISION_DAYS, LATE_CLIENT, feed, late_copy, prepare, run_days, with_actuals
from lib.settings import show

raw = load(ALL30)
df = prepare(raw)
# feed returns (train, known, pending count, read_by); [2] is the pending count
show("readings late for each decision cutoff", feed(late_copy(df, DECISION_DAYS[0]), DECISION_DAYS[0])[2])
errors = {}
for rule in ("forward", "week"):
    parts = []
    for day in DECISION_DAYS:
        # the day before's last six readings arrive at 09:00 on this day, after its cutoff, so the fill applies
        scored = with_actuals(run_days(late_copy(df, day), [day], fill=rule), df)
        # 18:00 to 23:00 of the day scored, which lag 24 reads from the day before
        parts.append(scored.loc[(scored["unique_id"] == LATE_CLIENT) & (scored["hour"] >= 18), "ae"])
    errors[rule] = pd.concat(parts).mean()
    show(f"hours scored, {rule} fill", len(pd.concat(parts)))
    show(f"MAE, lead client hours 18 to 23, {rule} fill", errors[rule])
show("chosen late fill", min(errors, key=errors.get))
```

Output:

```text
readings late for each decision cutoff: 6
hours scored, forward fill: 24
MAE, lead client hours 18 to 23, forward fill: 7.92
hours scored, week fill: 24
MAE, lead client hours 18 to 23, week fill: 5.03
chosen late fill: week
```

The week-earlier fill is the more accurate of the two. Over the lead client's hours from 18:00 to 23:00, across 24 forecast hours, its mean absolute error is 5.03 kWh, against 7.92 kWh with forward fill. The book uses it in every run.

The held-out week shows the rule at work. 6 readings arrived late, and the Dec 29 forecast filled 6 inputs. The Dec 29 refit left the same six readings out, so the branch that leaves a reading out of a refit runs in this week too. Dec 28 was complete at 09:00 on Dec 29, 2014.

A single day's MAE does not say whether the day was bad, because the level of demand changes from day to day. That is why the flag in the next section divides by a reference.

**Working rule:** Score each day when its last reading arrives, against the forecast logged at its cutoff, on the chapter's one scale. When a reading arrives late, score it then, count it in the log, and leave the logged forecast alone. Never change a forecast after its cutoff, even when the change would improve the score.

## Flag drift against the backtest {#flag-drift-against-the-backtest}

The daily scores from the [last section](#score-the-log-as-actuals-arrive) need a line that says when a day is out of line. A flag is a question about a day, not a verdict. The rule asks whether a day's daily MASE is out of line with what the model did before. A day is flagged when its daily MASE is above twice a reference. The reference is the median daily MASE of the calibration windows, Dec 9 to Dec 20, 2014. Those days are earlier than every day the flag judges. None of them is a Portuguese holiday, although Dec 9's lag-24 readings reach back to Dec 8, Imaculada Conceição, a national holiday. That echo is one day's worth, and the median damps it.

The factor of two is the book's rule of thumb. We know of no published threshold for this kind of flag, so the number is ours, and we label it as one. We checked it on the decision windows, where the four ordinary days test whether the flag fires without a known cause.

Example `drift-flag-rolling` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Flags drift against the backtest: a day is flagged when its daily MASE is above FLAG_FACTOR times the reference, the
median daily MASE of the calibration days, Dec 9 to Dec 20, 2014, which are earlier than the held-out week. The
`largest decision day` and `decision days flagged` lines show the ordinary days staying under the line, and the `MASE`
and `held-out days flagged` lines show which held-out days cross it."""
from lib.electricity import load
from lib.panel import ALL30
from lib.running import (CAL_DAYS, DECISION_DAYS, FLAG_FACTOR, HELD_OUT_DAYS, daily_mase, flagged, prepare, run_days,
                         scale_from, with_actuals)
from lib.settings import show

raw = load(ALL30)
df = prepare(raw)
scale = scale_from(raw)
reference = daily_mase(with_actuals(run_days(df, CAL_DAYS), df), scale).median()  # the median, not the mean, so one unusual calibration day moves it little
show("reference, median daily MASE", reference)
show("flag above, MASE", FLAG_FACTOR * reference)  # FLAG_FACTOR is 2, the chapter's rule of thumb
decision = daily_mase(with_actuals(run_days(df, DECISION_DAYS), df), scale)
show("largest decision day, times the reference", decision.max() / reference)
show("decision days flagged", len(flagged(decision, reference)))
held = daily_mase(with_actuals(run_days(df, HELD_OUT_DAYS), df), scale)
for day, value in held.items():
    show(f"MASE {day:%Y-%m-%d}", value)
show("held-out days flagged", len(flagged(held, reference)))
show("flagged held-out days", ", ".join(f"{day:%Y-%m-%d}" for day in flagged(held, reference)))
```

Output:

```text
reference, median daily MASE: 0.72
flag above, MASE: 1.43
largest decision day, times the reference: 1.44
decision days flagged: 0
MASE 2014-12-25: 4.62
MASE 2014-12-26: 2.00
MASE 2014-12-27: 0.73
MASE 2014-12-28: 0.76
MASE 2014-12-29: 0.85
MASE 2014-12-30: 0.76
MASE 2014-12-31: 1.14
held-out days flagged: 2
flagged held-out days: 2014-12-25, 2014-12-26
```

No ordinary decision day is flagged at a factor of two (0 flagged). The largest decision day is 1.44 times the reference. A smaller factor, such as 1.5, would leave little margin over that day, so a slightly unusual ordinary day could be flagged. We kept two for its margin. Four ordinary days cannot show how often ordinary days get flagged.

2 of the seven held-out days are over the line, Dec 25 and Dec 26. Their daily MASE is 4.62 and 2.00, against a flag line of 1.43. Dec 31, New Year's Eve, comes next, below the line.

Both flags have a known cause. Dec 25 is Christmas Day, a Portuguese holiday. The model has no holiday input. [Chapter 5](#measure-what-each-feature-adds) tried calendar and holiday columns on the subset's decision windows, the holiday ones among them, and none beat plain lags, so [Chapter 6](#one-model-for-many-clients) gave the global model none. Dec 26 is not a holiday in Portugal, but its lag-24 reading is Dec 25. The forecast for each hour of Dec 26 reads the same hour on Dec 25, so the holiday's pattern carries into the next day. These are calendar effects, not drift, and the loop cannot tell the difference. It only flags the days. A person reads each flag and checks the calendar.

One day is not a verdict. A single high day can come from a holiday, a storm or a reading that was entered wrongly. Two flags in a row, with no calendar cause, are a reason to look at the model. The next section says what to do then.

**Working rule:** Flag a day when its daily MASE is above twice the median of the earlier calibration windows, and then ask whether a calendar cause explains it. Record the cause beside the flag when there is one. Treat two unexplained flags in a row as the start of a retirement check, never as proof that the model is broken.

## Know when to retire a model {#know-when-to-retire-a-model}

Retiring a model means stopping its use and replacing it. A refit is the lighter move, and it comes first. This chapter names three triggers for either move, and each one reads a number the loop already logs. The first retirement trigger is sustained drift: flags on consecutive days with no calendar cause. The second retirement trigger is an uncovered band: the 80% band covers well under 80% of readings on several days. The third is a change in a library the model depends on.

The second trigger needs a sense of what ordinary coverage looks like. On the ordinary days, Dec 27 to Dec 31, coverage at 80% stays between 0.701 and 0.790. Dec 29, when the late readings arrive, and Dec 31, New Year's Eve, count as ordinary days here. Dec 25 is lower, at 0.232, and Dec 26 is at 0.431, for the reason the flags gave. Dec 25 is the holiday, and Dec 26 reads it through lag 24. The bands are a little narrow even on ordinary days, as [Chapter 8](#calibrate-on-earlier-windows) found. So a day between 0.70 and 0.80 is normal here, and a day far below that is not.

To show what the flags and bands do, the chapter stages a change. It is invented for the test, not a fact about the supplier's data. From Dec 27, 2014, every reading of all thirty clients is multiplied by 0.6, as if demand had dropped by 40%. The held-out week then runs on the changed readings, with the same cadence, reference, rule and bands as before.

Example `drift-after-a-change` (Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17), passing in Edition 2026.11.11:

```python
"""Shows how a change appears in the flags: a stated test, not a fact about the data, in which every reading from Dec 27,
2014 is multiplied by 0.6, as if demand had dropped by 40%. The calibration windows are earlier than the change, so the
reference and bands are fixed before it happens. The `held-out days flagged` and `coverage 80` lines show what the flags
and bands do, with the same cadence, reference, rule and bands as the held-out runs in drift-flag-rolling and
rolling-refit-seven-days."""
import pandas as pd

from lib.electricity import load
from lib.panel import ALL30
from lib.running import (CAL_DAYS, HELD_OUT_DAYS, band, coverage_by_day, daily_mase, flagged, prepare, run_days,
                         scale_from, with_actuals)
from lib.settings import show

raw = load(ALL30)
df = prepare(raw)
scale = scale_from(raw)
cal = with_actuals(run_days(df, CAL_DAYS), df)  # calibration windows come before the change, so the reference is fixed before it
reference = daily_mase(cal, scale).median()
changed = df.copy()
after = changed["ds"] >= pd.Timestamp("2014-12-27")  # the first changed day, Dec 27, from its midnight on
changed.loc[after, "y"] = changed.loc[after, "y"] * 0.6  # a stated test: every reading from Dec 27 at 60%, as if demand fell 40%
held = with_actuals(run_days(changed, HELD_OUT_DAYS), changed)  # the changed readings go through the same loop, refits included

show("reference, median daily MASE", reference)
mase = daily_mase(held, scale)
for day, value in mase.items():
    show(f"MASE {day:%Y-%m-%d}", value)
show("held-out days flagged", len(flagged(mase, reference)))
cov = coverage_by_day(band(held, cal, scale))  # the bands use the unchanged calibration days, as in the held-out run
for day, value in cov.items():
    show(f"coverage 80 {day:%Y-%m-%d}", value, digits=3)
```

Output:

```text
reference, median daily MASE: 0.72
MASE 2014-12-25: 4.62
MASE 2014-12-26: 2.00
MASE 2014-12-27: 4.74
MASE 2014-12-28: 2.93
MASE 2014-12-29: 1.68
MASE 2014-12-30: 1.78
MASE 2014-12-31: 1.89
held-out days flagged: 7
coverage 80 2014-12-25: 0.232
coverage 80 2014-12-26: 0.431
coverage 80 2014-12-27: 0.078
coverage 80 2014-12-28: 0.179
coverage 80 2014-12-29: 0.328
coverage 80 2014-12-30: 0.328
coverage 80 2014-12-31: 0.333
```

All 7 held-out days are flagged. Dec 25 and Dec 26 are flagged for Christmas and its echo, as before, and each day from Dec 27, the first changed day, is flagged too. On Dec 27 the daily MASE is 4.74, and coverage falls to 0.078 that day, the lowest of the week.

One scheduled refit did not clear the flags. The refit on Monday, Dec 29, is the only step in this test that refits on changed readings, and it read Dec 27 and Dec 28 at the lower level. Its daily MASE fell by about 40%, from 2.93 on Dec 28 to 1.68 on Dec 29, but Dec 30 and Dec 31 stayed above the line. Coverage recovered to 0.328 on Dec 29 and 0.333 on Dec 31, still far below 80%.

A run of flags like this, with coverage that low, is a case for an early refit, and the book's rule is to refit early. We did not test an early refit on a day of its own, so that rule is a rule of thumb and not a measured result.

The third trigger is a library change. Libraries release often, and a later release can change an output even when the code stays the same. The loop depends on 1.1.0 and 4.7.0. Prophet, which Chapter 9 pins at 1.5.0, is not part of this loop, but it is another pinned library, and its release history shows how often such a library moves ([PyPI release history](https://pypi.org/pypi/prophet/json)). A release is a reason to run this chapter's checks again. It is not, by itself, a reason to retire a model.

These checks are the forecast log, the flags and the coverage of the bands. They are not a monitoring system. A monitoring system covers alerts, dashboards, data quality across many feeds and the people who respond. A later book on monitoring models in production can continue this case, and this chapter ends where that book would begin.

**Working rule:** Refit early when flags run on several days in a row with no calendar cause, or when the 80% band covers far below 80% of readings on several days, and record the trigger with the log. Retire the model when an early refit does not clear the flags. Re-run this chapter's checks when a pinned library is released, and keep the pins until the checks pass on the new version.

## What goes wrong in production {#what-goes-wrong-ch10}

A feed can stop in the middle of a history. The loop does not fill a reading the feed never delivered. A gap in the middle would shift every lag, so the code raises an error instead. Only the clock-change hours are filled, by [Chapter 3's rule](#mark-clock-change-hours-as-missing). A reading still missing at the end of a history is different, and the [late-data rule](#score-the-log-as-actuals-arrive) handles it.

A refit can leak. If a refit reads the readings of the day it is about to forecast, the backtest looks better than the live run will be. You would notice it as live MASE well above the backtest's for the same days. The loop prevents this by its history rule, and that rule is the one thing to keep when the loop is rewritten. Keep the leak checks from [Chapter 3](#check-a-backtest-for-leakage), the shift test, and from [Chapter 7](#split-a-total-by-share), the poisoning check. The poisoning check writes wrong values over every reading after a cutoff, then confirms that the refit and the forecast at that cutoff do not move.

A flag on the mean of thirty clients can hide a change in one client. A large change in one client moves the daily MASE only by the change in that client's daily MASE divided by thirty. The stress example moves every client at once, so the flags fire. We did not test a change in one client alone. You would notice it as that client's own MASE far above its calibration median. That is the first gap we would close, perhaps with a second flag on each client's own MASE.

A pinned library can still change behaviour when the pin is updated. The null-value error in [the scoring section](#score-the-log-as-actuals-arrive) is behaviour of mlforecast 1.1.0. A later version could fill, drop or accept the gap, and the forecast would change. Updating the pin is a change to the forecast, and it needs the checks to run again.

The last failure is the log written after the fact. Nothing in an example stops a team from rebuilding a log later, and the rebuilt output would look correct. The two arrival columns and their checks are what show it. When those checks are removed, the log stops proving anything.

**Working rule:** Keep three checks in every run: the history rule that keeps the future out of each refit and forecast, the two arrival checks on each log row, and the leak checks from Chapters 3 and 7. When one of them fails, stop the loop and fix the cause before the next forecast. Do not repair a history or a log by hand, because a hand repair is the kind of change the checks exist to catch.

## Decisions this chapter made {#decisions-ch10}

- **Refit cadence:** every seven days, on Mondays. Weekly had the lower mean MASE on the decision windows, by 0.008, a gap inside the spread that model age alone produces, so the tie-break applies and the book takes the cadence with fewer refits.
- **One MASE scale:** read from the readings before Nov 25, 2014, and used for every run in this chapter, following the rule in [Chapter 2](#take-the-mase-scale-from-history). A decision day, a calibration window and a held-out day are therefore scored on one yardstick, and no scored hour helped set it.
- **Flag rule:** a day is flagged above twice the median daily MASE of the calibration windows, Dec 9 to Dec 20, 2014. The factor of two is a labelled rule of thumb, kept for its margin over the ordinary decision windows.
- **Bands:** 80% only, because the retirement trigger reads that level; [Chapter 8](#calibrate-on-earlier-windows) also offers 95%. The bands use the scaled residuals of the same hour, pooled over the thirty clients, from the twelve calibration windows, fixed before the held-out week. No total forecast, total band or purchase quantile is logged, so the log checks the client forecasts and not the purchase that Chapter 8 makes against the total. A supplier who buys against the total would add that quantile to the log.
- **Late readings:** left out of the refit, filled from the same hour a week earlier for the forecast, counted in the log, scored when they arrive, and never used to redo a forecast. The fill was chosen on the decision windows. The rule has no source and is the book's own.
- **Scope:** seven days for thirty clients. A week holds a holiday, a late reading and a weekly refit that falls inside it, so it shows each rule. A longer run would measure how often ordinary days get flagged, which this chapter does not measure.
- **Monitoring boundary:** this chapter's checks only. The monitoring system that would watch them over time is outside this book.
- **Your own case:** the book's last decision hands the work back. The supplier's case ends here: a weekly refit, a log that checks what each forecast read, scoring as the readings arrive, and a flag against the backtest. Your case needs these decisions made again on your own data: the refit cadence on your ordinary decision windows, the late-data rule on the feed you actually receive, the flag factor on your calibration windows, and the band level your buying rule needs. Make each one on days before your held-out week.
