# How the 2002 back-test was done

*What the agent (Claude Opus) did on 1 Oct 2026 to answer the framework's question: "Rebuild the dashboard with data from May 2002. Would the framework have flagged Regime A?" The result is in `backtest-2002.md`; what it means for 2026 is in `02-confidence-for-2026.md`.*

## 1. The approach in one paragraph

Rebuild every month-end dashboard from Jan 1999 to Dec 2006 using only data that existed (or can be faithfully reconstructed) at each date. Run the **unchanged** live vote code with the **unchanged** 2026 bands, then three variants. Add the parts of the framework that sit outside the vote: the Layer 2 driver scores and the journal's range method. Compare everything with what actually happened. All of this is one script driven by the data room, so every number in the report is computed, not typed.

## 2. The data problems and how each was solved

| Problem | Solution | Check |
|---|---|---|
| FRED's 5y5y (`T5YIFR`) and 10-year TIPS (`DFII10`) start in 2003 | Federal Reserve Board TIPS curve (Gürkaynak-Sack-Wright, file `feds200805.csv`), which starts Jan 1999. The data room's `fed_gsw` fetcher was generalised to read either Fed Board file; three new series added: `gsw_tips10y`, `gsw_bkeven10`, `gsw_bkeven5f5` | — |
| The Fed Board 10-year TIPS yield is built differently from FRED's | Used as is | 2003–2020 overlap: mean gap +0.07pt, correlation 0.997, 216 months |
| The Fed Board 5y5y runs higher than FRED's | Lowered by the mean gap over 2003–05 (0.37pt). The raw series is run as a separate version | Gap standard deviation 0.15pt, correlation 0.83, 36 months |
| Core PCE, unemployment and GDP have been revised since 2002 | First-published values from ALFRED (the St. Louis Fed's archive of past releases): a new `fetch_vintages` and `as_of` helper in `fetchers/fred.py`, with throttling, backoff, key redaction and a local cache | Real-time and revised compared in report section 9 |
| Federal interest outlays have no ALFRED history before 2008 | Today's BEA figures, lagged two months to when each quarter was first published | A version with quarter-end dating shows the effect |
| HY credit spread (FRED keeps it only from 2023) | `BAA10Y` (Moody's Baa minus 10-year) | — |
| Fed target rate before 2008 | `DFEDTAR` | — |
| What was known on a given date | `known_by()`: period end plus each series' typical publication lag | — |

The five new series are in `dataroom/catalog/series.yaml` as row-0 (support) series. They pass all data-room checks but are not yet signed off.

## 3. The four versions of the vote

| Version | What changes |
|---|---|
| **As written (main result)** | Nothing: `derive.regime_votes` from the live data room, 2026 bands |
| Raw 5y5y | No level adjustment to the pre-2003 5y5y |
| Quarter-end stamp | Interest/receipts dated at quarter end (as the live dashboard does) |
| Direction-aware | 5y5y, TIPS and term premium scored on 12-month change and position against their own trailing 10 years |

## 4. Validation

- **Tie-out of numbers to data:** all 51 numbers quoted in the report text were checked against the CSVs. All matched.
- **Tie-out to the live data room:** for 2003–2006 the quarter-end version matches the live register in 47 of 48 months. The exception (Feb 2003) is explained: the back-test has a TIPS reading for Dec 2002 from the Fed Board, while the live series starts in Jan 2003.
- **Tests:** a new parser test for the Fed Board TIPS file; 30 tests pass. `make verify` exits 0.
- **Rendering:** the HTML embeds the five charts; checked visually and as a 15-page PDF.

## 5. A bug found in the live data room

The live vote computation started in Jan 2003 with no earlier data, so the 3-month and 12-month changes were missing for 2003's first months. `pipeline.compute_votes` now reads from 2002 and keeps votes from 2003. This changed calls only for Jan 2003 to Jan 2004: a B call for Sep 2003 to Jan 2004 disappeared. The 1 Oct 2026 journal says "B called 57 months"; after the fix it is 52. The journal was deliberately not edited (past entries are never changed).

## 6. Mistakes caught during the work

- Publication lags were first set wrongly for the snapshot; replaced by `known_by()`.
- ALFRED vintage dates were compared as text against dates; now uses timestamps.
- The business-investment peak was taken over all history; now 1995 onward.
- Chart fixes: range bars overlapped the lines in the outcome chart; the core PCE axis was too tight.
- Text fixes: a claim of "3 of the last 4 months" was false and became "two months running"; the FOMC quote was made exact; the 30-year caveat was reworded (no 30-year auctions Aug 2001 to Feb 2006).

## 7. Results

- **May 2002: A1 B2 C2, no call.** No A call in any month from Nov 2001 to Dec 2006.
- A was called only in Feb 2001 and Sep–Oct 2001, and only with the adjusted 5y5y.
- The term premium (ACM) sat in the C band for all 43 months from Jun 2001 to Dec 2004; the 10-year TIPS yield (3.16%) was above every band.
- The direction-aware variant reached 3 A votes in four months of 2002–03 but never three months running.
- Layer 2 scored **−5** of −7: the right direction.
- Real-time core PCE (1.35%) was lower than today's revised figure (1.57%): disinflation looked stronger at the time.
- Range test: the actual May 2003 10-year (3.37%) was at the 0th percentile of the method's range.
- The live data room first calls A in Oct 2015.

## 8. Limitations

- One episode. The lessons are hypotheses.
- The 5y5y adjustment was measured after the test date (2003–05).
- The ACM term premium is re-estimated on today's full sample, so its 2002 values carry hindsight.
- Interest/receipts uses revised data.
- The early TIPS market (1999–2002) was small and illiquid.

## 9. Files

| File | What it is |
|---|---|
| `03-backtests/backtest-2002.md` / `.html` | The report (generated; do not edit) |
| `dataroom/backtest/backtest_2002.py` | The whole pipeline: panels, splice, votes, drivers, range test, charts, report |
| `dataroom/backtest/2002/narrative.md` | Report text with `{{placeholders}}`; edit this to change wording |
| `dataroom/backtest/2002/data/*.csv`, `summary.json` | Every table in the report |
| `dataroom/backtest/2002/charts/*.png` | Five charts: inputs, votes, splice, outcome, core PCE vintages |
| `dataroom/data/raw/alfred/cache/` | Cached ALFRED vintages (not in git) |

## 10. How to re-run, and how to add another year

Re-run: `cd dataroom && make backtest-2002` (about a minute; ALFRED is cached after the first run).

Add a year, for example 2021–22:

1. Copy `backtest/backtest_2002.py` to `backtest/backtest_2022.py`. Change the test date, the panel window, the working-table window and the output folder.
2. Copy `backtest/2002/narrative.md` to `backtest/2022/narrative.md` and rewrite the text. Keep numbers as placeholders.
3. Add a `backtest-2022` target to the `Makefile`, modelled on `backtest-2002`.
4. Run it, check the tie-outs (numbers to CSVs, back-test votes to the live register), and add a summary to `03-backtests/` and a line to `CHANGELOG.md`.