# Rates regime data room

Collects, checks, stores and charts every statistic in `../02-framework/rates-regime-framework-2026.md`.

Open `report/dataroom.html` for the current state: status, regime dashboard, audit register, cross-checks and a verification pack (source, identifier, checks, chart and raw files for each series).

## Quick start

```bash
cd dataroom
make setup                 # .venv + requirements
cp .env.example .env       # add FRED_API_KEY (free); without it FRED falls back to keyless CSV
make refresh               # about 2 minutes
make test
open report/dataroom.html
```

## Layout

| Path | What it holds |
|---|---|
| `catalog/series.yaml` | The contract: one entry per series (source, ID, units, range, staleness limit, expected publisher metadata, A/B/C bands, known gaps) |
| `catalog/signoff.yaml` | Who verified which series, tied to a fingerprint of the definition |
| `data/series/<key>.parquet` | One wide table per collected series: date, value, chg_1m, chg_3m, chg_12m, yoy_pct, zscore_5y, revised |
| `data/derived/<key>.parquet` | Derived metrics (slopes, interest/receipts, proxy auction tails, regime votes, ...) in the same shape |
| `data/panels/` | Wide panels by frequency (daily, weekly, monthly, quarterly, annual, events) and `dashboard_monthly.parquet` |
| `data/raw/<source>/<date>/` | Gzipped original downloads with `manifest.json` (sha256, URL with key redacted). Keeps 14 days plus the first snapshot of each month |
| `register/audit_register.csv` | One row per series: source, method, frequency, extent, observation count, latest date and value, status, checks, sign-off |
| `register/crosschecks.csv` | Independent-source and identity checks |
| `register/run_log.jsonl` | One line per run |
| `backtest/backtest_2002.py` | May 2002 back-test (`make backtest-2002`): reads the store, fetches ALFRED vintages (cached in `data/raw/alfred/cache/`), writes `backtest/2002/data/*.csv`, `backtest/2002/charts/*.png` and `../03-backtests/backtest-2002.md/.html` from `backtest/2002/narrative.md` |
| `journal/` | Dated forecast entries (`YYYY-MM-DD.md/.html`, one-page versions) and `forecasts.csv`; past entries are never edited |
| `charts/<key>.png` | Full history plus last 5 years, with A/B/C bands and source caption |
| `manual/` | Hand-entered series (hyperscaler capex); copy `hyperscaler_capex.csv.template` to `hyperscaler_capex.csv` |

## How a series is trusted

1. **Definition checks** (FRED): the publisher's title, units and frequency must contain what the catalog expects. A mismatch is critical.
2. **Data checks**: non-empty, no duplicate dates, values in range, history starts when expected, declared frequency, no unexplained gaps, fresh enough.
3. **Cross-checks**: e.g. DGS10 against the Fed's zero-coupon curve, FRED's T10Y2Y against our own 10y minus 2y, MSPD against Debt to the Penny, TIC history against the current table, MoF JGB against OECD.
4. **Anchor tests**: known historical values (10y at 0.52% on 4 Aug 2020, about 15.8% in Sep 1981, and others).
5. **Sign-off**: you review the verification pack and run `make signoff`. Changing a series' source, ID, parameters or units voids its sign-off.

A critical failure keeps the previous good data and marks the series FAILED.

## Sources and methods

| Source | Method | Series |
|---|---|---|
| FRED (St. Louis Fed) | JSON API, 0.6 s throttle, metadata cached 6 days, HTTP 429 backoff 15/30/60 s | Treasury yields, breakevens, TIPS, policy rate, inflation, GDP, BEA fiscal, FX, commodities, credit, Fed balance sheet, OECD foreign yields |
| NY Fed | XLS/XLSX download | ACM term premium, Holston-Laubach-Williams r* |
| TreasuryDirect | JSON API | 10y and 30y auction results (excludes TIPS and floating-rate notes) |
| Fiscal Data (Treasury) | JSON API | Debt to the Penny, Monthly Treasury Statement table 9, MSPD |
| Treasury TIC | Text file parse | Major foreign holders (current table plus history) |
| Japan MoF | CSV download | JGB 10y and 30y |
| Federal Reserve Board | CSV download | Gürkaynak-Sack-Wright zero-coupon yields (cross-check) and TIPS/breakeven curve (back-test history from 1999) |
| ALFRED (St. Louis Fed) | JSON API, first-release vintages | Core PCE, unemployment, GDP as published at the time (back-test only) |
| RBA | CSV (blocked, HTTP 403) | Falls back to the OECD Australian 10y on FRED |

Not collected (paid): MOVE index (proxy: realised 10y volatility), auction when-issued tails (proxy: high yield minus same-day DGS10).

## Schedule

`make schedule-install` installs a launchd job (`scripts/com.ml.dataroom.refresh.plist`) that runs `scripts/refresh.sh` daily at 07:00. Logs go to `logs/`, and a macOS notification appears if the run exits non-zero. `make schedule-remove` uninstalls it.

The Cursor rule `../.cursor/rules/dataroom.mdc` tells an agent how to refresh, triage failures and report back.