# How this folder was built

*A record of what the agent (Claude Opus, in Cursor) built on 1 Oct 2026, in the order it was built, and the decisions behind it. For how to use and update the result, see `01-how-to-update.md`.*

## Timeline

| Step | Request | Result |
|---|---|---|
| 1 | Synthesise three AI notes and a yield chart into an analytical framework for long rates from 2026 | `02-framework/rates-regime-framework-2026.md` and `.html` |
| 2 | Build a data room that fetches, stores, checks and charts every statistic in the framework, with clear confidence that the right series are collected | `dataroom/` |
| 3 | Write a dated forecast journal with probabilities, ranges and triggers | `dataroom/journal/2026-10-01.md`, one-page version, `forecasts.csv` |
| 4 | Explain each watch metric, and show the 5y5y as a month-end grid | Appendix to the journal; `data/tables/` |
| 5 | Back-test the framework on 2002 | `03-backtests/` and `dataroom/backtest/` |
| 6 | Organise and document the folder; commit to git | This file and the rest of `00-overview/`, root `README.md`, `.gitignore` |

---

## 1. The framework

**Inputs** (in `01-source-notes/`): three AI-written notes comparing 2002 with 2026 (Claude, Gemini and OpenAI) and a chart of 10-year and 30-year Treasury yields.

**What was done:**

1. Read the chart first: what it shows, and what it cannot show (it does not separate inflation from real rates).
2. Compared the three notes: where they agree, where they differ, and what all three leave out. All three left out the decomposition of the yield, the term premium, Japan and foreign demand, and any objective test.
3. Built a four-layer framework:
   - **Layer 1:** decompose the yield into expected inflation, real rate and term premium.
   - **Layer 2:** score seven structural drivers at −1, 0 or +1.
   - **Layer 3:** classify the regime as A (disinflation, like 2002–2020), B (higher real rates, anchored inflation) or C (fiscal or inflation, like 1965–1980), with shock overlays.
   - **Layer 4:** anticipate the policy reaction.
4. A signposts dashboard: numbered metrics with explicit A/B/C bands, five of which form the headline regime vote.
5. Three scenarios with starting probabilities (B 50%, A 25%, C 25%), bond arithmetic, decision rules and an Australian investor's view.
6. A section on making the thinking concrete: a journal, scoring, and a back-test against 2002. That suggestion led to steps 3 and 5.

The HTML version was made with the plain-html skill. One fix after review: numbers in narrow table columns wrapped ("21" shown as "2" above "1"); columns were set not to wrap.

## 2. The data room

**The goal:** a store that keeps itself current and gives clear evidence that each series is the right one. The design rests on one idea: **a written contract per series, checked on every run**.

### Architecture

```mermaid
flowchart LR
  catalog["catalog/series.yaml (contract)"] --> fetch["fetchers (9 sources)"]
  fetch --> raw["data/raw (gzipped originals + sha256)"]
  fetch --> validate["validate: definition + data checks"]
  validate --> store["data/series/*.parquet (wide tables)"]
  store --> derive["derive: slopes, ratios, regime votes"]
  store --> cross["crosscheck: independent sources"]
  derive --> register["register/*.csv"]
  cross --> register
  register --> report["report/dataroom.html + charts"]
  signoff["catalog/signoff.yaml (fingerprinted)"] --> register
```

| Part | File | What it does |
|---|---|---|
| Catalog | `catalog/series.yaml` | One entry per series: source, ID, units, frequency, plausible range, start date, staleness limit, the publisher's expected title and units, A/B/C bands and notes. 86 entries, about 70 collected and 15 derived |
| Fetchers | `src/dataroom/fetchers/` | FRED (JSON API), NY Fed (ACM term premium, r*), TreasuryDirect (auctions), Fiscal Data (debt, Monthly Treasury Statement, MSPD), Treasury TIC (foreign holders), Japan MoF (JGBs), Fed Board (zero-coupon and TIPS curves), RBA (blocked, falls back to OECD), manual CSV |
| Raw store | `rawstore.py` | Saves every download gzipped with a manifest (URL with the key redacted, sha256, time). Keeps 14 days plus the first snapshot of each month |
| Validation | `validate.py` | Definition checks (does FRED's title and units match the catalog?) and data checks (empty, duplicates, range, start date, frequency, gaps, staleness). A critical failure keeps the last good data |
| Store | `store.py` | One wide Parquet table per series: date, value, 1-, 3- and 12-month changes, year-on-year %, 5-year z-score, revision flag |
| Derived | `derive.py` | Curve slopes, interest/receipts, proxy auction tails, realised volatility, and the five-input regime vote |
| Cross-checks | `crosscheck.py` | 11 checks: DGS10 against the Fed Board curve, FRED's 10y–2y against our own, MSPD against Debt to the Penny, and others |
| Anchor tests | `tests/test_anchors.py` | Known historical values (for example the 10-year at 0.52% on 4 Aug 2020) |
| Register | `register.py` | `audit_register.csv` (one row per series: source, extent, latest value, status, checks, sign-off), `crosschecks.csv`, `regime_votes.csv` |
| Charts and report | `charts.py`, `report.py` | A PNG per series (full history and last 5 years, with bands), and `report/dataroom.html` with status, the regime dashboard and a verification pack per series |
| Sign-off | `signoff` command | You record that you checked a series. The sign-off stores a fingerprint of the series definition; editing the source, ID, parameters or units voids it |
| Automation | `scripts/refresh.sh`, launchd plist | Daily 07:00 refresh, logs, macOS notification on failure |
| Agent rule | `.cursor/rules/dataroom.mdc` | Tells any Cursor agent how to refresh, how to triage failures, and what it must never do (loosen checks, sign off, expose the key) |

### Decisions worth knowing

- **Parquet, one wide table per series:** easy to read in pandas or Excel (via CSV), and each series carries its own change columns.
- **Proxies are labelled.** The MOVE index and auction when-issued tails are paid data; the data room uses realised volatility and "high yield minus same-day DGS10" instead, and marks them as proxies.
- **HY OAS accumulates.** FRED now serves only three years of ICE high-yield spreads, so the data room appends each run's data to its own history. This file cannot be rebuilt, which is why it is the one Parquet file committed to git.
- **FRED rate limit.** At your request: a 0.6-second pause between calls, metadata cached for 6 days, and 15/30/60-second backoff on HTTP 429. The key lives only in `.env`, and is redacted in URLs, manifests and error messages.

### Fixes made after review

- The Fed total assets chart caption covered the plot; the layout was changed.
- A table of the regime votes was added under the votes chart (last 24 months, with each input and its vote).

## 3. The forecast journal

- `journal/2026-10-01.md`: calls, how the probabilities were reached, three reasons, the yield ranges and why, what would change the view, and readings that would halve each probability. Two forecasters are recorded: your gut view (`ml-gut`: B 50%, C 35%, A 15%; 10-year 5–7%, 30-year 6–9%) and the data-based view (`ml-data`: B 55%, C 30%, A 15%; 10-year 4.25–6.50%, 30-year 4.90–6.75%, each at 75%).
- `journal/2026-10-01-onepage.md`: the same on one printed page.
- `journal/forecasts.csv`: one row per quantified call, with a resolve date (1 Oct 2027) for scoring.
- **Appendix A1–A9:** each watch metric explained: what it is, how it is computed, what it measures, its range in context, what would push it up, and how today compares with history.
- **5y5y month-end grid:** years as rows and months as columns in the journal; both layouts as CSV in `data/tables/` (rebuilt on every refresh, for every series).
- `scripts/journal_html.py` turns journal markdown into HTML (`make journal-html`).

## 4. The 2002 back-test

See `03-backtests/01-how-the-backtest-was-done.md` for the method and `03-backtests/02-confidence-for-2026.md` for what it means.

## 5. Organising the folder

Modelled on `../pully`: numbered top-level folders, a master `README.md`, a `CHANGELOG.md`, a single root `.gitignore`, and a local-only git repository.

- Source notes moved to `01-source-notes/` (one file renamed to drop " (1)").
- The framework moved to `02-framework/`; its image and source links updated.
- The back-test report now builds into `03-backtests/`.
- `dataroom/` was **not** renamed or moved: the Python environment and the launchd job use its absolute path.