# How to use and update this folder

Step-by-step instructions, ordered from the most frequent task to the rarest. All commands run in Terminal from `financialmarkets/dataroom/` unless stated otherwise. `make help` lists every command.

| How often | Task | Time | Section |
|---|---|---|---|
| Daily (automatic) | Data refresh at 07:00 | 0 min, unless a notification appears | [2](#2-daily-automatic-refresh) |
| Monthly | Read the dashboard, note the regime vote | 15–30 min | [3](#3-monthly-review) |
| When the view changes, or quarterly | Write a journal entry | 30–60 min | [4](#4-write-a-forecast-journal-entry) |
| Quarterly | Re-score the structural drivers, review probabilities | 30 min | [5](#5-quarterly-review) |
| Annually, and on resolve dates | Score past forecasts | 30 min | [6](#6-score-forecasts) |
| As needed | Add or change a series, sign off | 15 min per series | [7](#7-add-or-change-a-series) |
| As needed | Re-run or add a back-test | 5 min / a few hours | [8](#8-back-tests) |
| After any change | Commit to git | 2 min | [9](#9-commit-to-git) |

---

## 1. One-time setup (new machine, or after moving the folder)

1. Check Python 3 is installed (`python3 --version`; the current environment uses 3.13).
2. `cd financialmarkets/dataroom`
3. `make setup` creates `.venv/` and installs `requirements.txt`.
4. `cp .env.example .env`, then put your free FRED key in `.env` as `FRED_API_KEY=...` (get one from [fred.stlouisfed.org](https://fred.stlouisfed.org/docs/api/api_key.html)). Never paste the key anywhere else.
5. `make refresh` takes about 2 minutes. Exit code 0 means everything passed.
6. `make test` should report all tests passing.
7. `make schedule-install` installs the daily 07:00 job. If the folder has moved, first edit the three absolute paths in `scripts/com.ml.dataroom.refresh.plist`.
8. `open report/dataroom.html`.

## 2. Daily: automatic refresh

The launchd job runs `scripts/refresh.sh` at 07:00. It fetches every series, checks it, stores it, recomputes the derived metrics and regime votes, and rebuilds the charts and `report/dataroom.html`.

- **No notification** means all is well. Nothing to do.
- **A "Data room: attention needed" notification** means a series FAILED or went STALE, or a critical cross-check failed. Then:
  1. Open `report/dataroom.html`; the "Needs attention" box lists the series.
  2. Or ask the Cursor agent: *"Refresh the data room and tell me what is wrong."* It follows `.cursor/rules/dataroom.mdc`: finds the cause first, never loosens a check to get a green run, and never signs off for you.
  3. Usual causes: a delayed release (STALE; wait), the publisher changed a file (FAILED; the fetcher needs fixing), or a network error (rerun `make refresh`).
- `make schedule-status` shows whether the job is installed; logs are in `logs/`. The job runs at 07:00 and again at login, but refreshes at most once a day and never before 07:00, so a Mac that was off at 07:00 catches up when you log in. To wake a sleeping Mac for the 07:00 run: `sudo pmset repeat wakeorpoweron MTWRFSU 06:55:00` (check with `pmset -g sched`; remove with `sudo pmset repeat cancel`).
- To refresh by hand: `make refresh`, or `make refresh-one KEYS="ust10y tips10y"` for a few series.
- **What changed since the last commit:** `make changes`, or `make changes ARGS="--vs HEAD~3"` to go further back. It lists series whose latest date, value or status moved, and those with revised history.
- **See series side by side:** `make show KEYS="ust10y tips10y acm_tp10" ARGS="--freq M --last 24"`. Add `--csv FILE` or `--xlsx FILE` to save a printable copy; `--panel monthly` shows a whole panel.
- **Printable audit of every series:** `make report-pdf` writes `report/dataroom-audit.pdf`, with a list of all tables. The refresh does not rebuild it.

## 3. Monthly review

1. `open report/dataroom.html`.
2. Read **Status**: the counts of OK, FAILED and STALE series, and the cross-checks.
3. Read **Regime dashboard**: the headline vote (for example "A1 B2 C0, call: none") and the table of the last 24 months.
4. For each headline metric, ask the framework's question: **which component moved?** Was it the breakeven (inflation), the real yield, or the term premium?
5. Check the "what would change my mind" and "halve the probability" triggers in the latest journal entry against the dashboard. If one has fired, write a new journal entry (section 4).
6. Remember the back-test lesson: the vote **confirms** a regime after it has arrived; it does not forecast one. Give direction of travel and the structural drivers equal weight.

## 4. Write a forecast journal entry

Entries live in `dataroom/journal/`. **Never edit a past entry.** Each revision is a new file.

1. Copy the structure of the latest entry (`2026-10-01.md` and `2026-10-01-onepage.md`) into new files named with today's date: `YYYY-MM-DD.md` and `YYYY-MM-DD-onepage.md`.
2. Fill in:
   - the date and data-as-of line;
   - regime probabilities (A, B, C) with how you reached them;
   - 12-month ranges for the 10-year and 30-year, with reasons;
   - three reasons for the view;
   - what would change your mind;
   - two or three dashboard readings that would halve each probability.
3. Add one row per quantified call to `journal/forecasts.csv`, with `forecaster` set to `ml-gut` or `ml-data` and a `resolve_date`. Leave `outcome` and `hit` empty.
4. `make journal-html` regenerates the `.html` copies.
5. Check that the one-pager prints on one page (Print → PDF in a browser).
6. Add a line to `CHANGELOG.md` and commit (section 9).

An agent can draft this: *"Write a new forecast journal entry for today using the data room."* It will not touch past entries.

## 5. Quarterly review

1. Re-score the seven structural drivers (framework Layer 2: savings vs investment, fiscal path, demographics, productivity, globalisation and energy, central-bank credibility, safe-asset demand) at −1, 0 or +1, each with a one-line reason and a date. Record them in the journal entry.
2. Compare the driver total with last quarter. **The change matters more than the level.** In 2002 this layer pointed the right way while the headline vote did not.
3. Review the regime probabilities and write a new journal entry if they move by 10 points or more.
4. Glance at `register/audit_register.csv` for series with no sign-off.

## 6. Score forecasts

On each `resolve_date` (the first ones fall on 1 Oct 2027):

1. In `journal/forecasts.csv`, fill in `outcome` (the actual value, or which regime occurred) and `hit` (1 or 0) for the rows that resolve. This is the only edit allowed to past rows.
2. Ranges: aim for about 75% hits over time. Many more hits means the ranges are too wide; many fewer means they are too narrow (the 2002 back-test suggests narrow is the usual error at turning points).
3. Regime probabilities: compute the Brier score (the mean of (probability − outcome)², where lower is better), separately for `ml-gut` and `ml-data`, to see which is better calibrated.

## 7. Add or change a series

1. Add or edit the entry in `catalog/series.yaml`: `key`, `row` (dashboard row), `source`, `source_id`, `freq`, `units`, `range`, `start_by`, `max_stale_days`, `expect` (FRED title and units), thresholds if it has bands, and a note explaining anything unusual.
2. `make refresh-one KEYS="newkey"`.
3. Look at `charts/newkey.png` and the series' row in `report/dataroom.html`. Check that the last value matches the publisher's website.
4. If possible, add a known historical value to `tests/test_anchors.py`, then `make test`.
5. Sign off when you are satisfied: `make signoff KEYS="newkey" BY="ml"`. Only you do this, never an agent. Editing a series' source, ID, parameters or units voids its sign-off, and the register shows it as unverified.
6. Five back-test support series added on 1 Oct 2026 are not yet signed off: `gsw_tips10y`, `gsw_bkeven10`, `gsw_bkeven5f5`, `baa10y`, `fed_funds_target_pre2008`.

## 8. Back-tests

- **Re-run the 2002 back-test** (after a data refresh or a code change): `make backtest-2002`. This rewrites `03-backtests/backtest-2002.md/.html`, the CSVs in `backtest/2002/data/` and the charts in `backtest/2002/charts/`.
- **Change its wording:** edit `backtest/2002/narrative.md`, not the generated report. Numbers come from `{{placeholders}}`, so text and data cannot drift apart, and the run fails if a placeholder has no value.
- **Add a new back-test** (for example 2021–22, the most useful next test):
  1. Copy `backtest/backtest_2002.py` to `backtest/backtest_YYYY.py` and change `ASOF`, `START`, `END`, `WORK` and the output folder.
  2. Copy `backtest/2002/narrative.md` to `backtest/YYYY/narrative.md` and rewrite the text.
  3. Add a Makefile target, run it, and summarise the result in `03-backtests/`.
  4. Or ask an agent: *"Back-test the framework on 2021–22, the same way as 2002."*

## 9. Commit to git

The repository is local-only (no remote), like `../pully`.

1. `cd financialmarkets`
2. `git status` shows what changed.
3. `git add -A && git commit -m "2026-10-15: monthly review; journal entry"`

Committing after each review makes the next `make changes` show exactly what moved since you last looked.

**What is committed:** all documents (`.md` and `.html`), code, catalog and sign-offs, tests, the journal, back-test CSVs and charts, the register CSVs, and `hy_oas.parquet`. That last file is the one series whose history the publisher no longer serves.

**What is not committed** (see `.gitignore`):

- `.env` (secret);
- `.venv/` (about 340 MB; rebuilt by `make setup`);
- raw downloads in `data/raw/` (rebuilt by `make refresh`);
- the other Parquet files, charts, `report/` and logs, all rebuilt by `make refresh` in about 2 minutes.

## 10. Update the framework document

`02-framework/rates-regime-framework-2026.md` is the master. Don't overwrite its reasoning when your view changes; record views in the journal. If you revise the framework itself (for example new bands after more back-tests):

1. Save it as a new version (`rates-regime-framework-2027.md`) and say at the top what changed and why.
2. Update the band values in `dataroom/catalog/series.yaml` to match. This voids sign-offs for the affected series, which is intended.
3. Regenerate the HTML. Ask an agent: *"Convert 02-framework/rates-regime-framework-2027.md to HTML with the plain-html skill."*
4. Add a `CHANGELOG.md` line and commit.