# Controlled market-data recovery and activation

Use this workflow when a disabled or stale daily valuation source must be diagnosed, repaired, and reactivated with narrowly authorized production writes.

## 1. Freeze scope and diagnose before changing code

1. Verify the exact base/deployment SHA, schema, service/timer state, and global/source gates.
2. Trace the row origin: identify which command/job wrote the last price, its last successful run, later attempts, and whether the stop is `paused`, provider unreachable, mapping missing, price missing, FX missing, writer blocked, or another technical error.
3. Inspect systemd units, journal history, runtime configuration, DNS/network/provider access, limits/auth, mappings, and writer constraints without printing secrets.
4. Inventory the relevant tables for prices, FX, valuation snapshots, run state, audit state, holdings, transactions, imports, and trades.

## 2. Build an exact, write-free live dry-run

- Use the existing approved provider; fail closed for any unapproved provider identity.
- Resolve every held asset through a canonical provider ID, never ticker alone.
- Preserve distinct wallets and zero holdings; do not infer trades or cashflows.
- Fetch prices and required FX with bounded timeout/retries, capture provider timestamps, calculate per-asset CHF value, total, oldest quote, age, and the exact planned table writes.
- Distinguish `complete`, `partial`, and `blocked`; any missing/stale/ambiguous quote suppresses the aggregate total.
- Store/report the actual provider timestamp; never relabel an old quote as current.
- Make freshness policy explicit and appropriate to the valuation cadence; report the oldest accepted age.

### SQLite WAL correctness

`mode=ro&immutable=1` can ignore committed rows still present only in a WAL file. For a live dry-run, use a genuinely read-only WAL-aware connection (`mode=ro` without `immutable=1`) or first create a consistent read-only snapshot and inspect that snapshot immutably. Verify the production DB hash or protected digest is unchanged after the dry-run.

## 3. Harden the existing daily source

- Keep provider calls out of dashboard render paths.
- Add a source-specific gate; a global gate alone is insufficient.
- Hold an application lock across fetch/revalidation/write; use `BEGIN IMMEDIATE` for the writer.
- Fetch before the DB transaction, then re-read and compare the inventory inside the write transaction.
- Write prices, valuation, run state, and audit atomically.
- Ensure at most one canonical valuation per source/day; a second identical invocation must return idempotently without calling the provider.
- Persist unsuccessful attempt status without writing partial prices or a false valuation.
- Bind status provenance to the exact run (for example exact `fetched_at` plus approved provider), not a broad timestamp window.
- Derive valued/missing counters from the same latest run. Keep `last successful run` separate from `last attempt`.
- Make source-gate activation itself transactionally idempotent; recheck under `BEGIN IMMEDIATE` to prevent duplicate audit rows.

## 4. Keep current valuation separate from performance history

A current marked-to-market valuation does not establish TTWROR/XIRR readiness. Never back-apply current holdings to earlier dates. Read-only assess wallet coverage, activity history, internal transfer pairing, boundary cashflows, and the earliest honest confirmed snapshot. Prepare historical backfill only as a preview until explicitly confirmed.

## 5. Verify on a production-shaped copy

Run the one-shot twice against a consistent copy:

- first run: exact expected price/FX/valuation/run/audit writes;
- second run: same run identity, zero additional writes, no provider call;
- integrity and foreign-key checks pass;
- holdings, transactions, cashflows, imports, and trade tables/digests stay unchanged.

Run focused and full backend/frontend suites, pinned Ruff scope, compileall, typecheck, production build, migration/schema gates, Git safety, and responsive browser UAT. Browser UAT should assert no provider requests during render, no raw JSON, no overflow, and explicit paused/partial/failed states.

## 6. Controlled production sequence

1. Finish independent re-reviews and close every P0/P1 before commit.
2. Commit, push, PR, CI, merge, and deploy the exact merge SHA.
3. Prove deployment/read-only sentinel equality before authorized writes, then exercise the deployed status API/UI. Missing provider/FX lineage discovered here is a normal follow-up release candidate: keep the timer disabled, fix through a reviewed PR/CI/merge, redeploy the new exact SHA, and repeat UAT—never patch the deployment worktree directly.
4. Create a WAL-coherent SQLite online backup with owner-only file and directory modes; verify SHA-256, integrity/FK checks, and an independent restore copy. A raw copy of only the main database file is not sufficient while WAL may contain committed state.
5. Capture baseline row counts plus deterministic logical digests for every protected table, and separate expected-mutation tables explicitly.
6. Run exactly one manual production source one-shot.
7. Reconcile every stored price and aggregate valuation to holdings.
8. Run the identical command again and prove idempotency.
9. Activate only the source gate; repeat the same confirmation once to prove no duplicate audit row.
10. Enable the global gate/timer only if all unapproved sources remain isolated. First execute once through the actual systemd service path and prove it is an idempotent no-op for the already-complete day.
11. Treat `Persistent=true` as a possible immediate execution on `enable --now`, not merely a future schedule. Verify service result/journal, exact table deltas, enabled/active state, and both nominal and systemd-adjusted next-run time after activation.
12. Keep operator-status truth aligned with runtime gate truth. If the status API reads process environment rather than systemd timer state, apply the reviewed gate to the read-only API process too and restart it before UAT; otherwise the UI can falsely remain `paused` while the timer is active.
13. Recompute all-table mutation evidence. Activation must be reported as `expected_mutation=true` with exact allowed row deltas, unchanged protected-table digests, schema, integrity, and foreign-key evidence.
14. If any gate fails, leave the timer disabled and display the last successful value with its real timestamp.

Never perform trades, holdings/transaction rewrites, cashflow confirms, unrelated source activation, or historical backfill unless separately authorized.