# Paper fleet watchdog + focused strategy lab pattern

Use this when a crypto paper-trading fleet is running many isolated strategy processes and VM/network/session restarts can leave the fleet silently stopped.

## Hourly watchdog pattern

- Keep the watchdog **script-only** (`no_agent=true`) and scheduled at a simple interval such as `every 1h`.
- Make healthy checks silent: non-empty output should mean an actionable alert or a restart report.
- Verify the expected paper fleet by reading each process environment:
  - `CTB_STRATEGY_ID` matches the focused candidate set.
  - `CTB_PAPER_TRADING=true`.
  - `CTB_DRY_RUN=false` for simulated paper fills only.
- Treat unexpected strategies as a restart condition, especially strategies already marked `Blocked`.
- Before restarting, check exchange DNS/API reachability. If the API is unreachable, report the outage and **do not thrash restarts**.
- Controlled restart sequence:
  1. terminate supervisor;
  2. terminate paper bot children;
  3. kill remaining bot children only after a grace period;
  4. relaunch the focused paper fleet;
  5. verify expected count and report only if restart occurred or failed.

## Supervisor launcher pattern

- Keep a small supervisor script under runtime state, e.g. `~/.local/state/<Bot>/run_paper_fleet.sh`.
- The supervisor should launch each isolated `AutoTrader.py` child with per-strategy env and logs, then remain alive and forward `SIGTERM`/`SIGINT` to children.
- Avoid unmanaged `nohup` in normal agent runs; use Hermes background process management for interactive launches. A cron watchdog script may use its own background relaunch only as a self-contained recovery action.

## Focus the fleet before adding more candidates

- Do not keep clearly losing research samplers running just because they exist.
- If a strategy reaches a meaningful sample with negative PnL / weak profit factor / long loss streak, remove it from the active paper fleet and mark it `Blocked` or `research-only`.
- Keep blocked strategies available for offline replay/analysis, but exclude them from watchdog expected sets.

## Strategy lab extension pattern

- Add new strategy archetypes via strict TDD:
  1. failing synthetic entry/exit test;
  2. failing rejection test for false positives;
  3. failing grid-coverage test so the archetype cannot disappear from the lab;
  4. minimal implementation;
  5. full suite.
- Replay winners are **paper candidates only**, never live candidates.
- For liquidation/flash-crash research, a useful side-effect-free lab archetype is `liquidation_reversal`:
  - enter on a fast drop from recent high;
  - require a volume spike versus recent average volume;
  - use tight stop, short max hold, and quick take-profit;
  - rank by profit factor, drawdown, sample size, and coin leakage.
- If replay shows positive total PnL but high drawdown, do not paper-promote yet; add regime/volatility/funding/open-interest filters first.

## Verification checklist

- `pytest -q` green after any lab/launcher change.
- Focused process count matches expected set.
- Blocked strategies are not running.
- Watchdog manual run returns exit 0 and empty output when healthy.
- Cron list shows the watchdog scheduled at the requested interval.