# Paper fleet watchdog and focused strategy supervision

Use this pattern when a multi-strategy crypto paper fleet is already running and the user wants resilient hourly supervision after VM/network restarts.

## When to apply

- Paper bots are launched as multiple isolated `AutoTrader.py` processes with per-strategy runtime dirs under `~/.local/state/<Bot>/experiments/<strategy_id>/`.
- Some strategy families have enough paper evidence to be `Blocked` and should no longer consume live paper API/scan capacity.
- The user wants a watchdog that restarts paper processes if missing, but does not spam when healthy.

## Focus the live-paper fleet before adding a watchdog

1. Compute a read-only scorecard from `trade_journal.jsonl` and `paper_state.json`:
   - closed trades, realized PnL, win rate, profit factor
   - max drawdown, max consecutive losses
   - coin leakage / dominant coin share
   - open paper positions
2. Keep only meaningful candidates/research samplers in the running paper fleet.
3. Remove clearly negative strategies from the live-paper process list and mark them `Blocked` / replay-only.
4. Do not delete their runtime data; keep journals for analysis.

Example outcome from a paper-tournament session:

- Keep: positive candidate samplers, survival variants, low-sample squeeze/multi-day research candidates.
- Pause from running paper fleet: trend-pullback and Bollinger/RSI mean-reversion variants that reached >20 closed trades with negative PnL and weak profit factor.

## Watchdog behavior

The watchdog should be script-only (`no_agent=true`) and normally silent.

Hourly checks:

- expected strategy IDs are running
- no unexpected/blocked strategy IDs are running
- each process environment has `CTB_PAPER_TRADING=true` and `CTB_DRY_RUN=false`
- supervisor process is present
- exchange API/DNS is reachable

Fail-safe rules:

- If exchange API/DNS is down, report the outage and do **not** restart. This avoids restart thrashing during network incidents.
- If processes are missing, unsafe, unexpected, or supervisor is gone while API is reachable: controlled restart of paper fleet only.
- Never place live orders, alter strategy parameters, or promote candidates from the watchdog.
- Empty stdout means healthy/silent; non-empty stdout should be concise and suitable for Telegram.

## Implementation notes

- Keep the canonical runner outside Git under the runtime state dir, e.g. `~/.local/state/CryptoTradingBot/run_paper_fleet.sh`.
- Keep the cron script under `~/.hermes/scripts/`, e.g. `crypto_paper_fleet_watchdog.sh`, because Hermes cron resolves script paths relative to that directory.
- For manual foreground restarts, avoid shell-level `nohup` wrappers in normal terminal calls. Use Hermes background process management for a supervised run; cron scripts may use backgrounding internally when they are the job itself.
- Verify after changes:
  - exact expected process count
  - no blocked strategies still running
  - all running strategies are paper-only
  - watchdog manual run returns exit code 0 and empty stdout when healthy
  - unit tests still pass

## Cron shape

Create an hourly script-only cron job:

```text
name: CryptoTradingBot Paper Fleet Watchdog
schedule: every 1h / every 60m
script: crypto_paper_fleet_watchdog.sh
no_agent: true
deliver: origin
```

Healthy runs should be silent. Restart/API-failure messages should be short and include process count and reason.