# Paper fleet supervisor and VM-restart watchdog pattern

Use this when running multiple isolated paper-trading strategy processes for a crypto bot, especially after VM reboots, network drops, or chat/session boundaries.

## Problem observed

A launcher that starts many `AutoTrader.py` children and then exits can leave processes unmanaged. After a VM restart or network disruption, the visible state may silently become `0` running bots even though the last report said the fleet was active.

## Durable pattern

1. **Check process reality first**
   - Count live bot processes with `ps`, not just old PID files.
   - Verify recent log mtimes and the last `NETZ AKTIV` / scan line per strategy.
   - Treat stale PID files as advisory only.

2. **Verify market-data connectivity separately**
   - Probe the exchange endpoint with a bounded timeout before blaming strategy code.
   - For Hyperliquid, a minimal `/info` meta request is enough to distinguish strategy failure from DNS/routing/API interruption.

3. **Start under a long-lived supervisor**
   - Launch each strategy with isolated runtime dirs/logs/state files.
   - Keep one parent/supervisor process alive so Hermes/process tooling can inspect and terminate the fleet as a unit.
   - On SIGTERM/SIGINT, terminate children cleanly and then kill stragglers after a short timeout.

4. **Verify startup before reporting success**
   - Confirm expected count of child processes.
   - Confirm each log contains startup mode flags: `Dry-run=False`, `Paper=True`, `TLS verify=True`.
   - Confirm each strategy reports active scanning and its paper state path.

5. **Add a watchdog, not live autonomy**
   - A cron/watchdog may alert or restart paper-only processes after VM/network interruptions.
   - It must not place live trades, change config, tune strategy parameters, or bypass kill-switch/daily-loss gates.
   - For live systems, prefer alert-only recovery unless the user explicitly approves restart semantics.

## Supervisor shell/Python sketch

```bash
cd /path/to/Crypto_Agent
.venv/bin/python multi_strategy_launcher.py --runtime-dir ~/.local/state/CryptoTradingBot --max-parallel 15 --strategies ...
.venv/bin/python - <<'PY'
import os, signal, subprocess
from pathlib import Path
from multi_strategy_launcher import build_strategy_command_plan
from strategy_registry import load_strategy_presets

runtime_dir = Path('~/.local/state/CryptoTradingBot').expanduser()
strategy_ids = ['candidate_a', 'candidate_b']
presets = load_strategy_presets()
plans = build_strategy_command_plan([presets[s] for s in strategy_ids], runtime_dir, max_parallel=len(strategy_ids))
children = []

def stop(signum, frame):
    for p in children:
        if p.poll() is None:
            p.terminate()
    for p in children:
        try:
            p.wait(timeout=10)
        except subprocess.TimeoutExpired:
            p.kill()
    raise SystemExit(128 + signum)

signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
for plan in plans:
    plan.log_path.parent.mkdir(parents=True, exist_ok=True)
    log = plan.log_path.open('a', encoding='utf-8')
    proc = subprocess.Popen(plan.command, env={**os.environ, **plan.env}, stdout=log, stderr=subprocess.STDOUT)
    plan.pid_path.write_text(str(proc.pid), encoding='utf-8')
    children.append(proc)
while children:
    for p in list(children):
        if p.poll() is not None:
            children.remove(p)
    if children:
        signal.pause()
PY
```

## Status report fields

For user-facing status after restart, include:

- running process count and supervisor PID/session handle;
- mode confirmation: paper vs dry-run vs live;
- recent log freshness and scanner activity;
- closed trades, PnL, open positions;
- strategy scorecard highlights;
- connectivity warnings separately from strategy performance.