# GPU-backed local STT for meeting-recording web apps

Use this reference when a local Python/Streamlit tool needs high-quality transcription of long meeting recordings while the user's everyday Hermes/Telegram STT should remain fast.

## Pattern

- Keep interactive/gateway voice-message STT on a smaller model such as `medium`.
- Give the domain tool its own STT configuration via project env vars, defaulting to the best local model that fits the GPU.
- For an RTX A5000 16GB class GPU, start with WhisperX / faster-whisper `large-v3` on CUDA with `float16`, `language=de`, and small batch size (`2`) for German/Swiss-German-adjacent meeting audio.
- Do not route this through Ollama; local Whisper STT uses WhisperX/faster-whisper/CTranslate2, not an LLM serving daemon.
- If another GPU resident service such as Chatterbox TTS is running, first try parallel operation. Add VRAM logging before/after model load and after unload.
- Free the transcription model before alignment/diarization so VRAM is available for the next stage and for the resident service.
- Add an automatic CUDA OOM fallback to `int8_float16` before asking the user to stop the other service.

## Suggested env surface

```env
AUTOPROTOCOL_WHISPER_MODEL=large-v3
AUTOPROTOCOL_WHISPER_DEVICE=cuda
AUTOPROTOCOL_WHISPER_COMPUTE_TYPE=float16
AUTOPROTOCOL_WHISPER_FALLBACK_COMPUTE_TYPE=int8_float16
AUTOPROTOCOL_WHISPER_BATCH_SIZE=2
AUTOPROTOCOL_WHISPER_LANGUAGE=de
```

Keep these separate from Hermes' global `stt.*` settings.

## Implementation checklist

1. Inspect the existing transcription path and do not globally change Hermes STT if the user only wants domain-tool recordings improved.
2. Add env-driven model/device/compute/batch/language settings in the domain project.
3. Log VRAM with `torch.cuda.mem_get_info()` at key points.
4. Load `large-v3` with primary compute type; catch `torch.cuda.OutOfMemoryError`, clean CUDA cache, then retry with fallback compute type.
5. After transcription, `del model`, `gc.collect()`, `torch.cuda.empty_cache()`, and where available `torch.cuda.ipc_collect()` before alignment/diarization.
6. Keep documentation and `.env.example` placeholder-only; never commit real tokens, recordings, or output transcripts.
7. Verify with:
   - syntax/import smoke;
   - direct model-load/unload smoke while the resident GPU service is still running;
   - a short clipped real-audio E2E through transcription + alignment + diarization;
   - web-app/service restart and local/Tailnet HTTP 200 if the tool is a Streamlit app.

## Reporting

Report real measurements: initial free VRAM, VRAM after model load, VRAM after unload, whether fallback was used, and a short harmless sample of transcript text if appropriate. Avoid claiming long-session stability from only a short smoke; phrase it as “parallel model-load and short E2E passed”.