# Subtitle and TTS Strategy

Date: 2026-05-24

## Decision summary

For phase 1, subtitles should be deterministic renderer-owned captions in a high-retention Hormozi-inspired style, not burned into AI-generated images or video clips.

Recommended TTS path for now:

```text
EdgeTTS for first English final voice tests
→ optional NVIDIA Magpie A/B test later
→ Whisper only for alignment/transcription, not as TTS
```

Reasoning:

- EdgeTTS is fast, free, no GPU, no API key, and leaves the GPU free for ComfyUI/Wan/Qwen video/image work.
- NVIDIA Magpie TTS may be worth testing for voice quality, but it is an external API dependency and not yet proven better for our exact Shorts style.
- Whisper is speech-to-text, not text-to-speech. It can align/generated-audio transcripts later, but it does not solve voice generation.
- Local neural TTS on GPU is not attractive while the same GPU is needed for video generation.

## Subtitle style

Use a `Hormozi-inspired`, not `Hormozi-cloned`, subtitle system:

- 2-5 words per subtitle chunk;
- active word highlight;
- strong contrast;
- center-lower safe-zone placement;
- occasional emphasis words in accent color;
- hard line breaks controlled by renderer;
- no baked text inside AI-generated images/clips;
- no copyrighted creator-template dependency;
- captions remain editable until final render.

Default visual rules:

```text
Font: bold sans-serif
Placement: lower-middle, above platform UI safe zone
Primary color: white
Stroke/shadow: black/dark outline
Accent: TrueTrace green/blue
Max lines: 2
Max words per chunk: 5
Chunk duration: roughly 0.7-1.6s
```

## Implemented planning modules

Current code-level planning boundary:

- `autoshorts.audio.subtitles.build_hormozi_subtitle_track()` creates continuous 2-5-word subtitle cues with active-word metadata, lower-middle safe-zone style, no side effects, and no external calls.
- `autoshorts.audio.subtitles.validate_subtitle_track()` rejects non-renderer-owned cues, overcrowded captions, bad timing, and missing active words.
- `autoshorts.audio.tts_plan.build_default_truetrace_tts_plan()` selects `edge_tts` with `en-GB-RyanNeural` for phase 1.
- `autoshorts.audio.tts_plan.build_tts_ab_test_plan()` keeps NVIDIA Magpie as a hosted A/B candidate and Whisper as `alignment_qa`, not voice generation.
- `autoshorts.demo.truetrace_preview_packages.build_top_truetrace_preview_packages()` attaches the subtitle track and EdgeTTS baseline plan to each top-3 preview artifact before any real audio generation.
- `autoshorts.audio.edge_tts_adapter.synthesize_edge_tts()` is the explicit EdgeTTS side-effect boundary. It calls `edge-tts`, writes an MP3, and reports `side_effects=("call_edge_tts", "write_audio")`.
- `autoshorts.audio.final_preview.build_audio_caption_preview_plan()` creates an ASS subtitle file plan from the cue track without writing files.
- `autoshorts.audio.final_preview.render_audio_caption_preview()` writes the ASS subtitle file and calls FFmpeg to mux EdgeTTS audio plus burn captions into a review MP4.
- `autoshorts.audio.voiceover.build_truetrace_final_voiceover()` provides beginner-friendly English final voiceover scripts for the top TrueTrace concepts. The first pilot uses the concrete `weird email` framing rather than abstract terms like `exception path`.
- `autoshorts.audio.voiceover.build_accessible_script_draft_from_voiceover()` bridges the final voiceover back into render planning, so top card text, scene cards, subtitles, and spoken audio share one source of truth.
- `autoshorts.audio.probe.probe_media_duration_seconds()` reads the generated MP3 duration via FFprobe so subtitle cues can be rebuilt against the actual voiceover length, not merely the estimated target.
- `render_audio_caption_preview(duration_mode="trim_to_audio")` may use `-shortest`, but only when explicitly requested. The default remains `preserve_video` to avoid silent truncation.

## Caption pipeline

Preferred deterministic pipeline:

```text
Script voiceover text
→ caption chunker
→ timed subtitle cues
→ renderer-owned caption layer
→ FFmpeg burn-in for final MP4
```

Later, after real TTS is added:

```text
Final script
→ TTS audio
→ optional Whisper alignment/transcript QA
→ corrected subtitle cues
→ final renderer burn-in
```

Important: Whisper is useful for checking/aligning what the TTS actually said, but it should not own the script. The source of truth remains the approved script.

## TTS options

### EdgeTTS

Use for phase 1 baseline.

Pros:

- no GPU;
- free/no key;
- quick iteration;
- predictable enough;
- available voices;
- keeps video GPU free.

Cons:

- common synthetic sound;
- less distinctive brand voice;
- commercial/licensing terms should be reviewed before scaling monetized production.

Suggested English voices to A/B later:

- `en-US-GuyNeural` — clear male baseline;
- `en-US-ChristopherNeural` — more mature male;
- `en-US-AriaNeural` — polished female baseline;
- `en-GB-RyanNeural` — British male, potentially fitting precise/dry tone.

### NVIDIA Magpie multilingual TTS

Use as an A/B candidate, not default yet.

Pros:

- likely higher modern neural quality than commodity voices;
- multilingual support;
- cloud-side, so it should not consume local video GPU.

Cons:

- external API dependency;
- cost/rate limits/terms unknown until tested;
- voice/style fit unknown;
- not integrated into the current deterministic pipeline;
- may create operational coupling to build.nvidia.com.

Recommendation: run a short 15-20 second voice A/B sample against EdgeTTS before making it default.

### Local TTS

Not recommended as default for phase 1 if it needs GPU.

Use only if:

- it can run CPU-fast enough; or
- it runs on a separate machine; or
- video generation is idle; or
- it gives a clearly superior branded voice.

The local GPU should be reserved for ComfyUI/Wan/Qwen video/image jobs when those are active.

### Whisper

Whisper is not a TTS model.

Use later for:

- checking TTS output text;
- subtitle timing alignment;
- transcript QA;
- detecting missing/garbled words.

Avoid using Whisper as a required step for every preview until the audio pipeline exists.

## Phase plan

### Phase 1A: Current preview layer

- German approval preview cards only.
- No final TTS required.
- Captions are card/caption beats for concept review.

### Phase 1B: First final-video audio test

- Use EdgeTTS English voice baseline.
- Generate deterministic subtitle cues from the approved script.
- Burn captions in renderer/FFmpeg.
- Send preview with audio + captions for approval.

### Phase 1C: Voice A/B

Create the same 15-20 second script using:

1. EdgeTTS baseline;
2. NVIDIA Magpie if API access is configured;
3. optional local CPU/offline TTS if available.

Compare:

- clarity;
- retention energy;
- pronunciation of AI/workflow terms;
- audio artifacts;
- latency/cost;
- operational reliability.

### Phase 2: Alignment QA

- Add Whisper transcription/alignment after TTS.
- Flag mismatches between approved script and spoken audio.
- Adjust subtitle cues if the TTS timing differs.

## Current recommendation

Start with:

```text
Subtitles: deterministic Hormozi-inspired renderer-owned captions
TTS: EdgeTTS English baseline
Magpie: A/B test later
Whisper: alignment/QA later, not TTS
GPU: reserve for video/image generation
```

This is the lowest-friction path and avoids blocking the video pipeline on voice experiments. Voice quality matters, but first we need to prove that the concepts and visuals are worth voicing. A beautiful voice reading weak content is merely expensive politeness.