# Existing executor reconciliation and production queue bridge

Use this reference when a dashboard/control-plane is being added on top of workflows that already exist outside the app: uploads, website generation, publishing, CI, batch jobs, or agent-run production steps.

## Core principle

Do **not** reimplement or re-trigger a side-effecting workflow just because the dashboard has a button. First reconcile what already happened, then model the external executor as a controlled backend.

A good dashboard for an existing automation stack should:

1. discover existing artifacts/audits/state,
2. import them into first-class dashboard records,
3. expose approval/request objects for future work,
4. let the existing executor poll or consume an outbox,
5. import completion/audit results back into the dashboard,
6. keep destructive/publishing actions gated by explicit approval.

## Reconcile before acting

Before any upload/publish/generate action, scan all known sources of truth:

- external audit logs,
- generated candidate/spec JSON,
- review/package manifests,
- website/content repo files,
- dashboard DB rows,
- provider IDs and URLs already present in local artifacts.

For each candidate/package build a compact reconciliation row:

- `candidate_id`
- `package_id`
- local asset path and checksum if present
- local post draft metadata
- real provider ID and canonical URL
- audit path and audit status
- website candidate JSON path
- website markdown/content path
- website slug
- link visibility / `website_link_allowed`
- website status: `pushed`, `generated`, `pending`, `waiting_for_public_video`, or `missing`

Never upload again when a real provider ID already exists. If both dry-run and real IDs exist, the real ID wins while the dry-run remains marked as test/audit history.

## Model external executors explicitly

Add an executor model/table instead of hardcoding one upload path:

```text
UploadExecutor
  id
  name
  type: dashboard_oauth | existing_jarvis_uploader | dry_run | imported_existing | ...
  status
  description
  config_json
  created_at
  updated_at
```

Default real side-effecting work to the existing proven executor when the user says one already exists. Keep new direct provider integrations optional/configured-required until credentials and policy are ready.

## Approval/request bridge pattern

For a side-effecting external executor, create a request object and outbox/API rather than calling the provider directly:

```text
UploadRequest / PublishRequest / JobRequest
  id
  asset_id / draft_id
  executor_type
  status: draft | approval_requested | approved | sent_to_executor | completed | failed
  approval_text
  approval_command
  expected_candidate_id
  expected_checksum_or_sha
  expected_privacy_or_mode
  metadata_snapshot_json
  outbox_path
  audit_id / external_id
  timestamps
  error
```

Outbox JSON must be token-free: include IDs, hashes, desired mode/privacy, metadata snapshots, and approval command only. No OAuth tokens, API keys, cookies, refresh tokens, or private credentials.

Useful internal agent API shape:

```text
GET  /api/agent/<kind>-requests/pending
POST /api/agent/<kind>-requests/{id}/mark-sent
POST /api/agent/<kind>-requests/{id}/attach-audit
POST /api/agent/<kind>-requests/{id}/complete
POST /api/agent/<kind>-requests/{id}/fail
```

## Import existing audit logs

Provide a script and API endpoint to import already-created audit logs:

```bash
python scripts/import_<provider>_audits.py --audit-dir /path/to/audits --api-base http://127.0.0.1:PORT --dry-run
python scripts/import_<provider>_audits.py --audit-dir /path/to/audits --api-base http://127.0.0.1:PORT --import
```

The importer should:

- parse candidate/version/provider IDs from audit JSON,
- link by exact candidate ID, external provider ID, or checksum,
- reuse an existing asset when checksum matches to avoid unique-key duplicates,
- update asset/draft/external-post records with real IDs,
- create an audit row with `external_existing_upload` / `imported_existing`,
- complete any matching request,
- emit activity events like `existing_upload_detected` and `state_reconciled`.

## Cross-repo pending/duplicate safety

When the dashboard controls another repo, treat uncommitted files in that repo as state, not clutter.

If a pending guide/content file looks like a duplicate of an existing guide:

- do not delete it,
- do not push it,
- surface `Potential duplicate` in UI,
- show existing slug/path and pending slug/path,
- recommend `merge / discard pending / keep as new`,
- require explicit confirmation before any file mutation or push.

## Production queue bridge

For “produce the next thing according to dashboard order,” add a queue item model and agent-facing endpoint:

```text
ProductionQueueItem
  id
  family_id
  idea_id nullable
  content_script_id nullable
  title
  brief
  script_constraints
  visual_constraints
  voice_constraints
  reason
  reason_lines / recommendation metadata JSON
  duplicate_risk
  duplicate_notes
  similar_content
  performance_reference
  rotation_hint
  priority
  position
  status: suggested | approved_next | locked_next | concept_proposed | concept_approved | producing | package_ready | in_review | needs_changes | upload_requested | uploaded | done | held | archived | archived_pre_real_run | failed
  source: recommendation | manual | chat_command
  expected_package_id
  expected_package_manifest
  actual_package_id nullable
  video_asset_id nullable
  attached_package_path
  timestamps
```

Ordering rule:

```text
locked_next wins;
else concept_proposed/concept_approved/producing continues to count as the current next item until rejected, unlocked, or completed;
else highest approved_next by position/priority;
else report "No locked/approved production item".
```

Agent API:

```text
GET  /api/agent/production/next
POST /api/agent/production/{id}/concept-proposed
POST /api/agent/production/{id}/approve-generation
POST /api/agent/production/{id}/reject-concept
POST /api/agent/production/{id}/mark-started
POST /api/agent/production/{id}/attach-package
POST /api/agent/production/{id}/mark-ready-for-review
POST /api/agent/production/{id}/mark-failed
```

Important confirmation gate: `GET /api/agent/production/next` should return a `confirm_command` such as `APPROVE_PRODUCTION <queue_item_id>`. The agent must present a detailed concept in chat first and wait for explicit approval before calling `mark-started`; direct production from a locked item should be rejected.

`GET /api/agent/production/next` should return the locked/approved item plus a deterministic expected package ID/path such as `package_<short_id>` and `storage/incoming/package_<short_id>/manifest.json`, plus production rules:

```json
{
  "no_final_wan_render_without_approval": true,
  "test_preview_allowed": true,
  "renderer_owned_text": true,
  "no_unreviewed_upload": true
}
```

## Mechanical end-to-end test package pattern

Before running a real expensive production render, build a low-cost test package generator that exercises the real control-plane edges:

```bash
python scripts/create_test_package_from_queue.py --api-base http://127.0.0.1:PORT --import
```

The generator should:

1. call `GET /api/agent/production/next`,
2. call `concept-proposed` and verify a human-facing `APPROVE_PRODUCTION <id>` command exists,
3. require approval or, for explicitly marked test fixtures only, call `approve-generation` before production,
4. call `mark-started` only after the queue item is `concept_approved`,
5. write a short ffmpeg color/test preview video,
6. write `storage/incoming/package_<short_id>/manifest.json`,
7. call the real import scan,
8. attach the imported `VideoAsset`,
9. call `mark-ready-for-review`,
10. never create an upload automatically.

Critical pitfall: import scans may process several old incoming packages. Do **not** attach the last returned `video_id`. Match the import log by `folder == package_id` and attach that exact `video_id`; otherwise the queue item can be linked to the wrong old package.

Prepared package manifests should include `queue_item_id`, `family_id`, optional `idea_id` / `content_script_id`, `working_title`, suggested platform metadata, script/hook/production notes, disclosure, and `status: ready_for_review`. For test previews, include an explicit `test_package: true` marker and user-facing metadata that says it is not final.

When testing upload-request transitions without a real provider call, allow `approved -> sent_to_executor -> completed` with no external ID. If a dummy provider ID is used during a probe, clean it from `VideoAsset`, `PostDraft`, and request state before reporting final live status; never leave fabricated external IDs in persistent state.

## Recommendation queue and duplicate-safety pattern

A production queue must be a steerable list, not a single hidden "next" value. Seed at least 10 actionable suggestions when the queue is empty or stale. Each recommendation should expose enough context for a creator to choose deliberately:

- title / working title,
- content family,
- hook idea,
- short brief,
- reason lines,
- expected style/format,
- status,
- priority and position,
- duplicate risk,
- similar content matches,
- performance reference,
- rotation hint.

Duplicate detection should combine normalized title/hook/script title, existing package IDs, website/candidate slugs, uploaded platform titles, and keyword overlap. Do not silently suppress borderline duplicates; surface them as `Similar content exists` with a recommendation such as `use only if hook angle is significantly different`.

Rotation should deliberately avoid several consecutive items from the same family. Balance: best family, second-best family, manual priority, under-used good family, and follow-up on top performers.

When generating briefs from mixed library/analytics sources, prefer tolerant field access (`format` OR `style` OR a safe default) so a single partial library idea cannot crash the seeding endpoint.

## Safe active-workspace reset

For a "clean dashboard before the first real run" request, never hard-delete historical assets. Add a dev/admin reset endpoint or script that archives active work-state only while keeping analytics/content/audits/logs:

```text
POST /api/dev/reset-active-workspace
python scripts/reset_active_workspace.py --archive-active-review --keep-analytics --keep-content-library
```

Reset should hide or archive:

- active `ready_for_review`, `in_review`, `needs_metadata` videos/drafts,
- test packages and sprint preview material,
- test UploadRequests / active UploadRequests,
- finished/test queue items when requested.

Keep:

- analytics,
- content library and families,
- scripts,
- upload audits,
- logs,
- website pending files.

Frontend active inbox should default to active review statuses only. After reset, show an explicit empty state such as `No videos waiting for review. Production plan ready. Next production not started yet.` Calendar views should show only actionable scheduled/failed uploads, not archived or succeeded history, otherwise old test items keep confusing the active workflow.

## Verification checklist

- Dry-run reconciliation lists existing external IDs without DB mutation.
- Import creates/updates audit/external-post/context rows.
- Real external IDs override dry-run IDs.
- Request/outbox JSON contains no secrets.
- Agent pending endpoint returns approved requests.
- Production `next` endpoint returns locked item first.
- Attach-package endpoint marks queue item `package_ready`.
- Duplicate detection catches pending-vs-existing cross-repo content.
- No provider upload occurs during reconciliation tests.
- No website/static repo push occurs without explicit approval.
- Backend tests, linter, frontend build, and live API smoke checks are run before reporting success.