# Budget Categories Cleanup & Duplicate Handling

Use this reference for FinanceManager Budget/Cashflow sprints where the user asks to reduce expense categories to a final structure, repair `Kategorien & Tags`, and stop duplicates from polluting normal review.

## Scope guard

- Do not start Fixkosten/Subscriptions until categories and duplicate handling are clean.
- Do not add new import sources, OCR, provider refreshes, or portfolio features during this sprint class.
- Do not report real amounts, raw rows, raw identifiers, secrets, DB paths containing sensitive names, or source files in chat beyond aggregate counts.
- Runtime DB/backups stay outside Git; no Runtime-DB/CSV/XLS/PDF/API keys in repo or frontend.
- No silent mutations: productive category migration requires backup + preview + explicit confirm + audit.

## Workflow

1. Preflight branch, Git status, runtime path, current running backend/frontend processes.
2. Inventory read-only before changes:
   - active expense categories and order
   - categories with confirmed transactions
   - categories with budget plan items
   - categories referenced by merchants or rules
   - child categories
   - empty categories
   - income categories separately
3. Write RED tests first for:
   - final expense category list/order
   - category add/rename/delete/archive audit behavior
   - reorder idempotency and dropdown ordering
   - income categories preserved
   - safe duplicates hidden from normal review
   - possible duplicates visible separately and not normal-confirmable
   - Migros credit-card rows staying `covered_by_migros`
   - manual booking vs CSV duplicate detection
4. Backend implementation:
   - `preview_budget_category_cleanup` must be read-only. Do not call `_ensure_*` or create target categories in preview.
   - `confirm_budget_category_cleanup` may create missing target categories, set `sort_order`, rewire only unambiguous mappings, archive referenced legacy categories, delete only truly empty/unreferenced categories, and audit every write.
   - Ambiguous mappings become review-required and must not be auto-moved.
   - Deletion requires no transactions, plans, merchants, rules, or children. Otherwise archive instead.
5. UI implementation on `Kategorien & Tags`:
   - Add/Rename/Delete/Archive all use Preview → Confirm → Audit or explicit safe action.
   - Show active/reference counts and choose Delete vs Archive based on references.
   - Desktop drag/drop reorder; mobile Up/Down fallback; save order via audited reorder endpoint.
   - After writes, reload category list/tree/dropdowns immediately.
6. Duplicate handling:
   - Safe duplicates: status `auto_ignored_duplicate` or equivalent; exclude from `Alle neuen` and normal confirm review; show only in secondary Admin/Debug tab.
   - Possible duplicates: status `duplicate_candidate`; show in `Mögliche Duplikate`; block normal confirm unless explicit `allow_duplicate` / `Trotzdem bestätigen` action with audit.
   - `covered_by_migros`/`covered_by_source` must not appear as normal reviewable expenses.
   - Preserve backward compatibility for legacy tests if older `duplicate` status is expected in default candidate lists, but ensure current review tabs exclude duplicate statuses from normal confirm flows.
7. Runtime operation:
   - Create verified runtime backup before any productive cleanup confirm.
   - Run cleanup preview and report aggregate counts only: active before, targets present/missing, with transactions, with plans, with merchant/rule refs, auto-mappable, review-required, empty, income count.
   - If any review-required mapping remains, stop before productive migration and report the review blockers.
8. Verification:
   - Python compile, full pytest, frontend tests from `frontend/`, frontend build.
   - Remove/ignore generated `__pycache__`, `.pytest_cache`, build/cache artifacts before Git-safety where applicable.
   - Source/build secret scans should not flag legitimate code identifiers like `api_key_env`; scan for actual token patterns/assignments.
   - `git diff --check`, Git-safety artifact scan, browser/Tailscale route sanity.
   - Restart FastAPI/Vite with browser-visible Tailscale API base; verify `/api/health`, categories page, review page, effective expenses, data explorer, merchant/rules.
   - Commit, push, and verify remote hash.

## Pitfalls learned

- A cleanup preview that creates missing target categories is a silent mutation. Keep preview strictly read-only.
- Do not treat `Sonstiges / Administration` or similar catch-all buckets as auto-mappable; mark review-required unless rules/merchant context makes the target obvious.
- Running `npm test` from repo root fails when package.json lives under `frontend/`; run frontend commands from `frontend/`.
- Pytest can create `__pycache__` files containing synthetic secret-test strings; clean caches before final secret scans or make scans skip generated caches.
- Browser-visible Vite API base matters for iPhone/Tailscale: do not start the frontend with localhost-only API base when the browser runs on another device.