# High-volume import preview performance and batched-decision workflow

Use this reference when a transactional CSV preview is functionally correct but too slow for a human review/confirm workflow.

## Performance workflow

1. Pin the exact deployed SHA, database snapshot/digest, source files, source hashes, profiles, account mappings, and prior preview fingerprint.
2. Run the baseline read-only. Do not overlap expensive preview jobs against the same production database.
   - Prefer lightweight phase timers, SQL trace counters, and aggregate query counts for the authoritative baseline.
   - Treat deterministic profilers such as `cProfile` as diagnostic-only: on function-call-heavy import loops they can multiply a 25-minute request beyond an hour. Give intrusive probes a separate timeout/attempt identity, never let them block the bounded UAT path, and do not report an interrupted profile as a completed timing result.
3. Add phase timing outside the fingerprinted preview core:
   - parsing per file;
   - normalization;
   - category/merchant/history lookup;
   - classification;
   - transfer pairing;
   - receipt linking;
   - fingerprint/readiness;
   - total API time.
4. Optimize only request-local work first:
   - bulk-load immutable lookup tables once per preview;
   - index confirmed history once instead of scanning it per row;
   - cache category resolution by ID/name/type;
   - cache repeated account/source mapping resolution;
   - reuse the first classification when no user decision changes the row.
5. Preserve the uncached query's exact semantics in every cache:
   - make tie-breaking explicit and identical in cached and uncached paths;
   - preserve priority and row ordering where the old query used `LIMIT 1`;
   - beware that a dictionary comprehension is last-row-wins while SQL `fetchone()` is commonly first-row-wins;
   - add a valid-schema adversarial probe with duplicate active canonical names even when the current production database has none. Prefer a database uniqueness invariant when the domain allows it; otherwise both paths must choose or reject duplicates identically.
6. Never persist request caches across previews unless invalidation is explicit. Confirm must reconstruct from current DB state.
7. Prove performance-only parity against the old classifier before adding calibration rules. Compare the complete stable response after removing non-deterministic timing fields, plus rows, items, clusters, counts, and fingerprint.
8. Then version and measure classification changes separately. A classifier rule change must not be disguised as a performance optimization.
9. Enforce the real-hardware hard UAT limit in a deterministic private runner; also record source-hash and DB before/after evidence and `confirm_called=false`.

## Fingerprint and readiness-threshold rules

Performance measurements are observational and non-deterministic. Add them only after constructing the canonical preview core and its fingerprint. Otherwise identical inputs produce different fingerprints and Confirm reconstruction becomes unstable.

Readiness thresholds are fingerprinted business policy. Define each limit once and reuse it for the displayed threshold, boolean gate, Confirm reconstruction, tests, and decision-package reporting. Add exact boundary probes (`limit` passes, `limit + 1` fails). If a threshold correction intentionally changes the fingerprint, prove performance parity by normalizing only that documented policy field and fingerprint; rows, items, clusters, counts, classifications, and every other stable response field must remain identical.

## Batched portal decisions

- Keep category overrides, cluster choices, row exclusions, and explicit transfer decisions in volatile in-memory draft state.
- Do not issue a server preview on each dropdown or checkbox.
- Provide one visible **Apply decisions** action that sends the complete accumulated draft and triggers one fingerprint-bound reconstruction.
- Preserve all untouched prior decisions when one draft entry changes.
- Track applied and current draft snapshots separately so the UI reports only unapplied changes.
- Clearing an already-applied row category or transfer decision requires an explicit draft tombstone/tri-state. Rendering must distinguish “no local draft” from “draft removal” instead of falling back to the old server value. The Apply serializer omits/removes the decision, dirty comparison includes the tombstone, and a successful reconstruction normalizes the draft to the server-applied request. Test the full add → Apply → clear → visible-empty → Apply-removal path for both categories and high-impact transfer decisions.
- Keep an immutable, deep-cloned copy of the exact request that produced the last successful Preview/Apply. Confirm submits that applied request plus its fingerprints; it must never reconstruct from mutable draft refs, even if the UI currently appears clean.
- On Apply failure, retain the last applied preview, the complete draft, and dirty state; keep Confirm disabled and permit an identical retry. A failed Apply is not permission to erase the review surface.
- A cluster draft may temporarily retain exclusions while its category is empty. Omit that incomplete cluster from the Apply payload; after a successful Apply, normalize visible draft state to the exact serialized request so omitted local-only state cannot appear applied.
- Snapshot the complete Apply request before awaiting the API. Input-generation changes or a newer Apply discard the late response atomically.
- Disable Confirm while unapplied changes exist.
- File, profile, account, or mapping changes must synchronously invalidate both preview and draft, including file-reader race protection.
- Store no raw rows in Local Storage, session storage, console logs, analytics events, or repository fixtures.

## Classification calibration guardrails

- Public deterministic merchant markers may create proposals only when an active canonical category exists and transaction semantics match.
- Never normalize every merchant/token to compact alphanumerics and then apply unrestricted substring containment. This turns innocent supersets into classifications (`hospitality` matching `hospital`, `sparkingshop` matching `parking`). Use boundary-aware exact tokens; when a public brand has a known glued or punctuation variant, add that explicit variant instead of enabling generic compact matching. Add adversarial near-miss probes for every token family that could cause a high-impact automatic proposal.
- Positive amount alone is never income evidence.
- Broad marketplaces, payment processors, cash withdrawals, mixed retailers, and possible own-account transfers remain review.
- Confirmed-history conflicts take precedence over otherwise plausible merchant-family rules. Do not overwrite confirmed counterexamples with a broad deterministic rule.
- Increment the classification version for semantic rule changes and rerun the real four-file preview.
- Build private cluster dossiers sorted by affected-row count. Present genuine user decisions in packages of at most 15–20 units without raw descriptions, row tokens, full account references, or private repository constants.
- Any late classifier-safety fix invalidates the prior calibrated Preview, readiness counts, sorted cluster dossier, and user decision package—even when the performance-only parity harness remains green. Explicitly withdraw the stale package, regenerate all private calibration artifacts from the fixed classifier, and present only the revised aggregate package. Never let a user answer against superseded counts or cluster ordering.

## Verification

Before release, require:

- stable-response and fingerprint parity for the performance-only path;
- full backend and frontend suites, typecheck, build, compile, lint, and git-safety checks;
- independent review of the actual final commit;
- exact-SHA deploy;
- backup and restore test;
- productive read-only interaction/performance UAT;
- DB integrity/FK checks and unchanged source hashes;
- an explicit stop before Confirm until the user gives the exact requested authorization.