## Outcome - Branch is clean at `9786857c814a29015987681e2416acb23275f9e5`, exactly equal to the supplied base; Sprint 16.2 has no tracked implementation yet. - Reviewed tracked source/tests only. No runtime/private evidence or production data was accessed. - Existing synthetic classification suite passes: **49 passed**. - No files were created or modified; no production preview, confirm, or import was executed. ## Current implementation and gaps ### `src/jarvis_finance/services/household_classification.py` Relevant functions: - `normalize_merchant()` — line 157 - `_account_role()` — line 189 - `_confirmed_history()` — line 267 - `_exact_rule()` — line 298 - `_source_rule()` — line 332 - `_similarity_rule()` — line 354 - `classify_household_row()` — line 377 Current version is `household_classification_v2.1`, not required `v2.2`. Current effective order is: 1. transfer/card-payment/refund semantics 2. explicit unmatched-transfer decision 3. bank interest 4. exact merchant/alias rule 5. confirmed history 6. public merchant family 7. source rule 8. similarity 9. unresolved Gaps: - No recurring-payment classification despite `budget_recurring_payments` being included in the preview baseline. - `_account_role()` is unused. History is restricted to exact `source_type`, not normalized account role as the documented contract requires. - Exact merchant, alias, source-rule, and public-family conflicts are resolved by deterministic first-row ordering rather than fail-closed conflict detection. - Generic positive bank rows remain `income_candidate`; bank refunds/reversals are not explicitly distinguished from income. - Any positive VISA row not recognized as a card payment becomes a refund. That is conservative against false income but lacks explicit refund evidence/provenance. - Merchant normalization can collapse distinct counterparties: - splitting at `ursprunglicher auftraggeber` can erase the actual originator; - blanket removal of six-plus-digit tokens can remove stable merchant identity; - an empty normalized merchant can group unrelated rows. - Existing tracked source already contains personal category labels such as `Auto Melanie`, and other budget modules contain additional personal labels. New v2.2 rules must not add or duplicate these; this existing leak needs separate remediation. ### `src/jarvis_finance/services/household_import.py` Relevant functions: - `_baseline()` — line 309 - `_input_fingerprint()` — line 355 - `_merchant_cluster_key()` — line 416 - `_apply_cluster_decisions()` — line 437 - `_normal_row()` — line 556 - `_preview_household_import_internal()` — line 866 - `preview_household_import()` — line 1225 - `confirm_household_import()` — line 1494 Current cluster behavior: - Only unresolved CHF mapped rows become clusters. - Cluster identity is merchant family/normalized merchant plus transaction semantics. - Members retain request order; cluster token sorts row tokens, but displayed members/examples are not independently canonicalized. - A request supplies an arbitrary active `category_id` and arbitrary subset of excluded member tokens. - Clusters can combine different source/account roles because role is absent from `_merchant_cluster_key()`. - Ambiguous marketplace families are grouped but disallow category decisions entirely. - There is no server-derived cluster proposal/evidence contract. Current strict readiness gates are at lines 1117–1191: - coverage ≥ 95% - individual review count ≤ 50 - individual review ratio ≤ 3% - merchant review clusters ≤ 25 - mapping/sign, card-payment, Migros-double-count and critical mapping checks Confirm independently enforces both technical confirmability and business readiness at lines 1517–1520. These definitions should remain byte-for-byte unchanged for Sprint 16.2. ## Safest phased implementation map ### Phase 1 — deterministic v2.2 classifier Modify only `household_classification.py`: 1. Set `CLASSIFICATION_VERSION = "household_classification_v2.2"`. 2. Introduce an explicit ordered rule pipeline rather than scattered early returns. Recommended precedence: 1. safe own transfer/card settlement 2. explicit refund/reversal 3. explicit row-scoped user decision 4. exact active merchant/alias rule 5. conflict-free confirmed history for normalized merchant + account role + transaction semantics 6. source-specific income/text/public-family rule 7. active recurring-payment evidence 8. controlled similarity 9. unresolved 3. Make every rule return a common candidate/evidence structure; if the winning priority contains conflicting categories, return `decision_needed`, never lexicographic first-wins. 4. Use `_account_role()` in history and recurring matching. Do not merge card-liability and bank-cash evidence. 5. Add explicit semantic helpers: - `_refund_or_reversal_semantics()` - `_income_rule()` - `_recurring_rule()` - conflict-aware exact/source-rule collectors 6. Recurring proposals should use only active rows, matching: - normalized merchant or linked merchant; - compatible account/account role; - currency; - amount tolerance; - date/frequency tolerance where available; - category type compatible with transaction semantics. `candidate`, `ignored`, `paused`, and `archived` rows must not auto-propose. 7. Harden normalization: - strip known transport envelopes without discarding the following originator; - strip only labeled volatile IDs; - preserve meaningful numbers and semantic words; - treat empty normalized identity as unclusterable; - do not embed private merchant/counterparty lists. No schema migration is needed. ### Phase 2 — finite, server-derived merchant cluster proposals Modify `household_import.py`: 1. Extend `_merchant_cluster_key()` with normalized account role; optionally source scope where a rule is source-specific. 2. Canonically sort cluster members by date, normalized merchant, amount, and opaque row token before producing `row_tokens`, `examples`, and `members`. 3. Derive a cluster proposal only when all eligible members yield one conflict-free category/evidence result. 4. Return opaque proposal data such as: - `proposal_token` - proposed category ID/name - proposal origin - evidence count - eligible member tokens 5. Replace free-form category assignment with a bounded decision enum, for example: - `accept_proposal` - `accept_proposal_except` - `review_individually` The accepted category must come from the emitted server proposal, not an arbitrary request category. 6. Bind decision action, proposal token, sorted exclusions, classification version, and member tokens into `_input_fingerprint()`. 7. Preserve current stale-token, duplicate-token, category-type, non-empty-selection, and member-subset rejection behavior. 8. Do not change readiness calculations or confirm gates. `src/jarvis_finance/api/routers/budget.py` needs no route change because it currently passes the payload through. If the frontend is in scope later, mirror the finite decision/proposal fields in `frontend/src/api/household.ts`; backend work should not depend on that UI change. ### Phase 3 — synthetic tests Extend `tests/unit/test_household_classification_v2_golden.py` or add a focused `tests/unit/test_household_classification_v22.py` with synthetic-only identifiers: - Assert exact version `household_classification_v2.2`. - Table-driven precedence test where exact rule, history, source rule, recurring evidence, similarity, refund, and transfer evidence intentionally conflict. - Two conflicting same-priority exact aliases/source rules must produce `decision_needed`. - Confirmed history: - same normalized merchant + same account role works; - card-liability versus bank-cash does not cross-learn; - conflicting categories fail closed; - one confirmation is only evidence, two consistent confirmations mark learned behavior. - Income/refund tests: - bank interest/payroll income; - positive bank refund text is not salary income; - VISA purchase, refund/reversal, and card settlement remain distinct; - card settlement always has zero budget effect. - Recurring tests: - active monthly match proposes category; - amount/date outside tolerance does not; - candidate/paused/archived rows do not; - wrong account role/currency/category type does not; - conflicting recurring rows require review. - Normalization tests: - equivalent transport envelopes normalize identically; - original-originator content is retained; - meaningful numbers remain; - labeled terminal/reference IDs disappear; - empty normalization never merges unrelated rows. - Cluster tests: - reversed file/row order produces identical cluster/proposal order and tokens; - bank/card account roles form separate clusters; - only emitted proposal categories can be accepted; - stale/duplicate/out-of-cluster exclusions fail; - every member remains visible; - finite actions are fingerprint-bound. - Readiness regression: - assert all existing threshold constants and readiness keys are unchanged; - unresolved rows still block business readiness; - resolving proposals changes only coverage inputs, not gate definitions; - `confirm_household_import()` still rejects a technically valid but business-unready preview. Run: ```bash JARVIS_FINANCE_FINGERPRINT_KEY=synthetic-test-key \ pytest -q tests/unit/test_household_classification_v2_golden.py \ tests/unit/test_household_import_v1_golden.py ``` Then full backend and Git-safety gates before any real preview: ```bash JARVIS_FINANCE_FINGERPRINT_KEY=synthetic-test-key pytest -q tests python -m jarvis_finance.cli.main git-safety-scan . ``` ## Principal risks - **High — privacy:** personal category names already exist in tracked production code/tests. Do not encode additional household names, merchants, counterparties, salary names, or runtime-derived rules. Add a tracked-source forbidden-name/privacy scan using only generic denylist fixtures supplied outside Git. - **High — false category propagation:** current clusters cross source/account roles and permit arbitrary categories over many rows. - **High — normalization collision:** aggressive stripping can collapse unrelated counterparties, causing incorrect history learning and bulk cluster decisions. - **High — silent rule conflicts:** first-row-wins behavior can auto-classify despite multiple matching categories. - **Medium — refund/income ambiguity:** positive bank credits can be income or refund; VISA positives can be refund or settlement. Semantics must precede category inference and remain fail-closed without sufficient evidence. - **Medium — recurring overreach:** merchant-only recurring matching would misclassify one-off purchases; require active status, role, currency, amount/date tolerance, and category-type compatibility. - **Medium — preview privacy:** browser preview intentionally includes merchant/description text. Ensure request bodies and responses are never logged, cluster/audit tokens remain HMAC-derived, and audits store only aggregate origins/counts—not merchant names or raw descriptions. - **Release boundary:** do not alter lines 1117–1191 readiness thresholds/checks or lines 1517–1520 confirm enforcement, and do not run a real import/confirm during this phase.