## Audit outcome - Audited exact base `039d005696e4ef097427d15b74a26a3c09e86a30`. - Production SQLite was opened with `mode=ro&immutable=1`; only aggregate data was inspected. - Repository remained clean; **no files created or modified**. ## Canonical review truth Canonical human-open predicate should be: ```sql household_batch_id IS NOT NULL AND status = 'needs_review' ``` `status` is authoritative. `requires_review` is a legacy/validation flag, not the lifecycle definition. Consequences: - **228** = genuinely open household decisions: `status='needs_review'`. - **265** = current overview predicate incorrectly adds **37** `covered_by_source` Migros receipt-detail rows: - 228 `needs_review` - 37 `covered_by_source` - **0** is not a database truth. It comes from frontend fallback state after the slow/failing request: - `HouseholdReviewPage.vue:130` initializes all counts/items to zero. - Its catch at lines 229–230 leaves those defaults intact. - Lines 92–94 then render **“Alles geprüft”** whenever items are empty, even when an error also exists. Terminal statuses such as `confirmed` remain closed even where stale `requires_review=1` survives. `covered_by_source` means non-monetary detail covered by another source and must not enter the human inbox. With dedicated-action rows classified before normal proposals, current aggregate truth is: - `total_open`: **228** - `proposal_ready`: **47** - `decision_needed`: **161** - `special_cases`: **20** - 16 credit-card payment/counterpost cases - 4 possible logical duplicates Current classifier output instead gives `51/161/16`, because four possible duplicates are overwritten as `proposal_ready` by their persisted category even though batch confirmation rejects that classification. This is a correctness mismatch. ## Root cause and performance `get_household_review()` is page-size truth, not queue truth: - `review_count = len(items)` at `household_import.py:2346`. - `proposal_count` and `decision_count` are also calculated only from returned items. - Default limit is 100; hard cap is 500. - There is no cursor. - Groups and items independently apply the same limit with unrelated meanings. The N+1 path is: 1. `_human_review_items()` loops candidates (`household_import.py:2210–2273`). 2. It calls `classify_household_row()` without `ClassificationLookupCache`. 3. Each ordinary row can rescan all confirmed categorized history at `household_classification.py:454–485`, plus merchant, alias, category, review-rule, similarity, and recurring lookups. 4. `_category_for_review()` adds another per-item query. Measured read-only: - `limit=1`: **2.21 s**, nine SELECTs, including one full confirmed-history scan. - Reclassifying all 228 rows using one request-local `ClassificationLookupCache`: **2.894 s total**, ten SELECTs—showing that query reuse removes the N+1 explosion. - Existing page query requires a temporary B-tree; no index covers the canonical predicate and ordering. ## Exact implementation surfaces ### Backend - `src/jarvis_finance/api/routers/budget.py` - overview route: lines 241–243 - review route: lines 271–273 - `src/jarvis_finance/services/household_import.py` - overview count: lines 2079–2136 - open-row predicate: lines 2196–2203 - N+1 item construction: lines 2210–2284 - page-local summary: lines 2287–2364 - batch selection also loads up to 500 rows: lines 2385 onward - `src/jarvis_finance/services/household_classification.py` - reusable `ClassificationLookupCache`: lines 351–451 - uncached full-history scan: lines 454–485 - classification precedence: lines 646–884 - `src/jarvis_finance/storage/migrations.py` - household schema/index changes belong near `_create_household_import_v1_tables`, starting line 2323. ### Frontend - `frontend/src/api/household.ts` - old review contract: lines 63–91 - no-argument review API: line 307 - `frontend/src/pages/HouseholdReviewPage.vue` - false zero defaults: line 130 - client-only filtering of one page: lines 152–160 - error/empty conflation: lines 92–94 and 213–230 - `frontend/src/pages/HouseholdOverviewPage.vue` - overview count/card/navigation: lines 38–40 and 60–66 - `frontend/src/components/household/HouseholdTabs.vue` - review navigation; currently no canonical badge provider - `frontend/src/pages/HouseholdUx.test.ts` - main frontend regression location - `tests/unit/test_household_import_v1_golden.py` - existing backend household contract tests ## Minimal canonical contract Use one canonical endpoint for overview, badge, and inbox: ```http GET /api/budget/household/review?state=&source=&limit=50&cursor= ``` ```json { "total_open": 228, "proposal_ready": 47, "decision_needed": 161, "special_cases": 20, "filtered_total": 228, "items": [], "cursor": null, "as_of": "UTC ISO-8601 timestamp", "data_version": "opaque revision" } ``` Required invariants: - `proposal_ready + decision_needed + special_cases == total_open` - `total_open` ignores filters. - `filtered_total` applies all active server-side filters but not pagination. - Overview, navigation badge, and inbox header consume the same `total_open`. - Counts are never inferred from `items.length`. - `cursor=null` means no next page. - `as_of` and all counts/page rows are read from one SQLite read snapshot. ### Cursor design Stable ordering: ```sql ORDER BY transaction_date DESC, transaction_candidate_id DESC ``` Opaque cursor binds: - `data_version` - normalized filter hash - last `transaction_date` - last `transaction_candidate_id` Keyset continuation: ```sql AND ( transaction_date < :last_date OR ( transaction_date = :last_date AND transaction_candidate_id < :last_id ) ) ORDER BY transaction_date DESC, transaction_candidate_id DESC LIMIT :page_size_plus_one ``` Reject malformed/filter-mismatched cursors with 422 and stale `data_version` with 409. Do not silently mix snapshots. ### Query/read-model design For exact global state counts without scanning/classifying up to 500 items: 1. Materialize canonical derived `review_state` (`proposal_ready`, `decision_needed`, `special_case`) with `classification_version` when classification is produced or invalidated. 2. Obtain the summary with one conditional aggregate query. 3. Obtain `filtered_total` with one `COUNT(*)` over the same canonical predicate plus filters. 4. Obtain `limit+1` items with one joined keyset query. 5. Bulk-load categories/rules/history once if fallback reclassification is needed—never once per row. Add a partial pagination index equivalent to: ```sql CREATE INDEX idx_household_review_open_page ON budget_transaction_candidates( transaction_date DESC, transaction_candidate_id DESC ) WHERE household_batch_id IS NOT NULL AND status = 'needs_review'; ``` `data_version` must change when candidates or classification inputs change: active categories, merchant/default rules, aliases, review rules, recurring rules, or confirmed categorized history. An opaque digest/revision is preferable to exposing raw timestamps or IDs. ## Risks ### P0 - **False queue truth:** overview reports 265 while canonical inbox is 228. - **False empty state:** timeout/API failure can display zero and “Alles geprüft.” - **Endpoint availability:** limit 1 already takes 2.21 seconds; default 100 is operationally unsafe. - **Unsafe action mismatch:** four duplicate cases appear proposal-ready/selectable, but the backend rejects ordinary category confirmation. ### P1 - Counts are page-local and capped at 500, not global. - Client filters operate only on the returned page, so filtered totals and source choices are incomplete. - No snapshot-bound cursor permits duplicate/missing rows after concurrent decisions. - Monthly overview displays a global all-year review count without saying so. - “Alle Buchungen” currently means all returned open review items, not all household bookings. - Batch selection loads up to 500 open rows to resolve a few submitted tokens. ### P2 - Group limit and item limit have unrelated semantics. - Group `can_confirm` is effectively true because `ignore` is always supported. - Categories are retransmitted with every review response. - Current status/order query uses a temporary B-tree due to missing composite partial index. ## Concrete regression tests ### Backend Add focused tests in `tests/unit/test_household_import_v1_golden.py` or a Sprint 17A test module: 1. **Canonical status matrix:** `needs_review` counts open; `covered_by_source`, `confirmed`, ignored and superseded do not, regardless of stale `requires_review`. 2. **Known aggregate:** synthetic 228/37 fixture yields `total_open=228`, never 265. 3. **Surface equality:** overview, review summary, and badge provider return the same `total_open`. 4. **Bucket invariant:** proposal + decision + special equals total; duplicates and card-payment classes are special, not batch-confirmable proposals. 5. **Global versus page counts:** limit 1 still reports full summary and `filtered_total`. 6. **Cursor traversal:** page through equal-date rows with no duplicate or omission. 7. **Cursor binding:** changed filters → 422; changed data version → 409. 8. **Filter semantics:** `total_open` stays fixed while `filtered_total` changes. 9. **Query budget:** trace callback proves SELECT count remains constant as fixture grows from 1 to 200 rows; no confirmed-history query per item. 10. **Query plan:** pagination uses the partial index and no temporary ORDER BY B-tree. 11. **Read-only guarantee:** summary/page requests leave `conn.total_changes` and DB dump unchanged. 12. **Privacy:** cursor and payload expose no candidate IDs, file names, account references, or raw fingerprints. ### Frontend Extend `frontend/src/pages/HouseholdUx.test.ts`: 1. Loading shows neither zero nor “Alles geprüft.” 2. Rejected/500 request shows error only; no zero-summary or success-empty state. 3. Successful `total_open=0` renders true empty. 4. `total_open>0 && filtered_total=0` renders “no results for filters,” not “Alles geprüft.” 5. Header, overview card, and badge all display the same `total_open`. 6. Pagination appends items while summary remains unchanged. 7. Stale-cursor 409 prompts reload and does not preserve a misleading partial list. 8. Special cases cannot invoke ordinary category batch confirmation. 9. Out-of-order filter/page responses cannot overwrite newer state. ## Issues encountered - The `sqlite3` CLI and repository `.venv` were unavailable; read-only aggregate probes used Python’s standard `sqlite3` module instead. - No source or production data was changed.