finance-system-architecture

---
name: finance-system-architecture
description: "Design/implement the user's JARVIS Finance System: CHF ledger, portfolio accounting, dashboard, imports, reports, secrecy."
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [finance, portfolio, accounting, dashboard, csv, fx, tax, privacy]
---

# Finance System Architecture

## When to use

Use this skill whenever the user asks about the **Finance System – Architektur & Scripts** topic or work involving the user's personal finance-management system: portfolio imports, transaction accounting, Bank/Broker/CSV processing, dashboards, reports, tax/income views, FX logic, market data jobs, alerts, or later implementation.

Do **not** use it for pure investment-strategy discussion; those belong in the separate finance strategy topic unless the user explicitly asks for technical implementation.

## Core principles

1. **No automated trading.** The system may prepare orders and recommendations, but never executes orders or connects to broker trading APIs for execution.
2. **Accounting first, opinions later.** MVP must prioritize correct transaction ledger, cash ledger, FX handling, income/tax data, and dashboard before complex scores, news, analyst ratings, or LLM analysis.
3. **Rule-based first.** Calculations, scores, alerts, rebalancing and reports must be deterministic and auditable. LLMs may only summarize/explain text or assist extraction with human review.
4. **CHF as base currency.** Store every transaction in original currency and CHF using historical FX at transaction/payment date; current valuations use current FX.
5. **Financial data secrecy.** Never expose real financial values, portfolio files, raw exports, reports with real numbers, DBs, API keys, or credentials in chat, GitHub, logs, or shared docs unless the user explicitly approves scope.
6. **Data ownership.** All data must be exportable: CSV, JSON, Parquet, SQLite/Postgres dump, Markdown/HTML/PDF reports.

## Current v0.3 decisions

See `references/spec-v0.2-decisions.md` for the baseline, `references/spec-v0.3-decisions.md` for the priority update, `references/spec-v0.4-implementation-blueprint.md` for the MVP 1 technical implementation blueprint, `references/spec-v0.4.1-corrections.md` for accepted corrections, `references/spec-v0.5-repo-blueprint.md` for the concrete repo/implementation plan, `references/mvp-safety-gate-implementation.md` for the implementation safety-gate sequence and verification commands, `references/mvp-import-pipeline-tasks-9-13.md` for the synthetic CSV import/FX/transaction pipeline pattern, `references/mvp-ledger-cash-wac-positions.md` for cash/WAC/position ledger calculation rules and tests, `references/mvp-crypto-wallet-ledger.md` for crypto wallet/assets/holdings/transfers/buy-sell/fee MVP rules, `references/crypto-xlsx-initial-snapshot-dry-run.md` for the controlled real-file XLSX crypto initial-snapshot dry-run pattern and mapping decisions, `references/crypto-xlsx-initial-snapshot-production-import.md` for productive safe-subset XLSX import operations, compatibility/partial-write recovery, and dashboard correction follow-up, `references/crypto-single-coin-runtime-test.md` for controlled one-coin productive runtime tests and the coin-vs-holding ambiguity guard, `references/multi-wallet-crypto-and-equity-manage-ui.md` for approved multi-wallet single-coin snapshots, SQLite Decimal/TEXT affinity pitfalls, and synthetic-first Equity/ETF Manage UI patterns, `references/crypto-price-refresh-and-idempotent-btc-runtime.md` for robust batch price refresh and re-run-safe BTC/single-coin runtime checks, `references/crypto-runtime-db-consistency-verification.md` for post-import runtime DB consistency checks, exact safe-set verification, dashboard no-live-API testing, and runtime-only report generation, `references/mvp-crypto-manual-management-ui.md` for audited dashboard add/correct/remove crypto holding workflows after an initial snapshot, `references/mvp-market-import-safety-hardening.md` for CoinGecko provider/backoff, CSV importer completion, alert lifecycle, Git-safety and crypto Decimal precision patterns, `references/git-safety-verification-coverage.md` for the verifier-warning workflow and minimum Git-safety test matrix, `references/mvp-streamlit-dashboard.md` for the safe local Streamlit dashboard shell/read-model/testing pattern, `references/mvp-crypto-report-export.md` for local-only crypto report context/export rules, `references/github-token-push-from-drive.md` for the safe Google-Drive-token push workflow when local FinanceManager commits are ahead of GitHub, `references/runtime-backup-and-broker-mapping.md` for runtime DB backup/restore hardening plus PostFinance/True Wealth/Raiffeisen structural mapping/dry-run rules, `references/import-wizard-mapping-layer.md` for the Phase B1 import wizard, instrument/account mapping, review queue, data-quality and read-only safety pattern, `references/broker-parser-dry-run-pipeline.md` for the Phase B2 synthetic broker parser, dry-run, runtime review-item and UI quality-flag pipeline, `references/controlled-real-broker-dry-run.md` for the Phase B3 controlled real-file dry-run pattern with backup, aggregate-only reporting, runtime-only review items, and no productive position writes, and `references/broker-review-queue-hardening.md` for Phase B4 manual review queue filters, auditable mapping actions, import-readiness gating, dry-run session management, and Data Quality Center hardening without productive portfolio writes, and `references/guided-manual-review-pilot.md` for Phase B5 small-scope runtime metadata pilots that move selected review items to `ready_for_import` without productive imports, `references/broker-import-execution-plan-mini-import.md` for Phase B6 final dry-run execution plans plus exactly-one productive broker mini-import, `references/broker-mini-import-regression-and-corrections.md` for post-mini-import regression, void/correction workflows, backup/restore checks, and one-at-a-time follow-up imports, and `references/broker-mini-import-completion-and-dq-consolidation.md` for completing the last ready True-Wealth mini-import plus active-vs-resolved Data Quality alert consolidation, `references/equity-fx-market-data-phase-c.md` for the local FX/market-data foundation after broker mini-imports and before broader Equity/ETF valuation/imports, and `references/true-wealth-valuation-pilot-c2.md` for the True-Wealth valuation pilot safeguards around hedge metadata, provider mappings, historical FX failures, timestamp alignment, corporate-action flags, and aggregate-only runtime reporting, `references/true-wealth-mapping-candidate-review-c3.md` for the controlled mapping-candidate review pattern before any productive Equity/ETF price or FX valuation, and `references/instrument-catalog-manual-add-workflow.md` for the general Instrument Catalog + dashboard manual-add workflow that must underpin future Equity/ETF/Crypto adds. Key decisions:

- Platform/account views: Raiffeisen, PostFinance, True Wealth, Crypto-Coins, plus Gesamtansicht.
- Existing Google Drive files under `Finanzen/01 Aktueller Stand Portfolios` are **initial-snapshot reality checks**, not full ledgers.
- MVP means the smallest useful first version: correct data foundation, ledger, CHF/FX logic, platform views, crypto inventory, watchlist, reports, workflows and audit log.
- Stock/ETF tax documents are downgraded: optional reference/control module and dividend/distribution transparency, not MVP 1 and not a complete tax dossier replacement.
- Crypto inventory is promoted to MVP 1: wallet/platform-level holdings, Crypto PDF, CoinGecko valuation, stale-price warnings and manual verification.
- Crypto snapshot values are stale; use quantities only and recalculate current values via CoinGecko later.
- MVP dashboard recommendation: Streamlit + SQLite + SQLAlchemy/Pandas/Plotly; later migrateable to FastAPI + React/PostgreSQL.
- MVP inputs: manual dashboard entry, standard CSV import/export, later broker/bank export import. No direct broker integrations initially.
- Telegram crypto buys/sells/transfers should be parsed as pending transactions and only saved after explicit confirmation.
- Every portfolio or crypto change must create a History/Audit entry.

## Required data model areas

At minimum design for:

- `platforms`, `accounts`, `instruments`
- `transactions` with original currency and CHF fields
- `cash_ledger`
- derived `positions_current`
- `dividends_distributions`
- `market_prices`, `fx_rates`
- `watchlist_items`
- `alerts`
- `decision_journal`
- `reports`
- `data_quality_issues`

## CSV/import rules

- For broker/bank imports (PostFinance, True Wealth, Raiffeisen), build the safe review layer before productive imports: Import Wizard, instrument mappings, platform/account mappings, dry-run summaries, Manual Review Queue, Data Quality Center and visible read-only/write-enabled state. Follow `references/import-wizard-mapping-layer.md`. When wiring parser output into the wizard, use the synthetic-first parser/dry-run/review-item pipeline in `references/broker-parser-dry-run-pipeline.md`. After explicit approval for real-file dry-runs, follow `references/controlled-real-broker-dry-run.md`: preflight, verified runtime backup, temp-only file handling, aggregate-only reporting, runtime-only review items, and verification that productive positions/ledger rows stayed unchanged. When the user asks to make real dry-run results manually editable, follow `references/broker-review-queue-hardening.md`: add filters/detail views, auditable review-metadata actions, per-item/session import-readiness status, dry-run session management, and Data Quality Center counts while keeping productive portfolio writes disabled. For a guided manual-review pilot on real runtime items, follow `references/guided-manual-review-pilot.md`: select 1–3 clear ETF/equity items (prefer True Wealth), write only audited review metadata, correct only presence booleans if needed, derive `ready_for_import` via readiness logic, and verify productive tables remain unchanged. Before the first productive broker position import, follow `references/broker-import-execution-plan-mini-import.md`: create runtime-only execution plans from the ready review items, validate exact Decimal-text payloads from the runtime source, preserve missing FX/cost-basis quality flags, then import exactly one ledger `initial_position_snapshot` after a verified backup; do not write `positions_snapshot` as the source of truth. After the first mini-import and before broader imports, follow `references/broker-mini-import-regression-and-corrections.md`: verify import links/audit/Decimal handling/re-import blocking, add or test safe void/correction flows, run backup/restore-to-test-DB regression, and only then optionally import one more ready plan. When the user approves finishing a remaining small set, follow `references/broker-mini-import-completion-and-dq-consolidation.md`: first disambiguate active vs resolved stale crypto alerts with `check-data-quality --scope crypto --resolve-fixed`, then import exactly one remaining ready execution plan after backup/verify, and consolidate Review Queue/Data Quality counts without exposing values.
- Do not pivot into later analytics modules while the user has asked for import/mapping hardening. Monte Carlo, backtesting, rebalancing, scores, heatmaps and projections are later phases unless explicitly re-authorized.
- Use a canonical CSV format before broker-specific importers.
- Imports must be idempotent via external transaction ID or row hash.
- Importers must separate dry-run and commit modes, collect validation errors explicitly, and always create an `import_sessions` record even for dry runs or failed imports.
- Import results should classify rows as new, existing, duplicate, or failed; do not rely on `INSERT OR IGNORE` as the only duplicate-detection mechanism.
- Row hashes must be stable and deterministic: normalize key order, whitespace, and `None` versus empty strings while leaving domain-specific canonicalization to the importer.
- If a source file has only current holdings and no full history, import it as `initial_position_snapshot` / `initial_cash_snapshot`, not fake historical buys.
- Missing historic FX, fees, taxes, dates, investment case, target weights, or dividend details must be flagged as data-quality gaps.

## FX and return rules

For foreign currency assets, always distinguish:

- price/security performance in original currency
- FX effect to CHF
- realized P&L
- unrealized P&L
- dividends/distributions/income
- fees and taxes
- Total Return in CHF

Every transaction/income event needs:

- original amount and currency
- historical FX rate to CHF, or explicit `fx_status=missing` with a critical data-quality alert
- FX source and timestamp
- CHF equivalent at event date when FX is known

Manual FX overrides require a non-empty note and an audit-log entry.

## Ledger calculation rules

- Cash after system start must be calculated from `transactions`; `cash_balances` is only a snapshot/cache/control view.
- `initial_cash_snapshot` and `initial_position_snapshot` are explicit starting truths, not fabricated historical trades.
- Implement Weighted Average Cost as the MVP internal performance method: buy fees increase cost basis, partial sells remove proportional cost basis, realized P&L is calculated against average cost, full sells zero remaining quantity/cost basis, and dividends do not alter quantity or cost basis.
- WAC is not tax-complete; do not add FIFO/LIFO/tax-lot logic in MVP 1, but keep transaction granularity so it can be added later.
- Position calculations must keep zero-quantity historical positions visible when sale history exists.
- Missing market price or FX must degrade data quality and suppress precise Total Return CHF rather than inventing a value.
- Before productive dashboard use, add alert deduplication per rule/entity/date.
- Market-FX for non-CHF market prices belongs to the later Market/FX provider block.

## Crypto MVP rules

- Crypto inventory is MVP-critical but must remain dummy-first until mapper/dry-run layers are ready; do not touch real coin/wallet files during core implementation. After explicit user approval, real files may be read only from temp/runtime for controlled dry runs using `references/crypto-xlsx-initial-snapshot-dry-run.md`; report only aggregate counts/labels and never quantities or values. Productive safe-subset imports require a second explicit approval and must follow `references/crypto-xlsx-initial-snapshot-production-import.md`: external runtime DB only, no real CSV/XLSX/reports/DBs in Git, Git-safety before/after, import-session + audit-log creation, and aggregate-only reporting.
- Wallet names must be unique; wallet addresses are optional and must not appear in tests/fixtures.
- Validate wallet type against: Hardware Wallet, Software Wallet, Exchange, Bank/Broker, DeFi, Sonstiges.
- CoinGecko ID is strongly recommended but not hard-required; missing ID creates a data-quality warning and blocks reliable valuation.
- Crypto symbols are not globally unique; symbol conflicts require warning/manual selection, never silent merging.
- Initial crypto holdings are confirmed snapshots by wallet+asset. Store legacy values for provenance/control only, never current valuation.
- Crypto holdings should be calculated/controlled from initial snapshots plus crypto transactions, not silently overwritten.
- Crypto transfers require different source/target wallets, quantity > 0, and same asset; coin fees reduce source-wallet balance.
- Negative wallet balances should be blocked in MVP and produce a critical alert unless a future explicit correction workflow is implemented.
- Crypto buys/sells with fiat impact must create/reference general ledger transactions via `source_type='crypto'`/`source_id` so cash remains ledger-consistent.
- Fiat fees should be ledger fees when unambiguous; avoid double counting when a fee is already embedded in sell net proceeds.
- Unknown fee currency creates a data-quality warning instead of invented accounting.
- Crypto Cost Basis/WAC is not required for MVP 1; MVP 1 prioritizes crypto holdings, wallet structure, current valuation, reports and audit.
- Crypto report context/export must be renderer-neutral and local-only: build from SQLite, aggregate by coin and wallet, use local cached prices only, preserve Decimal/Text values, ignore legacy snapshot values for current valuation, include data-quality warnings, and write generated files only to the external runtime reports directory.
- Crypto PDF export should be optional-dependency tolerant: prefer PDF when a renderer is available, otherwise fall back to HTML/Markdown with a clear warning while still recording report metadata and audit.
- After productive real-data crypto imports, run the post-import runtime verification in `references/crypto-runtime-db-consistency-verification.md`: exact safe-set membership checks, aggregate-only counts, audit/import-session traceability, dashboard render with no live API, runtime-only report generation, and Git-safety before/after.
- Manual dashboard additions/corrections/removals after an initial snapshot must follow `references/mvp-crypto-manual-management-ui.md`: no silent `crypto_holdings` mutation, explicit confirmation, mandatory notes for adjustments/removals, audit-log entries with old/new values, Decimal/TEXT quantity handling, no negative holdings, and no live API lookup during render.
- Crypto tax logic, tax lots, FIFO/LIFO and detailed cost basis are later modules.
- Keep crypto assets separate from `instruments` for MVP 1, but later evaluate a shared Asset Master if it improves reporting, performance views or watchlists.

## Income/tax rules

For dividends and ETF distributions, separately store:

- gross dividend/distribution
- Swiss withholding tax 35%
- foreign withholding tax
- other tax/withholding if present
- net amount
- original currency
- historic FX to CHF
- gross/net CHF equivalents
- platform/account
- ISIN/ticker
- payment date
- ex-dividend date if available

Plan later ESTV/ICTax support for Swiss year-end values and tax reports, but label outputs as technical reports, not tax advice.

## Market-data rules

- Dashboard reads primarily from local DB, not directly from APIs on page load.
- For Equity/ETF FX and market-data hardening after broker mini-imports, follow `references/equity-fx-market-data-phase-c.md`: validate active `missing_fx` alerts by original transaction currency first, correct only CHF false positives with audit, keep foreign-currency missing-FX alerts active until local historical rates exist, use ISIN as instrument identity and provider symbols only as market-data mappings, and run market-price providers only from explicit CLI/scheduled jobs.
- For the True-Wealth valuation pilot, follow `references/true-wealth-valuation-pilot-c2.md`: do not import more positions, do not guess provider symbols or hedge status, extend instrument metadata for currency hedging/status/valuation policy/corporate-action state, handle historical-FX paywalls as `manual_override_required` without crashing, align price/FX timestamps before claiming valuation precision, and report only aggregate runtime counts.
- For True-Wealth mapping-candidate review after the valuation pilot, follow `references/true-wealth-mapping-candidate-review-c3.md`: create runtime-only provider-symbol candidates keyed by ISIN/instrument, keep name-only or ticker-without-exchange candidates review-required, mark multiple candidates `needs_manual_review`, do not auto-select final mappings on ambiguity, keep hedge/instrument status unknown until confirmed, and ensure market-price dry runs call no provider when there are zero confirmed mappings.
- For the broader Instrument Catalog + manual-add workflow, follow `references/instrument-catalog-manual-add-workflow.md`: documents/import files are initial-population aids, but the dashboard must allow audited manual add of equities/ETFs and later coins via search, candidate selection, account/wallet selection, Decimal/Text quantity, date, cost-basis/snapshot info and notes. ISIN is primary for equities/ETFs but can have multiple listings; ticker/symbol alone is never unique; provider symbols are market-data mappings only; CoinGecko ID is primary for crypto provider identity. External lookups may use only non-confidential instrument metadata and must never send quantities, balances, account/wallet IDs, cost basis, private notes or source documents.
- First Streamlit MVP should be a safe, local, read-first shell over SQLite with explicit synthetic demo DB creation only; page render functions must not call CoinGecko/market APIs, and tests should prove that by monkeypatching provider methods to fail.
- Use scheduled jobs to update prices/FX/news.
- Store provider, timestamp, quality status and freshness for every price/FX point.
- Implement caching, local historical storage, batch requests where possible, provider rate-limit queues, and backoff on HTTP 429.
- CoinGecko integrations must be mockable and unit-tested without real API calls; treat HTTP 429 as non-fatal and return a non-fresh data-quality status plus alert after configured retries.
- Crypto price refresh belongs in CLI/scheduled-job commands, not dashboard/report rendering. Prefer CoinGecko batch requests over per-coin requests; expose conservative controls (`--sleep-seconds`, `--max-retries`, `--initial-backoff`, `--max-backoff`, `--only-missing`, `--only-stale`, `--only-symbol`, `--limit`, `--dry-run`, `--currency`). Commands should use aggregate-only output (`total_assets/updated/cached/skipped/stale/warnings/errors/assets_still_missing_local_price`), cache by configurable `max_age_seconds`, store exact Decimal prices as TEXT in `crypto_prices`, and verify dashboard/report contexts read local prices with `live_api_calls=False`. For details and tests see `references/crypto-price-refresh-and-idempotent-btc-runtime.md`.
- Candidate providers: Finnhub, Financial Modeling Prep, Twelve Data, Stooq, Alpha Vantage selectively, Yahoo Finance fallback only; CoinGecko for crypto; later DefiLlama; ESTV/ICTax for Swiss tax values.

## Crypto Decimal precision rules

SQLite has no exact native DECIMAL. In the crypto context, store quantities, prices, crypto fees and valuation-relevant amounts as exact Decimal strings (`TEXT`) and compute with Python `Decimal`. Never convert crypto quantity/price/fee/valuation through float/REAL. Format Decimal writes with `format(value, "f")` so small quantities such as `0.00000001` do not become scientific notation. CSV importers must validate Decimal strings, and tests should assert high-precision preservation plus SQLite text storage. For multi-wallet real-data snapshots and Equity/ETF ledger fields, also see `references/multi-wallet-crypto-and-equity-manage-ui.md`; SQLite `NUMERIC` affinity can coerce Decimal-looking text to `real`, so schema/migrations should use `TEXT` for exact quantities, FX rates, original amounts and CHF equivalents where precision matters.

## Alert lifecycle rules

Alerts must be deduplicated and lifecycle-aware before dashboards run repeated calculations. Use a dedup key from `rule_id`, `entity_type`, `entity_id`, `priority` and an error fingerprint. Alerts should carry `status` (`active`, `resolved`, `muted`), `resolved_at`, optional `muted_until`, `last_seen_at`, `occurrence_count`, `fingerprint` and `dedup_key`. Re-seeing an identical active/resolved/muted alert should update `last_seen_at`/`occurrence_count` rather than create a new active row; changed fingerprints may create a new alert.

## Git/GitHub safety

Code/scripts/docs may be stored in a private Git repo, but never commit financial data. Use the `github-repo-management` skill and its `references/sensitive-code-backup.md` guidance.

Minimum `.gitignore` for finance repos must block `data/`, `reports/`, `exports/`, `backups/`, DB files, spreadsheets, PDFs, DOCX, raw CSV/JSON, `.env`, tokens, credentials and client secrets. Only synthetic examples under `examples/` are acceptable.

## MVP implementation safety gate

Before any real broker/bank/portfolio/crypto data is imported, build and verify the safety foundation described in `references/mvp-safety-gate-implementation.md`: repo skeleton, `.gitignore`, `.env.example`, runtime outside repo, settings validation, SQLite schema, synthetic fixtures, audit log, CSV dry-run, duplicate protection, and Git-safety scan. Run the safety scan before staging, committing, or pushing. If GitHub credentials are needed from Drive, use them only in a temporary location and never echo token values.

Before broader productive imports, implement or verify the runtime backup loop in `references/runtime-backup-and-broker-mapping.md`: backup outside Git, SHA256 verification, restore requiring explicit confirmation, tests for backup/restore/checksum, and Git-safety coverage that blocks backup artifacts in the repo. For PostFinance/True Wealth/Raiffeisen, perform structural mapping and dry-runs only until the user approves import; report aggregate counts/field coverage/quality flags only, never real values or raw rows.

## Common pitfalls

- Starting later analytics (Monte Carlo, backtesting, rebalancing, scoring, heatmaps, projections) while import/mapping/data-quality phases are still being hardened or while the user explicitly asked to postpone them.
- Treating a holdings snapshot as a real transaction history.
- Mixing price P&L and FX P&L.
- Letting dashboard refreshes hammer APIs.
- Committing realistic sample data to Git because the repo is private. Private GitHub is not a financial data vault.
- Showing real portfolio values back to the user unnecessarily in summaries.
- Designing Git-safety scans that block source package names like `src/.../imports` or `src/.../reports`; block top-level runtime/data directories instead.
- In broker/bank real-file dry-runs, aggregating Data Quality flags only for `review_status='open'`. Blocked review items are still actionable data-quality issues; include `open` and `blocked` so defensive Raiffeisen aggregate-only rows remain visible.
- Letting manual review actions blur into productive import actions. Review-metadata writes (ISIN/ticker/exchange/account/instrument confirmations, ignore/resolve, dry-run archive/current status) may be allowed with audit logs, but they must not write `positions_snapshot`, `transactions`, `cash_balances`, ledger rows, or cash snapshots. Name-only mapping must never become `mapped`/`ready_for_import`; ticker-only without exchange remains review-needed; `aggregate_only` remains blocked for instrument import. In guided pilots, do not force `ready_for_import` by direct status edits; let readiness logic derive it after audited metadata actions, and if parser presence flags were missed, correct only boolean presence metadata with an audit event, never actual quantities/values.
- Skipping the execution-plan layer between ready review items and productive broker imports. A final dry-run should materialize runtime-only execution plans with source row hash, target IDs, exact Decimal-text payload summary, quality flags and planned write status. Productive mini-imports should consume exactly one planned ready payload, write exactly one ledger `initial_position_snapshot`, mark that plan/review item imported, leave the other ready plans untouched, and verify zero direct `positions_snapshot`/cash writes. After the first productive mini-import, do not proceed straight to mass import: run regression checks, implement/test void and correction audit paths, verify backup/restore to a separate test DB, and import at most one additional ready plan. When later completing the final ready plan, first resolve Data Quality alert ambiguity by counting only active alerts (`status='active'`) and treating resolved rows as history; then import exactly one plan and consolidate counts. See `references/broker-import-execution-plan-mini-import.md`, `references/broker-mini-import-regression-and-corrections.md`, and `references/broker-mini-import-completion-and-dq-consolidation.md`.
- Running compile/tests before Git-safety and forgetting generated `__pycache__`; remove or ignore generated caches before safety scans. `pytest` may recreate `.pytest_cache`, so remove `__pycache__`/`.pytest_cache` again after test execution and before `git-safety-scan`.
- Treating every active `missing_fx` alert after a broker mini-import as a bug. First aggregate by original currency: CHF rows may be false positives (`fx=1`, `not_needed`, resolve with audit), but foreign-currency rows without local historical FX should remain active and block precise CHF valuation until rates are populated.
- Auto-mapping Equity/ETF market prices by instrument name or ticker alone. ISIN is the instrument identity, but one ISIN can have multiple listings across exchange/currency, so even ISIN matches may need candidate selection. Provider symbols are separate market-data mappings, ticker without exchange is review-needed, and missing provider symbols should create deduplicated Data Quality alerts rather than triggering provider guesses. For mapping-candidate phases, candidate creation is allowed in the runtime DB, but final selection must remain manual/audited unless the user explicitly approves and the match is unambiguous.
- Designing future portfolio entry as import-file-only. Imports/documents are for initial population and existing holdings; the system must also support dashboard manual add/search workflows with audited ledger writes, explicit candidate selection, Decimal/Text quantities, required notes where history is incomplete, and no silent overwrites. See `references/instrument-catalog-manual-add-workflow.md`.
- Letting an Equity/ETF market-price dry run call providers when no confirmed mappings exist. The safe outcome is `total_mappings=0`, no provider calls, and aggregate reporting only.
- For valuation pilots on historical broker snapshots, treating a computable current market value as a clean performance result. `snapshot_only`, `cost_basis_uncertain`, missing hedge confirmation, missing historical FX, stale price/FX timestamp alignment, or suspected corporate actions must keep valuation `partially_valuable` or suppress precise Total Return until reviewed.
- Inferring CHF-hedged ETF behavior from the name alone. Hedge status is methodology-critical: unknown status should create a warning/review item; confirmed CHF hedging must prevent naive free FX-PnL attribution.
- Crashing or blocking on historical-FX provider paywalls. Persist `manual_override_required`, create deduplicated alerts, keep the flow moving, and require a noted audited manual override instead of guessing latest FX.
- Productive real-data imports can partially write before surfacing schema/importer compatibility issues. Do not rerun blindly or dump row-level data; inspect aggregate counts and unique identifiers needed for idempotency, patch/complete via missing-only inserts, then verify committed import session, audit log, Git clean and Git-safety OK.
- In controlled one-coin runtime tests from a real XLSX, distinguish **one coin/asset** from **one holding/wallet row**. A coin may be split across multiple wallet columns; if the user required an unambiguous target wallet or one holding, stop and ask rather than choosing a wallet silently. If the user explicitly approves multi-wallet import for that coin, import exactly one asset plus one audited holding per non-empty approved wallet cell; verify aggregate counts only. If re-running a BTC/single-coin test finds the exact approved asset and holdings already present, do not import duplicates; perform the aggregate-only idempotency checks and report that asset/holding counts did not increase because duplicate prevention succeeded. See `references/crypto-single-coin-runtime-test.md`, `references/multi-wallet-crypto-and-equity-manage-ui.md`, and `references/crypto-price-refresh-and-idempotent-btc-runtime.md`.
- After productive crypto imports, do a separate aggregate-only consistency pass before building new features: DB/runtime outside repo, exact safe-subset membership, no duplicate/negative/orphan holdings, correct snapshot date, null legacy-value fields when ignored, import sessions committed, holding-level audit traces present, dashboard local-read/no-live-API verified, and report generated only under runtime reports.
- Letting a dashboard correction UI mutate crypto holdings directly. Add/remove/edit operations must become audited transactions or `manual_adjustment` events with mandatory notes where appropriate; preserve history instead of silently overwriting balances.
- Treating verifier/file-mutation warnings as either automatically true or automatically false. Check the exact commit with `git show --name-status --patch <commit> -- <file>`, then separately verify whether the test matrix actually covers the promised cases.
- Adding broad Git-safety tests that only assert `len(findings) >= N`; prefer explicit assertions for token filename, PAT patterns, private keys, CSV/JSON path policy, runtime/data directories, allowed synthetic paths, `config.example/`, and normal `src/` false-positive protection.

To view linked files, call skill_view(name, file_path) where file_path is e.g. 'references/api.md' or 'assets/config.yaml'


terminal

PYTHONPATH=src python -m compileall -q src tests


Output:
M src/jarvis_finance/cli/main.py
 M src/jarvis_finance/dashboard/data.py
 M src/jarvis_finance/dashboard/pages/page_13_equity_manage.py
 M src/jarvis_finance/market_data/candidates.py
 M src/jarvis_finance/storage/migrations.py
 M tests/unit/test_schema.py
 M tests/unit/test_true_wealth_mapping_candidates_c3.py
 src/jarvis_finance/cli/main.py                     |  19 ++-
 src/jarvis_finance/dashboard/data.py               |  12 +-
 .../dashboard/pages/page_13_equity_manage.py       |  14 +-
 src/jarvis_finance/market_data/candidates.py       | 189 ++++++++++++++++++++-
 src/jarvis_finance/storage/migrations.py           |  12 +-
 tests/unit/test_schema.py                          |   2 +-
 .../unit/test_true_wealth_mapping_candidates_c3.py |   8 +
 7 files changed, 238 insertions(+), 18 deletions(-)


     1|from __future__ import annotations
     2|
     3|import csv
     4|import json
     5|import os
     6|import urllib.parse
     7|import urllib.request
     8|from dataclasses import dataclass, field
     9|from pathlib import Path
    10|from sqlite3 import Connection
    11|from typing import Protocol
    12|
    13|from jarvis_finance.audit.log import record_audit_event
    14|from jarvis_finance.imports.common import stable_id, utc_now
    15|from jarvis_finance.market_data.instruments import ensure_instrument_metadata_quality, resolve_instrument_alerts
    16|from jarvis_finance.market_data.mappings import confirm_instrument_price_mapping
    17|from jarvis_finance.quality.alerts import create_alert
    18|
    19|HEDGE_MARKERS = ("chf hedged", "hedged chf", "currency hedged", "hedged to chf", "chf-hedged")
    20|
    21|
    22|@dataclass(frozen=True)
    23|class ProviderCandidate:
    24|    provider: str
    25|    provider_symbol: str
    26|    exchange: str | None = None
    27|    currency: str | None = None
    28|    name: str | None = None
    29|    isin: str | None = None
    30|    asset_class: str | None = None
    31|    is_hedged: bool | None = None
    32|    hedged_to_currency: str | None = None
    33|    hedge_status: str = "unknown"
    34|    instrument_status: str = "unknown"
    35|    valuation_policy: str = "live_price"
    36|    evidence_source: str = "provider_lookup"
    37|    evidence_note: str = ""
    38|
    39|
    40|class InstrumentLookupProvider(Protocol):
    41|    name: str
    42|
    43|    def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
    44|        ...
    45|
    46|
    47|class LookupUnavailable(RuntimeError):
    48|    pass
    49|
    50|
    51|class EmptyLookupProvider:
    52|    name = "manual"
    53|
    54|    def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
    55|        return []
    56|
    57|
    58|class YahooSearchLookupProvider:
    59|    """Public metadata lookup via Yahoo Finance search endpoint; no prices or portfolio data."""
    60|
    61|    name = "yahoo_fallback"
    62|
    63|    def __init__(self, *, timeout_seconds: float = 8.0, max_results: int = 8):
    64|        self.timeout_seconds = timeout_seconds
    65|        self.max_results = max_results
    66|
    67|    def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
    68|        queries = [q for q in [isin, ticker, name] if q]
    69|        if not queries:
    70|            return []
    71|        found: dict[tuple[str, str | None], ProviderCandidate] = {}
    72|        for query in queries[:2]:
    73|            url = "https://query2.finance.yahoo.com/v1/finance/search?" + urllib.parse.urlencode({"q": query, "quotesCount": self.max_results, "newsCount": 0})
    74|            try:
    75|                req = urllib.request.Request(url, headers={"User-Agent": "JarvisFinance/1.0 public-instrument-lookup"})
    76|                with urllib.request.urlopen(req, timeout=self.timeout_seconds) as resp:
    77|                    payload = json.loads(resp.read().decode("utf-8"))
    78|            except Exception as exc:  # pragma: no cover - network dependent
    79|                raise LookupUnavailable(str(exc)) from exc
    80|            for quote in payload.get("quotes", [])[: self.max_results]:
    81|                symbol = quote.get("symbol")
    82|                if not symbol:
    83|                    continue
    84|                qtype = (quote.get("quoteType") or "").lower()
    85|                asset_class = "etf" if qtype == "etf" else "equity" if qtype in {"equity", "stock"} else None
    86|                exch = quote.get("exchDisp") or quote.get("exchange")
    87|                cand = ProviderCandidate(
    88|                    provider=self.name,
    89|                    provider_symbol=symbol,
    90|                    exchange=exch,
    91|                    currency=currency,
    92|                    name=quote.get("longname") or quote.get("shortname"),
    93|                    isin=isin if isin and query == isin else None,
    94|                    asset_class=asset_class,
    95|                    hedge_status="unknown",
    96|                    instrument_status="unknown",
    97|                    valuation_policy="live_price",
    98|                    evidence_source="yahoo_finance_search",
    99|                    evidence_note="Public symbol/search metadata only; no prices, holdings, account data, or portfolio quantities sent.",
   100|                )
   101|                found[(symbol, exch)] = cand
   102|        return list(found.values())
   103|
   104|
   105|class CompositePublicLookupProvider:
   106|    name = "public_composite"
   107|
   108|    def __init__(self, providers: list[InstrumentLookupProvider] | None = None):
   109|        self.providers = providers or [YahooSearchLookupProvider()]
   110|
   111|    def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
   112|        candidates: list[ProviderCandidate] = []
   113|        errors = 0
   114|        for provider in self.providers:
   115|            try:
   116|                candidates.extend(provider.lookup(isin=isin, name=name, ticker=ticker, exchange=exchange, currency=currency))
   117|            except Exception:
   118|                errors += 1
   119|        if errors and not candidates:
   120|            raise LookupUnavailable("all public lookup providers unavailable")
   121|        return candidates
   122|
   123|
   124|@dataclass
   125|class CandidateGenerationResult:
   126|    instruments_total: int = 0
   127|    candidates_found: int = 0
   128|    high_confidence: int = 0
   129|    medium_confidence: int = 0
   130|    low_confidence: int = 0
   131|    instruments_without_candidate: int = 0
   132|    instruments_with_hedge_unknown: int = 0
   133|    instruments_with_status_unknown: int = 0
   134|    provider_lookup_unavailable: int = 0
   135|    ambiguous_provider_mapping: int = 0
   136|    dry_run: bool = False
   137|    warnings: list[str] = field(default_factory=list)
   138|
   139|
   140|def _instrument_ids_for_true_wealth(conn: Connection, *, limit: int | None = None) -> list[str]:
   141|    sql = """
   142|        SELECT DISTINCT t.instrument_id
   143|        FROM transactions t
   144|        JOIN instruments i ON i.instrument_id=t.instrument_id
   145|        WHERE t.transaction_type='initial_position_snapshot'
   146|          AND t.source_type='broker_import_reviewed_snapshot'
   147|          AND COALESCE(t.is_voided,0)=0
   148|          AND t.instrument_id IS NOT NULL
   149|        ORDER BY i.isin, i.name
   150|    """
   151|    rows = conn.execute(sql).fetchall()
   152|    ids = [r["instrument_id"] for r in rows]
   153|    return ids[:limit] if limit is not None else ids
   154|
   155|
   156|def _infer_hedge(name: str | None, candidate: ProviderCandidate) -> tuple[bool | None, str | None]:
   157|    if candidate.is_hedged is not None:
   158|        return candidate.is_hedged, candidate.hedged_to_currency
   159|    haystack = " ".join(x for x in [name, candidate.name, candidate.evidence_note] if x).lower()
   160|    if any(marker in haystack for marker in HEDGE_MARKERS):
   161|        return True, "CHF"
   162|    return None, None
   163|
   164|
   165|def _confidence(inst, cand: ProviderCandidate) -> tuple[str, str]:
   166|    notes: list[str] = []
   167|    isin_match = bool(inst["isin"] and cand.isin and inst["isin"].upper() == cand.isin.upper())
   168|    ticker_match = bool(inst["ticker"] and cand.provider_symbol and inst["ticker"].upper() in cand.provider_symbol.upper())
   169|    has_exchange = bool(cand.exchange or inst["exchange"])
   170|    currency_match = bool((cand.currency or "").upper() == (inst["currency"] or "").upper()) if cand.currency and inst["currency"] else False
   171|    if isin_match:
   172|        notes.append("ISIN match")
   173|        if cand.provider_symbol and has_exchange:
   174|            return "high", "; ".join(notes + (["currency match"] if currency_match else []))
   175|        return "medium", "; ".join(notes + ["provider symbol/exchange incomplete"])
   176|    if ticker_match and has_exchange:
   177|        return "medium", "ticker+exchange match; requires manual review"
   178|    if cand.name:
   179|        return "low", "name-only or weak provider match; manual review required"
   180|    return "low", "weak candidate; manual review required"
   181|
   182|
   183|def _review_status_for(confidence: str, count: int, inst, cand: ProviderCandidate) -> str:
   184|    if count > 1:
   185|        return "needs_manual_review"
   186|    if confidence == "high":
   187|        return "proposed"
   188|    if confidence == "medium":
   189|        return "needs_manual_review"
   190|    return "needs_manual_review"
   191|
   192|
   193|def upsert_candidate(conn: Connection, *, instrument_id: str, candidate: ProviderCandidate, confidence: str, review_status: str, evidence_note: str) -> str:
   194|    inst = conn.execute("SELECT * FROM instruments WHERE instrument_id=?", (instrument_id,)).fetchone()
   195|    if inst is None:
   196|        raise ValueError(f"instrument not found: {instrument_id}")
   197|    hedged, hedged_to = _infer_hedge(inst["name"], candidate)
   198|    hedge_status = candidate.hedge_status if candidate.hedge_status in {"hedged", "unhedged", "unknown"} else "unknown"
   199|    if hedge_status == "unknown" and hedged is True:
   200|        hedge_status = "hedged"
   201|    elif hedge_status == "unknown" and hedged is False:
   202|        hedge_status = "unhedged"
   203|    candidate_id = stable_id("ipmc", instrument_id, candidate.provider, candidate.provider_symbol, candidate.exchange or "", candidate.currency or "")
   204|    now = utc_now()
   205|    conn.execute(
   206|        """
   207|        INSERT INTO instrument_price_mapping_candidates(
   208|            candidate_id, instrument_id, isin, candidate_provider, candidate_provider_symbol,
   209|            candidate_exchange, candidate_currency, candidate_name, candidate_asset_class,
   210|            candidate_is_hedged, candidate_hedged_to_currency, candidate_hedge_status,
   211|            candidate_instrument_status, candidate_valuation_policy, confidence,
   212|            evidence_source, evidence_note, review_status, created_at, updated_at
   213|        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
   214|        ON CONFLICT(candidate_id) DO UPDATE SET
   215|            candidate_exchange=excluded.candidate_exchange,
   216|            candidate_currency=excluded.candidate_currency,
   217|            candidate_name=excluded.candidate_name,
   218|            candidate_asset_class=excluded.candidate_asset_class,
   219|            candidate_is_hedged=excluded.candidate_is_hedged,
   220|            candidate_hedged_to_currency=excluded.candidate_hedged_to_currency,
   221|            candidate_hedge_status=excluded.candidate_hedge_status,
   222|            candidate_instrument_status=excluded.candidate_instrument_status,
   223|            candidate_valuation_policy=excluded.candidate_valuation_policy,
   224|            confidence=excluded.confidence,
   225|            evidence_source=excluded.evidence_source,
   226|            evidence_note=excluded.evidence_note,
   227|            review_status=excluded.review_status,
   228|            updated_at=excluded.updated_at
   229|        """,
   230|        (
   231|            candidate_id, instrument_id, inst["isin"], candidate.provider, candidate.provider_symbol,
   232|            candidate.exchange, candidate.currency or inst["currency"], candidate.name, candidate.asset_class or (inst["asset_class"] or "").lower(),
   233|            1 if hedged else 0 if hedged is False else None, hedged_to, hedge_status,
   234|            candidate.instrument_status or "unknown", candidate.valuation_policy or "live_price", confidence, candidate.evidence_source,
   235|            evidence_note or candidate.evidence_note, review_status, now, now,
   236|        ),
   237|    )
   238|    return candidate_id
   239|
   240|
   241|def generate_mapping_candidates(
   242|    conn: Connection,
   243|    *,
   244|    provider: InstrumentLookupProvider | None = None,
   245|    instrument_ids: list[str] | None = None,
   246|    source: str = "true_wealth",
   247|    limit: int | None = None,
   248|    dry_run: bool = False,
   249|) -> CandidateGenerationResult:
   250|    lookup = provider or EmptyLookupProvider()
   251|    ids = instrument_ids if instrument_ids is not None else _instrument_ids_for_true_wealth(conn, limit=limit)
   252|    result = CandidateGenerationResult(instruments_total=len(ids), dry_run=dry_run)
   253|    for instrument_id in ids:
   254|        inst = conn.execute("SELECT * FROM instruments WHERE instrument_id=?", (instrument_id,)).fetchone()
   255|        if inst is None:
   256|            continue
   257|        ensure_instrument_metadata_quality(conn, instrument_id=instrument_id)
   258|        if (inst["hedge_status"] or "unknown") == "unknown":
   259|            result.instruments_with_hedge_unknown += 1
   260|        if (inst["instrument_status"] or "unknown") == "unknown":
   261|            result.instruments_with_status_unknown += 1
   262|        try:
   263|            candidates = lookup.lookup(isin=inst["isin"], name=inst["name"], ticker=inst["ticker"], exchange=inst["exchange"], currency=inst["currency"])
   264|        except Exception as exc:
   265|            result.provider_lookup_unavailable += 1
   266|            result.warnings.append(instrument_id)
   267|            if not dry_run:
   268|                create_alert(conn, priority="warnung", category="market_data", entity_type="instrument", entity_id=instrument_id, rule_id="provider_lookup_unavailable", message="Provider lookup unavailable for instrument mapping candidates.", evidence={"provider": getattr(lookup, "name", "unknown"), "error_type": type(exc).__name__}, fingerprint=f"provider_lookup_unavailable:{getattr(lookup, 'name', 'unknown')}")
   269|            continue
   270|        if not candidates:
   271|            result.instruments_without_candidate += 1
   272|            if not dry_run:
   273|                create_alert(conn, priority="warnung", category="market_data", entity_type="instrument", entity_id=instrument_id, rule_id="missing_provider_symbol", message="No provider-symbol candidate found; manual mapping required.", evidence={"source": source, "provider": getattr(lookup, "name", "manual")}, fingerprint="missing_provider_symbol")
   274|            continue
   275|        if len(candidates) > 1:
   276|            result.ambiguous_provider_mapping += 1
   277|            if not dry_run:
   278|                create_alert(conn, priority="warnung", category="market_data", entity_type="instrument", entity_id=instrument_id, rule_id="ambiguous_provider_mapping", message="Multiple provider-symbol candidates require manual review.", evidence={"candidate_count": len(candidates), "provider": getattr(lookup, "name", "unknown")}, fingerprint=f"ambiguous_provider_mapping:{getattr(lookup, 'name', 'unknown')}:{len(candidates)}")
   279|        for cand in candidates:
   280|            confidence, note = _confidence(inst, cand)
   281|            if confidence == "high":
   282|                result.high_confidence += 1
   283|            elif confidence == "medium":
   284|                result.medium_confidence += 1
   285|            else:
   286|                result.low_confidence += 1
   287|            status = _review_status_for(confidence, len(candidates), inst, cand)
   288|            result.candidates_found += 1
   289|            if not dry_run:
   290|                upsert_candidate(conn, instrument_id=instrument_id, candidate=cand, confidence=confidence, review_status=status, evidence_note=note)
   291|    if not dry_run:
   292|        conn.commit()
   293|    return result
   294|
   295|
   296|def dedupe_missing_provider_symbol_alerts(conn: Connection) -> int:
   297|    """Keep one active missing-provider-symbol alert per instrument; resolve duplicates."""
   298|    now = utc_now()
   299|    resolved = 0
   300|    rows = conn.execute(
   301|        """
   302|        SELECT entity_id, GROUP_CONCAT(alert_id) AS ids, COUNT(*) AS c
   303|        FROM alerts
   304|        WHERE status='active' AND rule_id='missing_provider_symbol' AND entity_type='instrument'
   305|        GROUP BY entity_id
   306|        HAVING COUNT(*) > 1
   307|        """
   308|    ).fetchall()
   309|    for row in rows:
   310|        ids = [x for x in (row["ids"] or "").split(",") if x]
   311|        for alert_id in ids[1:]:
   312|            conn.execute("UPDATE alerts SET status='resolved', resolved_at=?, last_seen_at=? WHERE alert_id=?", (now, now, alert_id))
   313|            resolved += 1
   314|    conn.commit()
   315|    return resolved
   316|
   317|
   318|def export_mapping_review_template(conn: Connection, *, output_path: str | Path) -> dict[str, int | str]:
   319|    path = Path(output_path).expanduser().resolve()
   320|    path.parent.mkdir(parents=True, exist_ok=True)
   321|    rows = conn.execute(
   322|        """
   323|        SELECT i.instrument_id, i.asset_class, i.name, i.isin, i.ticker, i.exchange, i.currency,
   324|               c.candidate_provider, c.candidate_provider_symbol, c.candidate_exchange,
   325|               c.candidate_currency, c.confidence, c.candidate_hedge_status,
   326|               c.candidate_instrument_status, c.evidence_source, c.evidence_note,
   327|               c.review_status
   328|        FROM instruments i
   329|        LEFT JOIN instrument_price_mapping_candidates c ON c.instrument_id=i.instrument_id
   330|        WHERE i.instrument_id IN (
   331|            SELECT DISTINCT instrument_id FROM transactions
   332|            WHERE transaction_type='initial_position_snapshot'
   333|              AND source_type='broker_import_reviewed_snapshot'
   334|              AND COALESCE(is_voided,0)=0
   335|              AND instrument_id IS NOT NULL
   336|        )
   337|        ORDER BY i.isin, i.name, c.confidence, c.candidate_provider
   338|        """
   339|    ).fetchall()
   340|    fields = [
   341|        "instrument_id", "asset_class", "name", "isin", "ticker", "exchange", "currency",
   342|        "candidate_provider", "candidate_provider_symbol", "candidate_exchange", "candidate_currency",
   343|        "confidence", "hedge_status_candidate", "instrument_status_candidate", "evidence_source",
   344|        "evidence_note", "review_status", "reviewer_decision", "reviewer_note",
   345|    ]
   346|    with path.open("w", newline="", encoding="utf-8") as handle:
   347|        writer = csv.DictWriter(handle, fieldnames=fields)
   348|        writer.writeheader()
   349|        for row in rows:
   350|            writer.writerow({
   351|                "instrument_id": row["instrument_id"], "asset_class": row["asset_class"], "name": row["name"],
   352|                "isin": row["isin"], "ticker": row["ticker"], "exchange": row["exchange"], "currency": row["currency"],
   353|                "candidate_provider": row["candidate_provider"], "candidate_provider_symbol": row["candidate_provider_symbol"],
   354|                "candidate_exchange": row["candidate_exchange"], "candidate_currency": row["candidate_currency"],
   355|                "confidence": row["confidence"], "hedge_status_candidate": row["candidate_hedge_status"],
   356|                "instrument_status_candidate": row["candidate_instrument_status"], "evidence_source": row["evidence_source"],
   357|                "evidence_note": row["evidence_note"], "review_status": row["review_status"],
   358|                "reviewer_decision": "", "reviewer_note": "",
   359|            })
   360|    return {"output_path": str(path), "rows": len(rows), "candidate_rows": sum(1 for r in rows if r["candidate_provider_symbol"]), "instrument_count": len({r["instrument_id"] for r in rows})}
   361|
   362|
   363|def select_mapping_candidate(conn: Connection, *, candidate_id: str, note: str, created_by: str = "system") -> str:
   364|    if not note.strip():
   365|        raise ValueError("candidate selection requires a note")
   366|    cand = conn.execute("SELECT * FROM instrument_price_mapping_candidates WHERE candidate_id=?", (candidate_id,)).fetchone()
   367|    if cand is None:
   368|        raise ValueError(f"candidate not found: {candidate_id}")
   369|    mapping_id = confirm_instrument_price_mapping(
   370|        conn,
   371|        instrument_id=cand["instrument_id"],
   372|        provider=cand["candidate_provider"],
   373|        provider_symbol=cand["candidate_provider_symbol"],
   374|        provider_market=cand["candidate_exchange"],
   375|        confidence=cand["confidence"],
   376|        note=note,
   377|        exchange=cand["candidate_exchange"],
   378|        currency=cand["candidate_currency"],
   379|        hedge_status=cand["candidate_hedge_status"] if cand["candidate_hedge_status"] != "unknown" else None,
   380|        hedged_to_currency=cand["candidate_hedged_to_currency"],
   381|        instrument_status=cand["candidate_instrument_status"] if cand["candidate_instrument_status"] != "unknown" else None,
   382|        valuation_policy=cand["candidate_valuation_policy"] or ("live_price" if cand["candidate_instrument_status"] == "active" else None),
   383|        trading_currency=cand["candidate_currency"],
   384|        created_by=created_by,
   385|    )
   386|    now = utc_now()
   387|    conn.execute("UPDATE instrument_price_mapping_candidates SET review_status='selected', updated_at=? WHERE candidate_id=?", (now, candidate_id))
   388|    conn.execute("UPDATE instrument_price_mapping_candidates SET review_status='rejected', updated_at=? WHERE instrument_id=? AND candidate_id<>? AND review_status IN ('proposed','needs_manual_review')", (now, cand["instrument_id"], candidate_id))
   389|    record_audit_event(
   390|        conn,
   391|        source="instrument_price_mapping_candidate",
   392|        action="select_mapping_candidate",
   393|        entity_type="instrument_price_mapping_candidate",
   394|        entity_id=candidate_id,
   395|        new_values={"mapping_id": mapping_id, "instrument_id": cand["instrument_id"], "provider": cand["candidate_provider"], "provider_symbol": cand["candidate_provider_symbol"]},
   396|        user_text_note=note,
   397|        confirmed=True,
   398|        created_by=created_by,
   399|    )
   400|    resolve_instrument_alerts(conn, instrument_id=cand["instrument_id"], rule_ids=["missing_provider_symbol", "ambiguous_provider_mapping"])
   401|    conn.commit()
   402|    return mapping_id
   403|
   404|
   405|def reject_mapping_candidate(conn: Connection, *, candidate_id: str, note: str, created_by: str = "system") -> None:
   406|    if not note.strip():
   407|        raise ValueError("candidate rejection requires a note")
   408|    cand = conn.execute("SELECT * FROM instrument_price_mapping_candidates WHERE candidate_id=?", (candidate_id,)).fetchone()
   409|    if cand is None:
   410|        raise ValueError(f"candidate not found: {candidate_id}")
   411|    now = utc_now()
   412|    conn.execute("UPDATE instrument_price_mapping_candidates SET review_status='rejected', updated_at=? WHERE candidate_id=?", (now, candidate_id))
   413|    record_audit_event(
   414|        conn,
   415|        source="instrument_price_mapping_candidate",
   416|        action="reject_mapping_candidate",
   417|        entity_type="instrument_price_mapping_candidate",
   418|        entity_id=candidate_id,
   419|        old_values={"review_status": cand["review_status"]},
   420|        new_values={"review_status": "rejected"},
   421|        user_text_note=note,
   422|        confirmed=True,
   423|        created_by=created_by,
   424|    )
   425|    conn.commit()
   426|
   427|
   428|def get_mapping_candidate_summary(conn: Connection) -> dict[str, int]:
   429|    rows = conn.execute("SELECT confidence, COUNT(*) AS c FROM instrument_price_mapping_candidates GROUP BY confidence").fetchall()
   430|    out = {"total": 0, "high": 0, "medium": 0, "low": 0}
   431|    for row in rows:
   432|        key = row["confidence"] or "low"
   433|        if key in out:
   434|            out[key] += int(row["c"] or 0)
   435|        out["total"] += int(row["c"] or 0)
   436|    return out
   437|


     1|from __future__ import annotations
     2|
     3|from decimal import Decimal
     4|
     5|import pytest
     6|
     7|from jarvis_finance.dashboard import data
     8|from jarvis_finance.dashboard.pages import page_13_equity_manage
     9|from jarvis_finance.fx.overrides import set_manual_fx_override
    10|from jarvis_finance.market_data.candidates import (
    11|    LookupUnavailable,
    12|    ProviderCandidate,
    13|    dedupe_missing_provider_symbol_alerts,
    14|    export_mapping_review_template,
    15|    generate_mapping_candidates,
    16|    select_mapping_candidate,
    17|)
    18|from jarvis_finance.market_data.instruments import ensure_instrument_metadata_quality
    19|from jarvis_finance.market_data.prices import refresh_market_prices
    20|from jarvis_finance.storage.database import connect_memory
    21|from jarvis_finance.storage.migrations import apply_migrations
    22|
    23|
    24|class StaticLookupProvider:
    25|    name = "static"
    26|
    27|    def __init__(self, candidates_by_isin=None, *, fail=False):
    28|        self.candidates_by_isin = candidates_by_isin or {}
    29|        self.fail = fail
    30|
    31|    def lookup(self, *, isin, name, ticker, exchange, currency):
    32|        if self.fail:
    33|            raise LookupUnavailable("lookup down")
    34|        return list(self.candidates_by_isin.get(isin, []))
    35|
    36|
    37|class RaisingPriceProvider:
    38|    name = "raising"
    39|
    40|    def get_price(self, symbol: str):
    41|        raise AssertionError("price provider should not be called without confirmed mappings")
    42|
    43|
    44|class FakeTab:
    45|    def __enter__(self):
    46|        return self
    47|
    48|    def __exit__(self, *args):
    49|        return False
    50|
    51|
    52|class FakeSt:
    53|    def title(self, *args, **kwargs): pass
    54|    def caption(self, *args, **kwargs): pass
    55|    def warning(self, *args, **kwargs): pass
    56|    def subheader(self, *args, **kwargs): pass
    57|    def write(self, *args, **kwargs): pass
    58|    def success(self, *args, **kwargs): pass
    59|    def error(self, *args, **kwargs): pass
    60|    def info(self, *args, **kwargs): pass
    61|    def divider(self, *args, **kwargs): pass
    62|    def dataframe(self, *args, **kwargs): pass
    63|    def table(self, *args, **kwargs): pass
    64|    def tabs(self, labels): return [FakeTab() for _ in labels]
    65|    def selectbox(self, label, options, **kwargs): return options[0] if options else None
    66|    def text_input(self, *args, **kwargs): return kwargs.get("value", "")
    67|    def text_area(self, *args, **kwargs): return ""
    68|    def date_input(self, *args, **kwargs): return "2025-12-31"
    69|    def checkbox(self, *args, **kwargs): return False
    70|    def expander(self, *args, **kwargs): return FakeTab()
    71|    def button(self, *args, **kwargs): return False
    72|
    73|
    74|def setup_conn():
    75|    conn = connect_memory()
    76|    apply_migrations(conn)
    77|    conn.execute("INSERT INTO platforms(platform_id,name,platform_type,default_currency,created_at) VALUES('p1','True Wealth','broker','CHF','now')")
    78|    conn.execute("INSERT INTO accounts(account_id,platform_id,account_name,account_type,currency,created_at) VALUES('a1','p1','TW','brokerage','CHF','now')")
    79|    conn.execute("INSERT INTO instruments(instrument_id,asset_class,name,ticker,isin,exchange,currency,created_at) VALUES('inst1','ETF','Global Equity ETF','GETF','CH0000000001',NULL,'USD','now')")
    80|    conn.execute("INSERT INTO transactions(transaction_id,transaction_type,account_id,instrument_id,trade_date,quantity,currency_original,fx_status,source_type,is_confirmed,quality_status,created_at) VALUES('tx1','initial_position_snapshot','a1','inst1','2025-12-31','1','USD','missing','broker_import_reviewed_snapshot',1,'warning','now')")
    81|    return conn
    82|
    83|
    84|def test_candidate_generator_creates_high_candidate_from_isin_match() -> None:
    85|    conn = setup_conn()
    86|    provider = StaticLookupProvider({
    87|        'CH0000000001': [ProviderCandidate(provider='stooq', provider_symbol='GETF.SW', exchange='SWX', currency='USD', name='Global Equity ETF', isin='CH0000000001', instrument_status='active')]
    88|    })
    89|    result = generate_mapping_candidates(conn, provider=provider)
    90|    row = conn.execute("SELECT * FROM instrument_price_mapping_candidates WHERE instrument_id='inst1'").fetchone()
    91|    mapping_count = conn.execute("SELECT COUNT(*) AS c FROM instrument_price_mappings WHERE mapping_status='mapped'").fetchone()['c']
    92|    assert result.candidates_found == 1
    93|    assert result.high_confidence == 1
    94|    assert row['confidence'] == 'high'
    95|    assert row['review_status'] == 'proposed'
    96|    assert row['candidate_asset_class'] == 'etf'
    97|    assert row['candidate_hedge_status'] == 'unknown'
    98|    assert row['candidate_valuation_policy'] == 'live_price'
    99|    assert mapping_count == 0
   100|
   101|
   102|def test_name_only_candidate_is_review_required_low_confidence() -> None:
   103|    conn = setup_conn()
   104|    provider = StaticLookupProvider({'CH0000000001': [ProviderCandidate(provider='manual', provider_symbol='UNKNOWN', name='Global Equity ETF')]})
   105|    generate_mapping_candidates(conn, provider=provider)
   106|    row = conn.execute("SELECT confidence,review_status FROM instrument_price_mapping_candidates").fetchone()
   107|    assert row['confidence'] == 'low'
   108|    assert row['review_status'] == 'needs_manual_review'
   109|
   110|
   111|def test_multiple_candidates_are_manual_review_and_alerted() -> None:
   112|    conn = setup_conn()
   113|    provider = StaticLookupProvider({'CH0000000001': [
   114|        ProviderCandidate(provider='stooq', provider_symbol='GETF.SW', exchange='SWX', isin='CH0000000001'),
   115|        ProviderCandidate(provider='yahoo', provider_symbol='GETF.SW', exchange='SWX', isin='CH0000000001'),
   116|    ]})
   117|    result = generate_mapping_candidates(conn, provider=provider)
   118|    statuses = {r['review_status'] for r in conn.execute("SELECT review_status FROM instrument_price_mapping_candidates").fetchall()}
   119|    assert result.ambiguous_provider_mapping == 1
   120|    assert statuses == {'needs_manual_review'}
   121|    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='ambiguous_provider_mapping' AND status='active'").fetchone()['c'] == 1
   122|
   123|
   124|def test_no_candidate_keeps_missing_provider_symbol_active() -> None:
   125|    conn = setup_conn()
   126|    result = generate_mapping_candidates(conn, provider=StaticLookupProvider({}))
   127|    assert result.instruments_without_candidate == 1
   128|    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='missing_provider_symbol' AND status='active'").fetchone()['c'] == 1
   129|
   130|
   131|def test_hedge_and_instrument_status_unknown_alerts() -> None:
   132|    conn = setup_conn()
   133|    ensure_instrument_metadata_quality(conn, instrument_id='inst1')
   134|    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='hedge_status_unknown' AND status='active'").fetchone()['c'] == 1
   135|    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='instrument_status_unknown' AND status='active'").fetchone()['c'] == 1
   136|
   137|
   138|def test_chf_hedged_candidate_remains_proposed_until_confirmed() -> None:
   139|    conn = setup_conn()
   140|    provider = StaticLookupProvider({'CH0000000001': [ProviderCandidate(provider='stooq', provider_symbol='GETF.SW', exchange='SWX', currency='CHF', name='Global Equity ETF CHF Hedged', isin='CH0000000001', instrument_status='active')]})
   141|    generate_mapping_candidates(conn, provider=provider)
   142|    row = conn.execute("SELECT candidate_is_hedged,candidate_hedged_to_currency,review_status FROM instrument_price_mapping_candidates").fetchone()
   143|    inst = conn.execute("SELECT hedge_status FROM instruments WHERE instrument_id='inst1'").fetchone()
   144|    assert row['candidate_is_hedged'] == 1
   145|    assert row['candidate_hedged_to_currency'] == 'CHF'
   146|    assert row['review_status'] == 'proposed'
   147|    assert inst['hedge_status'] == 'unknown'
   148|
   149|
   150|def test_provider_lookup_error_does_not_crash() -> None:
   151|    conn = setup_conn()
   152|    result = generate_mapping_candidates(conn, provider=StaticLookupProvider(fail=True))
   153|    assert result.provider_lookup_unavailable == 1
   154|    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='provider_lookup_unavailable' AND status='active'").fetchone()['c'] == 1
   155|
   156|
   157|def test_mapping_selection_writes_audit_and_resolves_missing_provider_symbol() -> None:
   158|    conn = setup_conn()
   159|    conn.execute("INSERT INTO alerts(alert_id,priority,category,entity_type,entity_id,rule_id,message,status,created_at) VALUES('mps','warnung','market_data','instrument','inst1','missing_provider_symbol','missing','active','now')")
   160|    provider = StaticLookupProvider({'CH0000000001': [ProviderCandidate(provider='stooq', provider_symbol='GETF.SW', exchange='SWX', currency='USD', isin='CH0000000001', instrument_status='active')]})
   161|    generate_mapping_candidates(conn, provider=provider)
   162|    candidate_id = conn.execute("SELECT candidate_id FROM instrument_price_mapping_candidates").fetchone()['candidate_id']
   163|    mapping_id = select_mapping_candidate(conn, candidate_id=candidate_id, note='manual confirmation')
   164|    assert mapping_id
   165|    assert conn.execute("SELECT COUNT(*) AS c FROM audit_log WHERE action IN ('confirm_instrument_price_mapping','select_mapping_candidate')").fetchone()['c'] >= 2
   166|    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='missing_provider_symbol' AND status='active'").fetchone()['c'] == 0
   167|
   168|
   169|def test_fx_override_requires_note_and_resolves_missing_fx() -> None:
   170|    conn = setup_conn()
   171|    conn.execute("INSERT INTO alerts(alert_id,priority,category,entity_type,entity_id,rule_id,message,status,created_at) VALUES('fx1','kritisch','fx','instrument','inst1','missing_fx','missing','active','now')")
   172|    with pytest.raises(ValueError):
   173|        set_manual_fx_override(conn, base_currency='USD', quote_currency='CHF', rate_date='2025-12-31', rate=Decimal('0.9'), note='')
   174|    set_manual_fx_override(conn, base_currency='USD', quote_currency='CHF', rate_date='2025-12-31', rate=Decimal('0.9'), note='statement source')
   175|    assert conn.execute("SELECT COUNT(*) AS c FROM audit_log WHERE action='manual_fx_override'").fetchone()['c'] == 1
   176|    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='missing_fx' AND status='active'").fetchone()['c'] == 0
   177|
   178|
   179|def test_market_price_dry_run_without_confirmed_mapping_does_not_call_provider() -> None:
   180|    conn = setup_conn()
   181|    result = refresh_market_prices(conn, provider=RaisingPriceProvider(), asset_class='etf', limit=3, dry_run=True)
   182|    assert result.total_mappings == 0
   183|    assert result.updated_count == 0
   184|
   185|
   186|def test_dashboard_review_ui_renders_without_live_api() -> None:
   187|    conn = setup_conn()
   188|    page_13_equity_manage.render(FakeSt(), conn)
   189|


Explanation:
**Managing tests and failures**

I need to add tests and also run them to see if they fail. It's important to manage this process so I can catch any issues before they become bigger problems. I'll want to make sure everything is functioning correctly and take notes on any failures I encounter. That way, I can improve over time and create a more reliable system. Let’s make sure the tests are as thorough as possible!

PYTHONPATH=src pytest -q tests/unit/test_true_wealth_mapping_candidates_c3.py tests/unit/test_schema.py


Output:
...............                                                          [100%]
15 passed in 0.33s


     1|from __future__ import annotations
     2|
     3|import hashlib
     4|from datetime import datetime, timezone
     5|from sqlite3 import Connection
     6|
     7|from .schema import INITIAL_SCHEMA_SQL
     8|
     9|MIGRATION_VERSION = 13
    10|MIGRATION_NAME = "013_public_instrument_lookup_candidates"
    11|
    12|INSTRUMENT_OPTIONAL_COLUMNS = {
    13|    "position_category": "TEXT",
    14|    "ter": "TEXT",
    15|    "distribution_policy": "TEXT",
    16|    "index_name": "TEXT",
    17|    "fund_domicile": "TEXT",
    18|    "benchmark": "TEXT",
    19|    "is_currency_hedged": "INTEGER NOT NULL DEFAULT 0",
    20|    "hedged_to_currency": "TEXT",
    21|    "hedge_status": "TEXT NOT NULL DEFAULT 'unknown'",
    22|    "base_exposure_currency": "TEXT",
    23|    "trading_currency": "TEXT",
    24|    "instrument_status": "TEXT NOT NULL DEFAULT 'unknown'",
    25|    "valuation_policy": "TEXT NOT NULL DEFAULT 'live_price'",
    26|    "corporate_action_status": "TEXT NOT NULL DEFAULT 'not_checked'",
    27|    "split_or_corporate_action_review_required": "INTEGER NOT NULL DEFAULT 0",
    28|}
    29|
    30|TEXT_AFFINITY_COLUMNS = {
    31|    "transactions": {"quantity", "price_original", "gross_amount_original", "fee_original", "tax_original", "net_amount_original", "fx_rate_to_chf", "gross_amount_chf", "fee_chf", "tax_chf", "net_amount_chf"},
    32|    "crypto_holdings": {"quantity", "legacy_snapshot_value_original", "legacy_snapshot_value_chf"},
    33|    "crypto_transactions": {"quantity", "price_original", "gross_amount_original", "fee_quantity", "fee_original", "fx_rate_to_chf", "amount_chf"},
    34|    "crypto_prices": {"price", "market_cap", "volume_24h", "change_24h_pct"},
    35|    "fx_rates": {"rate"},
    36|    "market_prices": {"open", "high", "low", "close", "adjusted_close"},
    37|}
    38|
    39|
    40|def utc_now() -> str:
    41|    return datetime.now(timezone.utc).isoformat()
    42|
    43|
    44|def checksum_sql(sql: str) -> str:
    45|    return hashlib.sha256(sql.encode("utf-8")).hexdigest()
    46|
    47|
    48|def get_schema_version(conn: Connection) -> int:
    49|    row = conn.execute("SELECT MAX(version) AS version FROM schema_migrations").fetchone()
    50|    return int(row["version"] or 0) if row else 0
    51|
    52|
    53|def _table_columns(conn: Connection, table: str) -> dict[str, str]:
    54|    return {row["name"]: (row["type"] or "") for row in conn.execute(f"PRAGMA table_info({table})").fetchall()}
    55|
    56|
    57|def _add_missing_instrument_columns(conn: Connection) -> None:
    58|    existing = _table_columns(conn, "instruments")
    59|    for name, col_type in INSTRUMENT_OPTIONAL_COLUMNS.items():
    60|        if name not in existing:
    61|            conn.execute(f"ALTER TABLE instruments ADD COLUMN {name} {col_type}")
    62|
    63|
    64|def _rebuild_table_with_text_columns(conn: Connection, table: str, text_columns: set[str]) -> None:
    65|    cols = conn.execute(f"PRAGMA table_info({table})").fetchall()
    66|    if not cols:
    67|        return
    68|    needs_rebuild = any(row["name"] in text_columns and (row["type"] or "").upper() != "TEXT" for row in cols)
    69|    if not needs_rebuild:
    70|        return
    71|    tmp = f"{table}__text_migration"
    72|    col_defs: list[str] = []
    73|    pk_cols = [row["name"] for row in cols if row["pk"]]
    74|    for row in cols:
    75|        name = row["name"]
    76|        col_type = "TEXT" if name in text_columns else (row["type"] or "TEXT")
    77|        parts = [name, col_type]
    78|        if row["pk"] and len(pk_cols) == 1:
    79|            parts.append("PRIMARY KEY")
    80|        if row["notnull"]:
    81|            parts.append("NOT NULL")
    82|        if row["dflt_value"] is not None:
    83|            parts.append(f"DEFAULT {row['dflt_value']}")
    84|        col_defs.append(" ".join(parts))
    85|    if len(pk_cols) > 1:
    86|        col_defs.append("PRIMARY KEY (" + ", ".join(pk_cols) + ")")
    87|    conn.execute(f"DROP TABLE IF EXISTS {tmp}")
    88|    conn.execute(f"CREATE TABLE {tmp} (" + ", ".join(col_defs) + ")")
    89|    names = [row["name"] for row in cols]
    90|    select_exprs = [f"CAST({name} AS TEXT)" if name in text_columns else name for name in names]
    91|    conn.execute(f"INSERT INTO {tmp} ({', '.join(names)}) SELECT {', '.join(select_exprs)} FROM {table}")
    92|    conn.execute(f"DROP TABLE {table}")
    93|    conn.execute(f"ALTER TABLE {tmp} RENAME TO {table}")
    94|    if table == "crypto_holdings":
    95|        conn.execute("CREATE UNIQUE INDEX IF NOT EXISTS idx_crypto_holdings_asset_wallet ON crypto_holdings(asset_id, wallet_id)")
    96|
    97|
    98|def _add_missing_columns(conn: Connection, table: str, columns: dict[str, str]) -> None:
    99|    existing = _table_columns(conn, table)
   100|    for name, col_type in columns.items():
   101|        if name not in existing:
   102|            conn.execute(f"ALTER TABLE {table} ADD COLUMN {name} {col_type}")
   103|
   104|
   105|def _create_broker_bank_mapping_tables(conn: Connection) -> None:
   106|    conn.executescript(
   107|        """
   108|        CREATE TABLE IF NOT EXISTS instrument_mappings (
   109|            mapping_id TEXT PRIMARY KEY,
   110|            source_name TEXT,
   111|            source_platform TEXT NOT NULL,
   112|            source_account TEXT,
   113|            source_label TEXT NOT NULL,
   114|            normalized_name TEXT NOT NULL,
   115|            isin TEXT,
   116|            ticker TEXT,
   117|            exchange TEXT,
   118|            currency TEXT,
   119|            asset_class TEXT NOT NULL DEFAULT 'other',
   120|            source_instrument_name TEXT,
   121|

Use offset=121 to continue reading (showing 1-120 of 451 lines)


Explanation:
**Modifying tests**

I need to tweak my approach to the tests. I'll focus on using the write append method because it seems like the right fit for what I'm trying to achieve. It's always interesting to explore how small adjustments can lead to better results. I wonder if there are specific details I should pay more attention to in the tests to ensure everything runs smoothly. Let me see how this modification plays out!

def test_missing_provider_symbol_dedupes_per_instrument_only() -> None:
    conn = setup_conn()
    conn.execute("INSERT INTO alerts(alert_id,priority,category,entity_type,entity_id,rule_id,message,status,created_at) VALUES('mps1','warnung','market_data','instrument','inst1','missing_provider_symbol','missing','active','now')")
    conn.execute("INSERT INTO alerts(alert_id,priority,category,entity_type,entity_id,rule_id,message,status,created_at) VALUES('mps2','warnung','market_data','instrument','inst1','missing_provider_symbol','missing','active','now')")
    resolved = dedupe_missing_provider_symbol_alerts(conn)
    assert resolved == 1
    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='missing_provider_symbol' AND status='active'").fetchone()['c'] == 1

def test_mapping_template_contains_candidate_fields(tmp_path) -> None:
    conn = setup_conn()
    provider = StaticLookupProvider({'CH0000000001': [ProviderCandidate(provider='stooq', provider_symbol='GETF.SW', exchange='SWX', currency='USD', isin='CH0000000001')]})
    generate_mapping_candidates(conn, provider=provider)
    out = tmp_path / 'mapping_template.csv'
    summary = export_mapping_review_template(conn, output_path=out)
    text = out.read_text(encoding='utf-8')
    assert summary['instrument_count'] == 1
    assert summary['candidate_rows'] == 1
    assert 'candidate_provider,candidate_provider_symbol,candidate_exchange,candidate_currency,confidence,hedge_status_candidate,instrument_status_candidate,evidence_source' in text
    assert 'reviewer_decision,reviewer_note' in text

def test_data_quality_center_shows_candidate_review_need() -> None:
    conn = setup_conn()
    provider = StaticLookupProvider({'CH0000000001': [
        ProviderCandidate(provider='stooq', provider_symbol='GETF.SW', exchange='SWX', isin='CH0000000001'),
        ProviderCandidate(provider='yahoo', provider_symbol='GETF.SW', exchange='SWX', isin='CH0000000001'),
    ]})
    generate_mapping_candidates(conn, provider=provider)
    checks = {row['check']: row for row in data.get_data_quality(conn)}
    assert checks['Mapping-Kandidaten unbestätigt']['count'] == '2'
    assert checks['Instrumente mit mehreren Mapping-Kandidaten']['count'] == '1'
    assert checks['ambiguous provider mapping']['count'] == '1'

def test_missing_provider_symbol_not_resolved_until_candidate_confirmed() -> None:
    conn = setup_conn()
    generate_mapping_candidates(conn, provider=StaticLookupProvider({}))
    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='missing_provider_symbol' AND status='active'").fetchone()['c'] == 1
    provider = StaticLookupProvider({'CH0000000001': [ProviderCandidate(provider='stooq', provider_symbol='GETF.SW', exchange='SWX', currency='USD', isin='CH0000000001')]})
    generate_mapping_candidates(conn, provider=provider)
    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='missing_provider_symbol' AND status='active'").fetchone()['c'] == 1
    candidate_id = conn.execute("SELECT candidate_id FROM instrument_price_mapping_candidates").fetchone()['candidate_id']
    select_mapping_candidate(conn, candidate_id=candidate_id, note='manual confirmation')
    assert conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE rule_id='missing_provider_symbol' AND status='active'").fetchone()['c'] == 0


No linter for .txt files


python - <<'PY'
from pathlib import Path
p=Path('tests/unit/test_true_wealth_mapping_candidates_c3.py')
p.write_text(p.read_text()+Path('/tmp/new_tests.txt').read_text(), encoding='utf-8')
PY


Output:
...................                                                      [100%]
19 passed in 0.41s


     1|from __future__ import annotations
     2|
     3|import os
     4|from dataclasses import dataclass
     5|from pathlib import Path
     6|
     7|from .paths import DEFAULT_RUNTIME_DIR, RuntimePaths, is_relative_to
     8|
     9|
    10|@dataclass(frozen=True)
    11|class Settings:
    12|    env: str = "local"
    13|    base_currency: str = "CHF"
    14|    timezone: str = "Europe/Zurich"
    15|    runtime_paths: RuntimePaths = RuntimePaths.from_base(DEFAULT_RUNTIME_DIR)
    16|    log_level: str = "INFO"
    17|    block_runtime_inside_repo: bool = True
    18|
    19|    @property
    20|    def db_path(self) -> Path:
    21|        return self.runtime_paths.db_path
    22|
    23|
    24|def find_repo_root(start: Path | None = None) -> Path:
    25|    current = (start or Path.cwd()).resolve()
    26|    for candidate in [current, *current.parents]:
    27|        if (candidate / ".git").exists() or (candidate / "pyproject.toml").exists():
    28|            return candidate
    29|    return current
    30|
    31|
    32|def load_settings(repo_root: Path | None = None, environ: dict[str, str] | None = None) -> Settings:
    33|    env = environ or os.environ
    34|    runtime_dir = Path(env.get("JARVIS_FINANCE_RUNTIME_DIR", str(DEFAULT_RUNTIME_DIR))).expanduser()
    35|    paths = RuntimePaths.from_base(runtime_dir)
    36|    db_override = env.get("JARVIS_FINANCE_DB_PATH")
    37|    if db_override:
    38|        object.__setattr__(paths, "db_path", Path(db_override).expanduser().resolve())
    39|    settings = Settings(
    40|        env=env.get("JARVIS_FINANCE_ENV", "local"),
    41|        base_currency=env.get("JARVIS_FINANCE_BASE_CURRENCY", "CHF"),
    42|        timezone=env.get("JARVIS_FINANCE_TIMEZONE", "Europe/Zurich"),
    43|        runtime_paths=paths,
    44|        log_level=env.get("JARVIS_FINANCE_LOG_LEVEL", "INFO"),
    45|        block_runtime_inside_repo=env.get("JARVIS_FINANCE_BLOCK_RUNTIME_INSIDE_REPO", "1") not in {"0", "false", "False"},
    46|    )
    47|    validate_settings(settings, repo_root=repo_root)
    48|    return settings
    49|
    50|
    51|def validate_settings(settings: Settings, repo_root: Path | None = None) -> None:
    52|    root = (repo_root or find_repo_root()).resolve()
    53|    if settings.block_runtime_inside_repo and is_relative_to(settings.runtime_paths.base_dir, root):
    54|        raise ValueError(f"Runtime directory must be outside the Git repository: {settings.runtime_paths.base_dir}")
    55|    if settings.base_currency != "CHF":
    56|        raise ValueError("MVP 1 base currency must be CHF")
    57|
    58|
    59|def ensure_runtime_dirs(settings: Settings) -> None:
    60|    settings.runtime_paths.ensure_dirs()
    61|


     1|from __future__ import annotations
     2|
     3|from dataclasses import dataclass
     4|from pathlib import Path
     5|
     6|DEFAULT_RUNTIME_DIR = Path("~/jarvis_runtime/finance-system").expanduser()
     7|
     8|
     9|@dataclass(frozen=True)
    10|class RuntimePaths:
    11|    base_dir: Path
    12|    data_dir: Path
    13|    imports_dir: Path
    14|    exports_dir: Path
    15|    reports_dir: Path
    16|    backups_dir: Path
    17|    secrets_dir: Path
    18|    logs_dir: Path
    19|    db_path: Path
    20|
    21|    @classmethod
    22|    def from_base(cls, base_dir: str | Path = DEFAULT_RUNTIME_DIR) -> "RuntimePaths":
    23|        base = Path(base_dir).expanduser().resolve()
    24|        return cls(
    25|            base_dir=base,
    26|            data_dir=base / "data",
    27|            imports_dir=base / "imports",
    28|            exports_dir=base / "exports",
    29|            reports_dir=base / "reports",
    30|            backups_dir=base / "backups",
    31|            secrets_dir=base / "secrets",
    32|            logs_dir=base / "logs",
    33|            db_path=base / "data" / "finance.sqlite3",
    34|        )
    35|
    36|    def ensure_dirs(self) -> None:
    37|        for path in [
    38|            self.data_dir,
    39|            self.imports_dir,
    40|            self.exports_dir,
    41|            self.reports_dir,
    42|            self.backups_dir,
    43|            self.secrets_dir,
    44|            self.logs_dir,
    45|        ]:
    46|            path.mkdir(parents=True, exist_ok=True)
    47|
    48|
    49|def is_relative_to(path: Path, parent: Path) -> bool:
    50|    try:
    51|        path.resolve().relative_to(parent.resolve())
    52|        return True
    53|    except ValueError:
    54|        return False
    55|


patch

class OpenFIGILookupProvider:
    """Public ISIN-to-instrument metadata lookup via OpenFIGI; no prices or portfolio data."""

    name = "openfigi"

    def __init__(self, *, timeout_seconds: float = 8.0):
        self.timeout_seconds = timeout_seconds

    def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
        if not isin:
            return []
        token = os.environ.get("OPENFIGI_API_KEY")
        headers = {"Content-Type": "application/json", "User-Agent": "JarvisFinance/1.0 public-instrument-lookup"}
        if token:
            headers["X-OPENFIGI-APIKEY"] = token
        payload = json.dumps([{"idType": "ID_ISIN", "idValue": isin}]).encode("utf-8")
        req = urllib.request.Request("https://api.openfigi.com/v3/mapping", data=payload, headers=headers, method="POST")
        try:
            with urllib.request.urlopen(req, timeout=self.timeout_seconds) as resp:
                data = json.loads(resp.read().decode("utf-8"))
        except Exception as exc:  # pragma: no cover - network dependent
            raise LookupUnavailable(str(exc)) from exc
        out: list[ProviderCandidate] = []
        for item in (data[0].get("data", []) if data else []):
            symbol = item.get("ticker") or item.get("figi")
            if not symbol:
                continue
            sec_type = " ".join(str(item.get(k) or "") for k in ("securityType", "securityType2", "marketSector")).lower()
            asset_class = "etf" if "fund" in sec_type or "etf" in sec_type else "equity" if "equity" in sec_type else None
            out.append(ProviderCandidate(
                provider=self.name,
                provider_symbol=symbol,
                exchange=item.get("exchCode") or item.get("marketSector"),
                currency=item.get("currency") or currency,
                name=item.get("name"),
                isin=isin,
                asset_class=asset_class,
                hedge_status="unknown",
                instrument_status="unknown",
                valuation_policy="live_price",
                evidence_source="openfigi_isin_mapping",
                evidence_note="Public ISIN mapping metadata only; no prices, holdings, account data, or portfolio quantities sent.",
            ))
        return out

class YahooSearchLookupProvider:
    """Public metadata lookup via Yahoo Finance search endpoint; no prices or portfolio data."""


--- a//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
+++ b//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
@@ -53,6 +53,52 @@

     def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
         return []
+
+
+class OpenFIGILookupProvider:
+    """Public ISIN-to-instrument metadata lookup via OpenFIGI; no prices or portfolio data."""
+
+    name = "openfigi"
+
+    def __init__(self, *, timeout_seconds: float = 8.0):
+        self.timeout_seconds = timeout_seconds
+
+    def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
+        if not isin:
+            return []
+        token = os.environ.get("OPENFIGI_API_KEY")
+        headers = {"Content-Type": "application/json", "User-Agent": "JarvisFinance/1.0 public-instrument-lookup"}
+        if token:
+            headers["X-OPENFIGI-APIKEY"] = token
+        payload = json.dumps([{"idType": "ID_ISIN", "idValue": isin}]).encode("utf-8")
+        req = urllib.request.Request("https://api.openfigi.com/v3/mapping", data=payload, headers=headers, method="POST")
+        try:
+            with urllib.request.urlopen(req, timeout=self.timeout_seconds) as resp:
+                data = json.loads(resp.read().decode("utf-8"))
+        except Exception as exc:  # pragma: no cover - network dependent
+            raise LookupUnavailable(str(exc)) from exc
+        out: list[ProviderCandidate] = []
+        for item in (data[0].get("data", []) if data else []):
+            symbol = item.get("ticker") or item.get("figi")
+            if not symbol:
+                continue
+            sec_type = " ".join(str(item.get(k) or "") for k in ("securityType", "securityType2", "marketSector")).lower()
+            asset_class = "etf" if "fund" in sec_type or "etf" in sec_type else "equity" if "equity" in sec_type else None
+            out.append(ProviderCandidate(
+                provider=self.name,
+                provider_symbol=symbol,
+                exchange=item.get("exchCode") or item.get("marketSector"),
+                currency=item.get("currency") or currency,
+                name=item.get("name"),
+                isin=isin,
+                asset_class=asset_class,
+                hedge_status="unknown",
+                instrument_status="unknown",
+                valuation_policy="live_price",
+                evidence_source="openfigi_isin_mapping",
+                evidence_note="Public ISIN mapping metadata only; no prices, holdings, account data, or portfolio quantities sent.",
+            ))
+        return out

 class YahooSearchLookupProvider:


                    isin=None,
                    asset_class=asset_class,

                    isin=isin if isin and query == isin else None,
                    asset_class=asset_class,


--- a//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
+++ b//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
@@ -136,7 +136,7 @@
                     exchange=exch,
                     currency=currency,
                     name=quote.get("longname") or quote.get("shortname"),
-                    isin=isin if isin and query == isin else None,
+                    isin=None,
                     asset_class=asset_class,
                     hedge_status="unknown",
                     instrument_status="unknown",


self.providers = providers or [OpenFIGILookupProvider(), YahooSearchLookupProvider()]

self.providers = providers or [YahooSearchLookupProvider()]


--- a//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
+++ b//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
@@ -152,7 +152,7 @@
     name = "public_composite"

     def __init__(self, providers: list[InstrumentLookupProvider] | None = None):
-        self.providers = providers or [YahooSearchLookupProvider()]
+        self.providers = providers or [OpenFIGILookupProvider(), YahooSearchLookupProvider()]

     def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
         candidates: list[ProviderCandidate] = []


Explanation:
**Figuring out tests**

I think I need to add some tests to compile things properly. It’s crucial to ensure everything works as it should, right? I wonder if there are specific tests I should focus on first. I could start with the most critical functions or elements and then work my way through the less essential ones. Testing might help catch any potential issues early, which would be super helpful! Let's get started on that!


terminal

Output:
...................                                                      [100%]
19 passed in 0.39s


Explanation:
**Testing and updating tasks**

I need to run full tests later and update my to-do list. But first, I should start with a runtime run. It seems there’s a need for “no portfolio data to external.” The CLI public lookup provides ISIN, ticker, and name, which includes metadata from the DB, though not the quantity. I think I need to query before proceeding. Oh, and I should check the active counts for any generated alerts. I'll use terminal Python to aggregate that information.

PYTHONPATH=src python - <<'PY'
from collections import Counter, defaultdict
from jarvis_finance.config.settings import load_settings
from jarvis_finance.storage.database import connect
from jarvis_finance.storage.migrations import apply_migrations
settings=load_settings(); conn=connect(settings.db_path); apply_migrations(conn)
rows=conn.execute("SELECT alert_id,entity_type,entity_id,fingerprint FROM alerts WHERE status='active' AND rule_id='missing_provider_symbol'").fetchall()
print({'active_missing_provider_symbol': len(rows), 'by_entity_type': dict(Counter(r['entity_type'] for r in rows)), 'duplicate_entities': sum(1 for k,v in Counter((r['entity_type'], r['entity_id']) for r in rows).items() if v>1)})
conn.close()
PY


Output: {'active_missing_provider_symbol': 6, 'by_entity_type': {'instrument': 6}, 'duplicate_entities': 3}


Explanation:
**Running CLI for backup**

I need to run the CLI with the dedupe public lookup export. Before doing that, maybe I should back up first? That feels like a safe approach. Then, I could apply the necessary migration writes. I’m also contemplating running a backup of the runtime database since it’s crucial for real-time writing. Using the CLI backup seems like the right move to ensure everything is secure!

PYTHONPATH=src python -m jarvis_finance.cli.main backup-runtime-db

PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .


Output:
.pytest_cache/CACHEDIR.TAG: runtime_cache_file_blocked
.pytest_cache/README.md: runtime_cache_file_blocked
.pytest_cache/.gitignore: runtime_cache_file_blocked
.pytest_cache/v/cache/nodeids: runtime_cache_file_blocked


rm -rf .pytest_cache && find . -type d -name __pycache__ -prune -exec rm -rf {} + && PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .


PYTHONPATH=src python -m jarvis_finance.cli.main generate-mapping-candidates --source true_wealth --limit 3 --public-lookup --dedupe-missing-provider-alerts --export-template


Output: MAPPING_CANDIDATES instruments_total=3 candidates_found=336 high=315 medium=18 low=3 instruments_without_candidate=0 hedge_unknown=3 instrument_status_unknown=3 provider_lookup_unavailable=0 ambiguous=3 deduped_missing_provider_alerts=3 template_updated=true template_instruments=3 template_candidates=329 dry_run=false


Explanation:
**Evaluating candidate limits**

I'm noticing we have 336 candidates due to excessive OpenFIGI returns, which seems like too many. I wonder if we should limit the number of results per provider. Maybe capping OpenFIGI to a max of 20 results could help, but the current runtime has already exceeded that. How about implementing a cleanup before generating new candidates? An optional generator could be useful to prune prior candidates as well. Overall, we definitely need to manage this more effectively.
**Planning cleanup and adjustments**

I’m considering doing a manual cleanup before rerunning the process, while also setting a default cap for OpenFIGI results. I'm wondering if a max of 8 results is enough, but since one ISIN can relate to multiple exchanges, I think 12 would be better. After that, I could perform a runtime cleanup by deleting unconfirmed candidates for the three instruments and rerun the process. I should also run query counts after the cleanup since it seems like there's a high candidate distribution.

patch

    def __init__(self, *, timeout_seconds: float = 8.0, max_results: int = 12):
        self.timeout_seconds = timeout_seconds
        self.max_results = max_results

    def __init__(self, *, timeout_seconds: float = 8.0):
        self.timeout_seconds = timeout_seconds


--- a//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
+++ b//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
@@ -60,8 +60,9 @@

     name = "openfigi"

-    def __init__(self, *, timeout_seconds: float = 8.0):
+    def __init__(self, *, timeout_seconds: float = 8.0, max_results: int = 12):
         self.timeout_seconds = timeout_seconds
+        self.max_results = max_results

     def lookup(self, *, isin: str | None, name: str | None, ticker: str | None, exchange: str | None, currency: str | None) -> list[ProviderCandidate]:
         if not isin:


for item in (data[0].get("data", []) if data else [])[: self.max_results]:

for item in (data[0].get("data", []) if data else []):


--- a//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
+++ b//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/market_data/candidates.py
@@ -79,7 +79,7 @@
         except Exception as exc:  # pragma: no cover - network dependent
             raise LookupUnavailable(str(exc)) from exc
         out: list[ProviderCandidate] = []
-        for item in (data[0].get("data", []) if data else []):
+        for item in (data[0].get("data", []) if data else [])[: self.max_results]:
             symbol = item.get("ticker") or item.get("figi")
             if not symbol:
                 continue