# FinanceManager Backlog

This backlog records approved future modules that are intentionally not built in the current MVP hardening block.

## Near-term sequence

1. Broker/bank structural mapping and dry-run importers.
2. Equity/ETF initial snapshot dry run.
3. Productive equity/ETF import only after explicit approval and verified runtime backup.
4. Equity/ETF market data + FX.
5. Gesamtportfolio dashboard with visual analytics.
6. Data Quality Center.
7. Watchlist / Decision Journal.
8. Scores / Rebalancing.
9. Backtesting / Monte Carlo / Projected Return.

## Portfolio Health page

- Total value in CHF.
- Asset classes.
- Platforms.
- Data quality.
- Cash quota.
- Missing prices and FX.

## Reconciliation / Abgleich page

- Dashboard holdings vs. last snapshot.
- Manual review marker: holding checked on date X.
- Make differences visible.

## Data Quality Center

- Missing prices.
- Stale prices.
- Missing FX.
- Missing ISIN.
- Missing CoinGecko ID.
- Incomplete history.
- Old/unresolved alerts.
- Import problems.
- Button: Recheck.
- Button: Resolve fixed alerts.

## Manual Review Queue

- Skipped coins.
- Uncertain crypto mappings.
- Missing stock/ETF mappings.
- Import rows with errors.
- Open alerts.
- Open data-quality problems.
- Manual decision capture.

## Backup / Restore

- Runtime DB backup.
- Runtime DB restore with explicit confirmation.
- Backup verification with SHA256 checksum.
- Optional encrypted backup.
- Restore test before broader productive imports.

## Safe Mode / Read-only Mode

- Dashboard read-only by default.
- Explicit enablement for write actions.
- Review/confirm before every write.
- Visible environment indicator: `Live Runtime DB` vs. `Demo DB`.
- Guard rails against accidental changes.

## Command Center UX

The Command Center should immediately show:

- Total value CHF.
- Overall data quality.
- Critical alerts.
- Missing prices/FX.
- Largest movements.
- Portfolio allocation.
- Open decisions.
- Report buttons.
- Last backup timestamp.

## Import Wizard UX

For real broker data:

1. Select file.
2. Select source.
3. Review mapping.
4. Dry run.
5. Show warnings.
6. Create/verify backup.
7. Explicit commit confirmation.
8. Show result and audit summary.

## Visual Analytics Layer

### Portfolio Heatmap / Treemap

- Tile size = portfolio weight.
- Color = performance, risk score or data quality.
- Click on asset opens detail view.

### Allocation charts

- Asset class.
- Platform/depot.
- Currency.
- Country/region.
- Sector.
- Core vs. opportunity.
- Crypto vs. traditional assets.

### Contribution Analysis

- Contribution of each asset to total gain/loss in CHF.
- Contribution by asset class.
- Contribution by platform.
- Separate dividend/distribution contribution.

### Currency Exposure

- Direct currency per position.
- USD/EUR/CHF shares.
- FX P&L separately.
- Warning for high USD/EUR concentration.

### Performance Waterfall

- Start value.
- Deposits/withdrawals.
- Price gain/loss.
- Dividends/distributions.
- FX effect.
- Fees/taxes.
- End value.

## Risk & Correlation Tools

- Correlation matrix.
- Rolling correlation.
- Volatility.
- Maximum drawdown.
- Drawdown chart.
- Top-5 risk contributions.
- Concentration score.
- Platform/broker risk.
- Crypto share and crypto drawdown.
- ETF look-through later.

## Benchmark / Backtesting

- Portfolio vs. MSCI World.
- Portfolio vs. S&P 500.
- Portfolio vs. Nasdaq 100.
- Portfolio vs. BTC.
- Portfolio vs. own blended benchmark, e.g. 80/20.
- Relative strength vs. benchmark.
- Rolling returns.
- Backtest current allocation.
- Backtest scoring rules only when historical data is sufficient.

## Projection / Monte Carlo

Clearly label as simulation, not forecast.

- Monte Carlo simulation based on historical return/volatility.
- Scenarios:
  - conservative
  - neutral
  - optimistic
- Output:
  - range of possible outcomes
  - probability of reaching target amount
  - drawdown risk
  - transparent assumptions
- No language implying guaranteed returns or AI prediction.

## Scenario Stress Tests

- CHF +10%.
- USD/CHF -10%.
- Equity market -20%.
- Nasdaq -30%.
- Crypto -40%.
- Interest-rate increase.
- Inflation shock.
- Risk-off combined scenario.
- Single position -30%.

Output:

- Impact in CHF.
- Impact per asset class.
- Impact per platform.
- Cash quota after scenario.
- Limit breaches.

## Watchlist and Decision Journal

- Watchlist pipeline.
- Target prices.
- Triggers.
- Investment case.
- Bear case.
- Review date.
- Transfer to portfolio.
- Decision Journal as mandatory option for buy/sell/override.
- Obsidian Markdown export:
  - one Markdown file per decision
  - frontmatter with asset, date, action, reason, risk, review date
  - no sensitive values unless explicitly allowed.

## Local LLM Integration

Later only, never for final accounting/tax/recommendation decisions.

- Local news summaries.
- Local PDF summaries of bank documents.
- Bear-case drafts.
- Report-text drafts.
- Never final bookkeeping, FX, tax values or binding recommendations.

## Schweizer Income-/Tax-Kontrollmodul

Control module only; broker tax documents remain authoritative.

- Gross/net dividends.
- Swiss withholding tax 35%.
- Foreign withholding tax.
- Distributions per year.
- Hinweis: Broker-Steuerunterlagen bleiben maßgeblich.
- Not a complete tax dossier replacement.

## Eye Health / Low-Strain Mode

- Dark low-strain theme.
- High contrast.
- Few glaring colors.
- Clear traffic-light colors.
- Optional larger font.
- Suitable for longer analysis sessions.

Explanation:
**Updating backlog and UX**

I'm thinking about how to tackle the backlog and UX updates. Should I apply a patch or write a complete update from scratch? Maybe it makes more sense to extend the existing file to achieve what I want. I’m weighing my options here and wondering which approach would be the most efficient and effective. It’s important to keep everything organized while ensuring any updates enhance the user experience!


No linter for .md files


terminal

set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
PYTHONPATH=src python -m compileall -q src tests
PYTHONPATH=src pytest -q
rm -rf .pytest_cache src/jarvis_finance/**/__pycache__ tests/**/__pycache__
PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .
git status --short --branch

Explanation:
**Considering migration and tests**

I need to focus on running tests and ensuring git safety. It might be good to commit and push for stages 4 and 5, even though the user didn't require it after stage 4. There was also a pycache issue from a previous command that I should remove. Migrating the new version could potentially impact the backup test. I need to run compile pytest and make sure everything is set before applying any migrations.


Output:
........................................................................ [ 58%]
...................................................                      [100%]
123 passed in 3.81s
GIT_SAFETY_OK
## main...origin/main
 M docs/finance-backlog.md
 M src/jarvis_finance/storage/migrations.py
 M src/jarvis_finance/storage/schema.py
 M tests/unit/test_schema.py
?? docs/broker-bank-mapping-plan.md


set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
git add docs/finance-backlog.md docs/broker-bank-mapping-plan.md src/jarvis_finance/storage/migrations.py src/jarvis_finance/storage/schema.py tests/unit/test_schema.py
git commit -m "docs: prepare broker bank mapping backlog"

Explanation:
**Figuring out database migration**

I need to commit and push, but first, there's a potential migration issue to resolve. The existing database is at version 3, but version 2 is missing. For the existing database with version 2, it applies create and records 3. A fresh database would only have records 1 and 3, which seems odd since the current schema version should be 3. I guess I’ll go ahead with the commit and push, using a token.


Output:
[main 41f11d2] docs: prepare broker bank mapping backlog
 5 files changed, 424 insertions(+), 8 deletions(-)
 create mode 100644 docs/broker-bank-mapping-plan.md


set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
python - <<'PY'
import re, subprocess, sys
from pathlib import Path
repo=Path('/home/agent/.hermes/repos/FinanceManager')
text=Path.home().joinpath('jarvis_runtime/finance-system/secrets/Github_token_FinanceManager.csv').read_text()
m=re.search(r'(github_pat_[A-Za-z0-9_]+|ghp_[A-Za-z0-9_]+|gho_[A-Za-z0-9_]+)', text)
if not m:
    print('TOKEN_PARSE=failed'); sys.exit(12)
token=m.group(1); auth_url=f'https://{token}@github.com/Gamexgit/FinanceManager.git'
def run(args, check=True): return subprocess.run(args, cwd=repo, text=True, capture_output=True, check=check)
try:
    run(['git','fetch',auth_url,'main:refs/remotes/origin/main','--quiet'])
    local=run(['git','rev-parse','HEAD']).stdout.strip(); origin=run(['git','rev-parse','origin/main']).stdout.strip(); base=run(['git','merge-base','HEAD','origin/main']).stdout.strip()
    print('REMOTE_BEFORE='+origin[:7]); print('LOCAL_HEAD='+local[:7])
    if origin != base:
        print('REMOTE_HAS_NEW_COMMITS=yes'); sys.exit(3)
    push=run(['git','push',auth_url,'main:main'], check=False)
    if push.returncode:
        print('PUSH=failed'); print((push.stderr or push.stdout).replace(token,'[REDACTED]')[-1000:]); sys.exit(push.returncode)
    run(['git','fetch',auth_url,'main:refs/remotes/origin/main','--quiet'])
    print('PUSH=success'); print('REMOTE_AFTER='+run(['git','rev-parse','--short','origin/main']).stdout.strip())
finally:
    token=None; auth_url=None
PY
PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .
git status --short --branch


Output:
REMOTE_BEFORE=1e2ca9b
LOCAL_HEAD=41f11d2
PUSH=success
REMOTE_AFTER=41f11d2
GIT_SAFETY_OK
## main...origin/main


todo

Broker-/Bank-Dateien strukturell mappen und Mapping-/Dry-Run-Plan erstellen ohne produktive DB-Änderung

Backlog/UX-Konzept um Visual Analytics, Risk, Benchmarks, Projections, Scenario Tests, Watchlist/Decision Journal, Local LLM, Tax, Dashboard UX erweitern


Push der 2 lokalen Commits nach Compile, Tests, Git-Safety, Status; kein Force-Push

Phase A: Crypto Alert-Lifecycle Auto-Resolve + check-data-quality CLI + Tests + Runtime-Recheck + Commit/Push

Minimal Backup/Restore CLI + Tests + Git-Safety + Commit/Push


Erledigt, Sir.

## Status Push / Repo

- Remote HEAD: `41f11d2`
- Branch: `main`
- Git Status: sauber, `main...origin/main`
- Force-Push: nein
- Remote wurde vor Push geprüft; keine neuen Remote-Commits überschrieben.

## Schritt 1–3 abgeschlossen

### 1. Zwei alte lokale Commits gepusht

- `1b5028d` — Crypto Price Refresh Hardening
- `943c424` — Equity/ETF Management UI

Danach weitergebaut.

### 2. Crypto Alert Lifecycle

Implementiert und gepusht:

- Auto-Resolve für fixed stale/missing price alerts
- `finance check-data-quality`
- Optionen:
  - `--scope crypto`
  - `--resolve-fixed`
  - `--dry-run`
- Tests ergänzt für Alert-Lifecycle/Dedupe
- Produktiver Recheck aggregiert durchgeführt, ohne Preise/Mengen/Werte im Chat

Commit:

- `75e3a3a` — Alert Lifecycle / Data Quality Recheck

### 3. Minimal Backup / Restore

Implementiert und gepusht:

- `finance backup-runtime-db`
- `finance restore-runtime-db --from <backup-file> --yes`
- `finance verify-backup --file <backup-file>`
- Backup ausschließlich außerhalb Git im Runtime-Backups-Ordner
- Timestamp-Dateiname
- SHA256-Checksum
- Restore nur mit expliziter Bestätigung / `--yes`
- Git-Safety blockiert Backup-Dateien im Repo

Produktiver Backup-Test:

- Backup erzeugt: ja
- Backup verifiziert: ja
- Checksum OK: ja
- Speicherort: externer Runtime-Ordner, nicht Git

Commit:

- `1e2ca9b` — Runtime DB Backup/Restore

Tests:

- `122 passed`

Git-Safety:

- OK

## Schritt 4: Broker-/Bank-Dateien strukturell gemappt

Keine produktive DB-Änderung. Keine echten Werte/Mengen im Chat. Dateien wurden nur im externen Runtime-Bereich analysiert, nicht ins Repo gelegt.

Dokument erstellt:

- `docs/broker-bank-mapping-plan.md`

Zusätzlich vorbereitet:

- Schema-Fundament für Mapping/Dry-Runs:
  - `instrument_mappings`
  - `platform_account_mappings`
  - `broker_import_dry_runs`

Commit:

- `41f11d2` — Broker/Bank Mapping + Backlog

### PostFinance

- Dateityp: DOCX
- Struktur:
  - mehrere Tabellen
  - Cash-/Account-Zusammenfassungen
  - Positionstabelle vorhanden
- Felder strukturell vorhanden:
  - Produkt/Name
  - Anzahl/Menge
  - Währung
  - Markt-/Totalwert
  - Einstand-/Kostenfeld als Legacy-/Kontrollinfo
  - Cash-/Kontobereiche
- Nicht klar erkannt:
  - ISIN
  - Exchange
  - eindeutige Ticker-Metadaten
  - Gebühren/Steuern
  - sauberer Stichtag in Tabellenstruktur
- Quality Flags:
  - `missing_isin`
  - `snapshot_only`
  - `cost_basis_uncertain`
  - `market_value_legacy`
  - `needs_manual_review`

### True Wealth

- Dateityp: DOCX
- Struktur:
  - eine gemischte Tabelle mit Cash-/Konto- und positionsähnlichen Zeilen
- Felder strukturell vorhanden:
  - Cash/Balances
  - Account-/Portfolio-Kontext
  - Währung
  - Mengen-/Anteilsinformationen
  - datumähnliche Struktur
- Nicht klar erkannt:
  - ISIN
  - Ticker
  - Cost Basis
  - Gebühren/Steuern
  - sauber trennbare Market-Value-Felder
- Quality Flags:
  - `missing_isin`
  - `snapshot_only`
  - `cost_basis_uncertain`
  - `missing_fx` falls Fremdwährung ohne FX
  - `needs_manual_review`

### Raiffeisen

- Dateityp: XLSX
- Struktur:
  - ein Sheet
  - kompakte Konto-/Anlagenübersicht
- Felder strukturell vorhanden:
  - Konto-/Plattformlabels
  - Cash-/Account-Zeilen
  - aggregierte Depot-/Anlagezeilen
- Nicht klar erkannt:
  - ISIN
  - Ticker
  - Mengen
  - detaillierte Instrumentpositionen
  - Cost Basis
  - Gebühren/Steuern
- Nutzungsempfehlung:
  - vorerst Cash-/Account-/Kontrollsnapshot
  - kein produktiver Instrumentimport aus dieser Datei ohne detaillierteren Export

## Schritt 5: Backlog / UX-Konzept erweitert

Aktualisiert:

- `docs/finance-backlog.md`

Ergänzt:

- Visual Analytics Layer
  - Treemap/Heatmap
  - Allocation Charts
  - Contribution Analysis
  - Currency Exposure
  - Performance Waterfall
- Risk & Correlation Tools
- Benchmark / Backtesting
- Projection / Monte Carlo als Simulation, nicht Prognose
- Scenario Stress Tests
- Watchlist & Decision Journal inkl. Obsidian Export
- Local LLM Integration — nur Zusammenfassung/Entwurf, keine finale Buchhaltung
- Schweizer Income-/Tax-Kontrollmodul
- Command Center UX
- Safe Mode / Read-only Mode
- Data Quality Center
- Manual Review Queue
- Import Wizard
- Eye Health / Low-Strain Mode

## Final Verification

Nach den letzten Änderungen:

- Compile: OK
- Tests: `123 passed`
- Git-Safety: OK
- Git Status: sauber
- Remote HEAD: `41f11d2`

Keine echten Aktien-/ETF-Daten importiert. Keine produktive DB-Änderung für Broker/Bank-Daten. Keine echten Werte/Mengen im Chat.


[M G] Sehr gut. Der aktuelle Stand ist akzeptiert.

Die abgeschlossenen Schritte sind bestätigt:

* alte lokale Commits gepusht
* Alert-Lifecycle behoben
* Backup/Restore implementiert
* Broker-/Bank-Dateien strukturell analysiert
* Backlog/UX-Konzept erweitert
* Tests: 123 passed
* Git-Safety: OK
* Remote HEAD: 41f11d2
* keine echten Aktien-/ETF-Daten produktiv importiert

Bitte jetzt nicht mit Monte-Carlo, Backtesting, Rebalancing, Scores oder Heatmaps beginnen. Diese Module kommen später.

Der nächste sinnvolle Schritt ist:

## Phase B1 — Equity/ETF Import Wizard und Instrument Mapping

Ziel:
Wir brauchen einen sicheren Import-/Review-Prozess für PostFinance, True Wealth und Raiffeisen, bevor echte Aktien-/ETF-/Cash-Daten produktiv übernommen werden.

Die strukturelle Analyse zeigt, dass die vorhandenen Dateien nicht direkt sauber genug für einen automatischen Import sind. Vor allem fehlen oder sind unklar:

* ISIN
* Ticker
* Exchange
* eindeutige Instrument-Metadaten
* Stichtag
* Cost Basis
* Gebühren/Steuern
* detaillierte Instrumentpositionen bei Raiffeisen

Deshalb bitte jetzt einen Import Wizard und Mapping-Layer bauen, aber weiterhin keinen produktiven Import echter Aktien-/ETF-Daten durchführen.

## 1. Instrument Master / Mapping Layer

Bitte implementiere oder erweitere:

### Tabelle / Modell: instrument_mappings

Zweck:
Quellbezeichnungen aus Broker-/Bank-Dateien werden auf interne Instrumente gemappt.

Felder:

* mapping_id
* source_name
* source_platform: PostFinance, True Wealth, Raiffeisen
* source_account
* source_label
* normalized_name
* isin
* ticker
* exchange
* currency
* asset_class: equity, etf, cash, other
* instrument_id optional
* mapping_status:

  * mapped
  * needs_manual_review
  * missing_isin
  * missing_ticker
  * ambiguous
  * ignored
* confidence
* notes
* created_at
* updated_at

Regeln:

* ISIN ist für Aktien/ETFs primäre Identifikation.
* Ticker + Exchange ist Fallback.
* Name allein reicht nicht für automatisches Mapping.
* Unsichere Mappings dürfen nicht produktiv importiert werden.
* Manuelle Bestätigung muss Audit-Log erzeugen.

## 2. Platform / Account Mapping

Bitte implementiere oder erweitere:

### Tabelle / Modell: platform_account_mappings

Zweck:
Quellkonten aus Broker-/Bank-Auszügen werden auf interne platforms/accounts gemappt.

Felder:

* mapping_id
* source_platform
* source_account_label
* normalized_platform
* normalized_account_name
* internal_platform_id
* internal_account_id
* account_type:

  * brokerage
  * cash
  * robo_portfolio
  * reserve
  * other
* currency
* mapping_status
* notes
* created_at
* updated_at

Regeln:

* PostFinance, True Wealth und Raiffeisen müssen eindeutig trennbar sein.
* Cash-Konten und Depot-Konten dürfen nicht vermischt werden.
* Raiffeisen soll vorerst primär als Cash-/Kontrollsnapshot behandelt werden, falls keine detaillierten Instrumentpositionen vorhanden sind.

## 3. Broker Import Dry-Run Model

Bitte implementiere oder erweitere:

### Tabelle / Modell: broker_import_dry_runs

Zweck:
Speichert Ergebnisse von Import-Dry-Runs, ohne produktiv ins Ledger zu schreiben.

Felder:

* dry_run_id
* source_platform
* source_file_type: docx, xlsx, csv, pdf, unknown
* source_filename_hash
* detected_snapshot_date
* snapshot_date_status:

  * clear
  * inferred
  * missing
  * conflict
* rows_total
* candidate_positions
* candidate_cash_rows
* mapped_positions
* blocked_positions
* warnings_count
* errors_count
* quality_flags_json
* summary_json
* created_at

Wichtig:
Keine echten Werte oder Mengen in Git oder Docs. Dry-Run-Details mit echten Daten nur Runtime.

## 4. Import Wizard UI

Bitte baue im Dashboard eine neue Seite:

### Import Wizard

Ziel:
Ein Nutzer soll echte Broker-/Bankdateien kontrolliert prüfen können, bevor sie importiert werden.

Workflow:

1. Quelle wählen

* PostFinance
* True Wealth
* Raiffeisen
* Manuell / Sonstige

2. Datei auswählen

* Nur Runtime-/Upload-Kontext
* Keine Datei ins Repo
* Dateityp anzeigen

3. Struktur erkennen

* Tabellen/Abschnitte anzeigen
* erkannte Spalten/Feldarten
* erkannte
Kandidatenpositionen
* erkannte Cash-Zeilen
* erkannte Stichtage

4. Mapping prüfen

* Instrumente mit ISIN direkt mappen
* Instrumente ohne ISIN als needs_manual_review
* Account-/Depot-Mapping prüfen
* Currency prüfen
* Assetklasse prüfen

5. Dry Run anzeigen
   Nur aggregiert oder mit maskierbaren Details:

* Anzahl Zeilen
* Anzahl Kandidatenpositionen
* Anzahl Cash-Zeilen
* Anzahl sicher gemappt
* Anzahl manuell zu prüfen
* Anzahl blockiert
* Quality Flags

6. Review Queue erzeugen
   Unsichere Positionen sollen in eine Manual Review Queue:

* missing_isin
* missing_ticker
* ambiguous_name
* missing_fx
* cost_basis_uncertain
* snapshot_only
* market_value_legacy

7. Commit noch nicht produktiv
   Für diesen Block noch kein produktiver Import echter Aktien-/ETF-Daten.
   Nur Dry Run, Mapping und Review.

## 5. Manual Review Queue

Bitte baue oder vorbereite eine Seite:

### Manual Review Queue

Sie soll zeigen:

* unsichere Instrument-Mappings
* fehlende ISIN
* fehlende Ticker
* unklare Accounts
* unklare Stichtage
* blockierte Import-Zeilen
* fehlende FX
* cost_basis_uncertain
* snapshot_only

Aktionen:

* Mapping bestätigen
* ISIN ergänzen
* Ticker ergänzen
* Exchange ergänzen
* Account zuweisen
* Position ignorieren
* Notiz speichern

Alle Änderungen:

* Audit-Log Pflicht
* Keine produktive Positionserstellung in diesem Block

## 6. Data Quality Center vorbereiten

Bitte erweitere die Data-Quality-Seite oder baue eine erste MVP-Version.

Sie soll anzeigen:

* aktive Alerts
* resolved Alerts
* fehlende Preise
* stale prices
* fehlende FX
* fehlende ISIN
* fehlende CoinGecko-ID
* snapshot_only Positionen
* cost_basis_uncertain
* alte Alerts, die resolved werden könnten
* Button: Data Quality Recheck
* Button: Resolve Fixed Alerts

## 7. Safe Mode / Read-only Mode vorbereiten

Bitte implementiere mindestens eine sichtbare UI-Anzeige:

* aktuelle DB: Runtime oder Demo
* Modus: Read-only oder Write-enabled
* Warnung vor produktiven Writes
* Import Commit Buttons nur aktiv, wenn Write-enabled
* Default: Read-only für Import Wizard

Noch keine komplexe Rollen-/Auth-Logik nötig.

## 8. Tests

Bitte ergänze Tests für:

* instrument_mappings anlegen
* missing ISIN erzeugt needs_manual_review
* Name ohne ISIN/Ticker wird nicht automatisch gemappt
* platform_account_mapping für PostFinance
* platform_account_mapping für True Wealth
* platform_account_mapping für Raiffeisen Cash
* broker_import_dry_run speichert nur Summary, nicht produktive Positionen
* Import Wizard Module importieren sauber
* Manual Review Queue lädt offene Mapping-Probleme
* Data Quality Center zeigt fehlende ISIN / FX / Preise
* Read-only Mode blockiert Commit-Aktionen
* Audit-Log bei manueller Mapping-Bestätigung
* keine echten Dateien im Repo
* Git-Safety bleibt OK

## 9. Echte Dateien

Du darfst die vorhandenen echten Dateien aus Google Drive für strukturelle Dry-Run-Tests lesen, aber:

* nicht ins Repo kopieren
* keine echten Werte/Mengen im Chat ausgeben
* keine produktive DB-Veränderung
* nur Runtime-/Temp-Kontext
* nach Analyse Temp-Dateien entfernen
* Git-Safety vor/nachher prüfen

## 10. Nach Abschluss

Bitte ausführen:

* Compile
* pytest
* Git-Safety
* Git Status

Wenn sauber:

* Commit mit sinnvoller Message
* Push auf main

Bitte berichten:

* implementierte Module
* neue/geänderte Dateien
* Tests
* Testergebnis
* Git-Safety
* Commit Hash
* Push Status
* Status der Import Wizard UI
* welche Daten weiterhin nicht produktiv importiert wurden
* nächste Empfehlung

Wichtig:
Ziel dieses Blocks ist nicht, PostFinance/True Wealth/Raiffeisen bereits produktiv zu importieren. Ziel ist, den sicheren Review- und Mapping-Prozess zu schaffen, damit der spätere produktive Initial Snapshot sauber, nachvollziehbar und auditierbar erfolgen kann.


Explanation:
**Planning finance actions**

I realize I need to take action, possibly using the finance skills I already have. It seems like I'm set up, but before replying, I should ensure that the current relevant information is loaded properly. That way, I can inspect the project thoroughly since it seems to be a big task. I should make sure to utilize the appropriate tools. Let's see how I can move forward effectively!


finance-system-architecture

---
name: finance-system-architecture
description: "Design/implement the user's JARVIS Finance System: CHF ledger, portfolio accounting, dashboard, imports, reports, secrecy."
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [finance, portfolio, accounting, dashboard, csv, fx, tax, privacy]
---

# Finance System Architecture

## When to use

Use this skill whenever the user asks about the **Finance System – Architektur & Scripts** topic or work involving the user's personal finance-management system: portfolio imports, transaction accounting, Bank/Broker/CSV processing, dashboards, reports, tax/income views, FX logic, market data jobs, alerts, or later implementation.

Do **not** use it for pure investment-strategy discussion; those belong in the separate finance strategy topic unless the user explicitly asks for technical implementation.

## Core principles

1. **No automated trading.** The system may prepare orders and recommendations, but never executes orders or connects to broker trading APIs for execution.
2. **Accounting first, opinions later.** MVP must prioritize correct transaction ledger, cash ledger, FX handling, income/tax data, and dashboard before complex scores, news, analyst ratings, or LLM analysis.
3. **Rule-based first.** Calculations, scores, alerts, rebalancing and reports must be deterministic and auditable. LLMs may only summarize/explain text or assist extraction with human review.
4. **CHF as base currency.** Store every transaction in original currency and CHF using historical FX at transaction/payment date; current valuations use current FX.
5. **Financial data secrecy.** Never expose real financial values, portfolio files, raw exports, reports with real numbers, DBs, API keys, or credentials in chat, GitHub, logs, or shared docs unless the user explicitly approves scope.
6. **Data ownership.** All data must be exportable: CSV, JSON, Parquet, SQLite/Postgres dump, Markdown/HTML/PDF reports.

## Current v0.3 decisions

See `references/spec-v0.2-decisions.md` for the baseline, `references/spec-v0.3-decisions.md` for the priority update, `references/spec-v0.4-implementation-blueprint.md` for the MVP 1 technical implementation blueprint, `references/spec-v0.4.1-corrections.md` for accepted corrections, `references/spec-v0.5-repo-blueprint.md` for the concrete repo/implementation plan, `references/mvp-safety-gate-implementation.md` for the implementation safety-gate sequence and verification commands, `references/mvp-import-pipeline-tasks-9-13.md` for the synthetic CSV import/FX/transaction pipeline pattern, `references/mvp-ledger-cash-wac-positions.md` for cash/WAC/position ledger calculation rules and tests, `references/mvp-crypto-wallet-ledger.md` for crypto wallet/assets/holdings/transfers/buy-sell/fee MVP rules, `references/crypto-xlsx-initial-snapshot-dry-run.md` for the controlled real-file XLSX crypto initial-snapshot dry-run pattern and mapping decisions, `references/crypto-xlsx-initial-snapshot-production-import.md` for productive safe-subset XLSX import operations, compatibility/partial-write recovery, and dashboard correction follow-up, `references/crypto-single-coin-runtime-test.md` for controlled one-coin productive runtime tests and the coin-vs-holding ambiguity guard, `references/multi-wallet-crypto-and-equity-manage-ui.md` for approved multi-wallet single-coin snapshots, SQLite Decimal/TEXT affinity pitfalls, and synthetic-first Equity/ETF Manage UI patterns, `references/crypto-price-refresh-and-idempotent-btc-runtime.md` for robust batch price refresh and re-run-safe BTC/single-coin runtime checks, `references/crypto-runtime-db-consistency-verification.md` for post-import runtime DB consistency checks, exact safe-set verification, dashboard no-live-API testing, and runtime-only report generation, `references/mvp-crypto-manual-management-ui.md` for audited dashboard add/correct/remove crypto holding workflows after an initial snapshot, `references/mvp-market-import-safety-hardening.md` for CoinGecko provider/backoff, CSV importer completion, alert lifecycle, Git-safety and crypto Decimal precision patterns, `references/git-safety-verification-coverage.md` for the verifier-warning workflow and minimum Git-safety test matrix, `references/mvp-streamlit-dashboard.md` for the safe local Streamlit dashboard shell/read-model/testing pattern, `references/mvp-crypto-report-export.md` for local-only crypto report context/export rules, `references/github-token-push-from-drive.md` for the safe Google-Drive-token push workflow when local FinanceManager commits are ahead of GitHub, and `references/runtime-backup-and-broker-mapping.md` for runtime DB backup/restore hardening plus PostFinance/True Wealth/Raiffeisen structural mapping/dry-run rules. Key decisions:

- Platform/account views: Raiffeisen, PostFinance, True Wealth, Crypto-Coins, plus Gesamtansicht.
- Existing Google Drive files under `Finanzen/01 Aktueller Stand Portfolios` are **initial-snapshot reality checks**, not full ledgers.
- MVP means the smallest useful first version: correct data foundation, ledger, CHF/FX logic, platform views, crypto inventory, watchlist, reports, workflows and audit log.
- Stock/ETF tax documents are downgraded: optional reference/control module and dividend/distribution transparency, not MVP 1 and not a complete tax dossier replacement.
- Crypto inventory is promoted to MVP 1: wallet/platform-level holdings, Crypto PDF, CoinGecko valuation, stale-price warnings and manual verification.
- Crypto snapshot values are stale; use quantities only and recalculate current values via CoinGecko later.
- MVP dashboard recommendation: Streamlit + SQLite + SQLAlchemy/Pandas/Plotly; later migrateable to FastAPI + React/PostgreSQL.
- MVP inputs: manual dashboard entry, standard CSV import/export, later broker/bank export import. No direct broker integrations initially.
- Telegram crypto buys/sells/transfers should be parsed as pending transactions and only saved after explicit confirmation.
- Every portfolio or crypto change must create a History/Audit entry.

## Required data model areas

At minimum design for:

- `platforms`, `accounts`, `instruments`
- `transactions` with original currency and CHF fields
- `cash_ledger`
- derived `positions_current`
- `dividends_distributions`
- `market_prices`, `fx_rates`
- `watchlist_items`
- `alerts`
- `decision_journal`
- `reports`
- `data_quality_issues`

## CSV/import rules

- Use a canonical CSV format before broker-specific importers.
- Imports must be idempotent via external transaction ID or row hash.
- Importers must separate dry-run and commit modes, collect validation errors explicitly, and always create an `import_sessions` record even for dry runs or failed imports.
- Import results should classify rows as new, existing, duplicate, or failed; do not rely on `INSERT OR IGNORE` as the only duplicate-detection mechanism.
- Row hashes must be stable and deterministic: normalize key order, whitespace, and `None` versus empty strings while leaving domain-specific canonicalization to the importer.
- If a source file has only current holdings and no full history, import it as `initial_position_snapshot` / `initial_cash_snapshot`, not fake historical buys.
- Missing historic FX, fees, taxes, dates, investment case, target weights, or dividend details must be flagged as data-quality gaps.

## FX and return rules

For foreign currency assets, always distinguish:

- price/security performance in original currency
- FX effect to CHF
- realized P&L
- unrealized P&L
- dividends/distributions/income
- fees and taxes
- Total Return in CHF

Every transaction/income event needs:

- original amount and currency
- historical FX rate to CHF, or explicit `fx_status=missing` with a critical data-quality alert
- FX source and timestamp
- CHF equivalent at event date when FX is known

Manual FX overrides require a non-empty note and an audit-log entry.

## Ledger calculation rules

- Cash after system start must be calculated from `transactions`; `cash_balances` is only a snapshot/cache/control view.
- `initial_cash_snapshot` and `initial_position_snapshot` are explicit starting truths, not fabricated historical trades.
- Implement Weighted Average Cost as the MVP internal performance method: buy fees increase cost basis, partial sells remove proportional cost basis, realized P&L is calculated against average cost, full sells zero remaining quantity/cost basis, and dividends do not alter quantity or cost basis.
- WAC is not tax-complete; do not add FIFO/LIFO/tax-lot logic in MVP 1, but keep transaction granularity so it can be added later.
- Position calculations must keep zero-quantity historical positions visible when sale history exists.
- Missing market price or FX must degrade data quality and suppress precise Total Return CHF rather than inventing a value.
- Before productive dashboard use, add alert deduplication per rule/entity/date.
- Market-FX for non-CHF market prices belongs to the later Market/FX provider block.

## Crypto MVP rules

- Crypto inventory is MVP-critical but must remain dummy-first until mapper/dry-run layers are ready; do not touch real coin/wallet files during core implementation. After explicit user approval, real files may be read only from temp/runtime for controlled dry runs using `references/crypto-xlsx-initial-snapshot-dry-run.md`; report only aggregate counts/labels and never quantities or values. Productive safe-subset imports require a second explicit approval and must follow `references/crypto-xlsx-initial-snapshot-production-import.md`: external runtime DB only, no real CSV/XLSX/reports/DBs in Git, Git-safety before/after, import-session + audit-log creation, and aggregate-only reporting.
- Wallet names must be unique; wallet addresses are optional and must not appear in tests/fixtures.
- Validate wallet type against: Hardware Wallet, Software Wallet, Exchange, Bank/Broker, DeFi, Sonstiges.
- CoinGecko ID is strongly recommended but not hard-required; missing ID creates a data-quality warning and blocks reliable valuation.
- Crypto symbols are not globally unique; symbol conflicts require warning/manual selection, never silent merging.
- Initial crypto holdings are confirmed snapshots by wallet+asset. Store legacy values for provenance/control only, never current valuation.
- Crypto holdings should be calculated/controlled from initial snapshots plus crypto transactions, not silently overwritten.
- Crypto transfers require different source/target wallets, quantity > 0, and same asset; coin fees reduce source-wallet balance.
- Negative wallet balances should be blocked in MVP and produce a critical alert unless a future explicit correction workflow is implemented.
- Crypto buys/sells with fiat impact must create/reference general ledger transactions via `source_type='crypto'`/`source_id` so cash remains ledger-consistent.
- Fiat fees should be ledger fees when unambiguous; avoid double counting when a fee is already embedded in sell net proceeds.
- Unknown fee currency creates a data-quality warning instead of invented accounting.
- Crypto Cost Basis/WAC is not required for MVP 1; MVP 1 prioritizes crypto holdings, wallet structure, current valuation, reports and audit.
- Crypto report context/export must be renderer-neutral and local-only: build from SQLite, aggregate by coin and wallet, use local cached prices only, preserve Decimal/Text values, ignore legacy snapshot values for current valuation, include data-quality warnings, and write generated files only to the external runtime reports directory.
- Crypto PDF export should be optional-dependency tolerant: prefer PDF when a renderer is available, otherwise fall back to HTML/Markdown with a clear warning while still recording report metadata and audit.
- After productive real-data crypto imports, run the post-import runtime verification in `references/crypto-runtime-db-consistency-verification.md`: exact safe-set membership checks, aggregate-only counts, audit/import-session traceability, dashboard render with no live API, runtime-only report generation, and Git-safety before/after.
- Manual dashboard additions/corrections/removals after an initial snapshot must follow `references/mvp-crypto-manual-management-ui.md`: no silent `crypto_holdings` mutation, explicit confirmation, mandatory notes for adjustments/removals, audit-log entries with old/new values, Decimal/TEXT quantity handling, no negative holdings, and no live API lookup during render.
- Crypto tax logic, tax lots, FIFO/LIFO and detailed cost basis are later modules.
- Keep crypto assets separate from `instruments` for MVP 1, but later evaluate a shared Asset Master if it improves reporting, performance views or watchlists.

## Income/tax rules

For dividends and ETF distributions, separately store:

- gross dividend/distribution
- Swiss withholding tax 35%
- foreign withholding tax
- other tax/withholding if present
- net amount
- original currency
- historic FX to CHF
- gross/net CHF equivalents
- platform/account
- ISIN/ticker
- payment date
- ex-dividend date if available

Plan later ESTV/ICTax support for Swiss year-end values and tax reports, but label outputs as technical reports, not tax advice.

## Market-data rules

- Dashboard reads primarily from local DB, not directly from APIs on page load.
- First Streamlit MVP should be a safe, local, read-first shell over SQLite with explicit synthetic demo DB creation only; page render functions must not call CoinGecko/market APIs, and tests should prove that by monkeypatching provider methods to fail.
- Use scheduled jobs to update prices/FX/news.
- Store provider, timestamp, quality status and freshness for every price/FX point.
- Implement caching, local historical storage, batch requests where possible, provider rate-limit queues, and backoff on HTTP 429.
- CoinGecko integrations must be mockable and unit-tested without real API calls; treat HTTP 429 as non-fatal and return a non-fresh data-quality status plus alert after configured retries.
- Crypto price refresh belongs in CLI/scheduled-job commands, not dashboard/report rendering. Prefer CoinGecko batch requests over per-coin requests; expose conservative controls (`--sleep-seconds`, `--max-retries`, `--initial-backoff`, `--max-backoff`, `--only-missing`, `--only-stale`, `--only-symbol`, `--limit`, `--dry-run`, `--currency`). Commands should use aggregate-only output (`total_assets/updated/cached/skipped/stale/warnings/errors/assets_still_missing_local_price`), cache by configurable `max_age_seconds`, store exact Decimal prices as TEXT in `crypto_prices`, and verify dashboard/report contexts read local prices with `live_api_calls=False`. For details and tests see `references/crypto-price-refresh-and-idempotent-btc-runtime.md`.
- Candidate providers: Finnhub, Financial Modeling Prep, Twelve Data, Stooq, Alpha Vantage selectively, Yahoo Finance fallback only; CoinGecko for crypto; later DefiLlama; ESTV/ICTax for Swiss tax values.

## Crypto Decimal precision rules

SQLite has no exact native DECIMAL. In the crypto context, store quantities, prices, crypto fees and valuation-relevant amounts as exact Decimal strings (`TEXT`) and compute with Python `Decimal`. Never convert crypto quantity/price/fee/valuation through float/REAL. Format Decimal writes with `format(value, "f")` so small quantities such as `0.00000001` do not become scientific notation. CSV importers must validate Decimal strings, and tests should assert high-precision preservation plus SQLite text storage. For multi-wallet real-data snapshots and Equity/ETF ledger fields, also see `references/multi-wallet-crypto-and-equity-manage-ui.md`; SQLite `NUMERIC` affinity can coerce Decimal-looking text to `real`, so schema/migrations should use `TEXT` for exact quantities, FX rates, original amounts and CHF equivalents where precision matters.

## Alert lifecycle rules

Alerts must be deduplicated and lifecycle-aware before dashboards run repeated calculations. Use a dedup key from `rule_id`, `entity_type`, `entity_id`, `priority` and an error fingerprint. Alerts should carry `status` (`active`, `resolved`, `muted`), `resolved_at`, optional `muted_until`, `last_seen_at`, `occurrence_count`, `fingerprint` and `dedup_key`. Re-seeing an identical active/resolved/muted alert should update `last_seen_at`/`occurrence_count` rather than create a new active row; changed fingerprints may create a new alert.

## Git/GitHub safety

Code/scripts/docs may be stored in a private Git repo, but never commit financial data. Use the `github-repo-management` skill and its `references/sensitive-code-backup.md` guidance.

Minimum `.gitignore` for finance repos must block `data/`, `reports/`, `exports/`, `backups/`, DB files, spreadsheets, PDFs, DOCX, raw CSV/JSON, `.env`, tokens, credentials and client secrets. Only synthetic examples under `examples/` are acceptable.

## MVP implementation safety gate

Before any real broker/bank/portfolio/crypto data is imported, build and verify the safety foundation described in `references/mvp-safety-gate-implementation.md`: repo skeleton, `.gitignore`, `.env.example`, runtime outside repo, settings validation, SQLite schema, synthetic fixtures, audit log, CSV dry-run, duplicate protection, and Git-safety scan. Run the safety scan before staging, committing, or pushing. If GitHub credentials are needed from Drive, use them only in a temporary location and never echo token values.

Before broader productive imports, implement or verify the runtime backup loop in `references/runtime-backup-and-broker-mapping.md`: backup outside Git, SHA256 verification, restore requiring explicit confirmation, tests for backup/restore/checksum, and Git-safety coverage that blocks backup artifacts in the repo. For PostFinance/True Wealth/Raiffeisen, perform structural mapping and dry-runs only until the user approves import; report aggregate counts/field coverage/quality flags only, never real values or raw rows.

## Common pitfalls

- Building a fancy dashboard before ledger correctness is verified.
- Treating a holdings snapshot as a real transaction history.
- Mixing price P&L and FX P&L.
- Letting dashboard refreshes hammer APIs.
- Committing realistic sample data to Git because the repo is private. Private GitHub is not a financial data vault.
- Showing real portfolio values back to the user unnecessarily in summaries.
- Designing Git-safety scans that block source package names like `src/.../imports` or `src/.../reports`; block top-level runtime/data directories instead.
- Running compile/tests before Git-safety and forgetting generated `__pycache__`; remove or ignore generated caches before safety scans. `pytest` may recreate `.pytest_cache`, so remove `__pycache__`/`.pytest_cache` again after test execution and before `git-safety-scan`.
- Productive real-data imports can partially write before surfacing schema/importer compatibility issues. Do not rerun blindly or dump row-level data; inspect aggregate counts and unique identifiers needed for idempotency, patch/complete via missing-only inserts, then verify committed import session, audit log, Git clean and Git-safety OK.
- In controlled one-coin runtime tests from a real XLSX, distinguish **one coin/asset** from **one holding/wallet row**. A coin may be split across multiple wallet columns; if the user required an unambiguous target wallet or one holding, stop and ask rather than choosing a wallet silently. If the user explicitly approves multi-wallet import for that coin, import exactly one asset plus one audited holding per non-empty approved wallet cell; verify aggregate counts only. If re-running a BTC/single-coin test finds the exact approved asset and holdings already present, do not import duplicates; perform the aggregate-only idempotency checks and report that asset/holding counts did not increase because duplicate prevention succeeded. See `references/crypto-single-coin-runtime-test.md`, `references/multi-wallet-crypto-and-equity-manage-ui.md`, and `references/crypto-price-refresh-and-idempotent-btc-runtime.md`.
- After productive crypto imports, do a separate aggregate-only consistency pass before building new features: DB/runtime outside repo, exact safe-subset membership, no duplicate/negative/orphan holdings, correct snapshot date, null legacy-value fields when ignored, import sessions committed, holding-level audit traces present, dashboard local-read/no-live-API verified, and report generated only under runtime reports.
- Letting a dashboard correction UI mutate crypto holdings directly. Add/remove/edit operations must become audited transactions or `manual_adjustment` events with mandatory notes where appropriate; preserve history instead of silently overwriting balances.
- Treating verifier/file-mutation warnings as either automatically true or automatically false. Check the exact commit with `git show --name-status --patch <commit> -- <file>`, then separately verify whether the test matrix actually covers the promised cases.
- Adding broad Git-safety tests that only assert `len(findings) >= N`; prefer explicit assertions for token filename, PAT patterns, private keys, CSV/JSON path policy, runtime/data directories, allowed synthetic paths, `config.example/`, and normal `src/` false-positive protection.

To view linked files, call skill_view(name, file_path) where file_path is e.g. 'references/api.md' or 'assets/config.yaml'


todo

Schema/Modelle für Instrument-, Account-Mapping und Broker Dry-Run Phase B1 erweitern

Import Wizard, Manual Review Queue, Data Quality Center, Safe Mode UI als Dashboard-Seiten/MVP implementieren

Tests für Mapping, Dry-Run, Review, DQ, Read-only, Audit und Git-Safety ergänzen

Compile, pytest, Git-Safety, Status, Commit und Push durchführen


     1|from __future__ import annotations
     2|
     3|import hashlib
     4|from datetime import datetime, timezone
     5|from sqlite3 import Connection
     6|
     7|from .schema import INITIAL_SCHEMA_SQL
     8|
     9|MIGRATION_VERSION = 3
    10|MIGRATION_NAME = "003_broker_bank_mapping_foundation"
    11|
    12|INSTRUMENT_OPTIONAL_COLUMNS = {
    13|    "position_category": "TEXT",
    14|    "ter": "TEXT",
    15|    "distribution_policy": "TEXT",
    16|    "index_name": "TEXT",
    17|    "fund_domicile": "TEXT",
    18|    "benchmark": "TEXT",
    19|}
    20|
    21|TEXT_AFFINITY_COLUMNS = {
    22|    "transactions": {"quantity", "price_original", "gross_amount_original", "fee_original", "tax_original", "net_amount_original", "fx_rate_to_chf", "gross_amount_chf", "fee_chf", "tax_chf", "net_amount_chf"},
    23|    "crypto_holdings": {"quantity", "legacy_snapshot_value_original", "legacy_snapshot_value_chf"},
    24|    "crypto_transactions": {"quantity", "price_original", "gross_amount_original", "fee_quantity", "fee_original", "fx_rate_to_chf", "amount_chf"},
    25|    "crypto_prices": {"price", "market_cap", "volume_24h", "change_24h_pct"},
    26|}
    27|
    28|
    29|def utc_now() -> str:
    30|    return datetime.now(timezone.utc).isoformat()
    31|
    32|
    33|def checksum_sql(sql: str) -> str:
    34|    return hashlib.sha256(sql.encode("utf-8")).hexdigest()
    35|
    36|
    37|def get_schema_version(conn: Connection) -> int:
    38|    row = conn.execute("SELECT MAX(version) AS version FROM schema_migrations").fetchone()
    39|    return int(row["version"] or 0) if row else 0
    40|
    41|
    42|def _table_columns(conn: Connection, table: str) -> dict[str, str]:
    43|    return {row["name"]: (row["type"] or "") for row in conn.execute(f"PRAGMA table_info({table})").fetchall()}
    44|
    45|
    46|def _add_missing_instrument_columns(conn: Connection) -> None:
    47|    existing = _table_columns(conn, "instruments")
    48|    for name, col_type in INSTRUMENT_OPTIONAL_COLUMNS.items():
    49|        if name not in existing:
    50|            conn.execute(f"ALTER TABLE instruments ADD COLUMN {name} {col_type}")
    51|
    52|
    53|def _rebuild_table_with_text_columns(conn: Connection, table: str, text_columns: set[str]) -> None:
    54|    cols = conn.execute(f"PRAGMA table_info({table})").fetchall()
    55|    if not cols:
    56|        return
    57|    needs_rebuild = any(row["name"] in text_columns and (row["type"] or "").upper() != "TEXT" for row in cols)
    58|    if not needs_rebuild:
    59|        return
    60|    tmp = f"{table}__text_migration"
    61|    col_defs: list[str] = []
    62|    pk_cols = [row["name"] for row in cols if row["pk"]]
    63|    for row in cols:
    64|        name = row["name"]
    65|        col_type = "TEXT" if name in text_columns else (row["type"] or "TEXT")
    66|        parts = [name, col_type]
    67|        if row["pk"] and len(pk_cols) == 1:
    68|            parts.append("PRIMARY KEY")
    69|        if row["notnull"]:
    70|            parts.append("NOT NULL")
    71|        if row["dflt_value"] is not None:
    72|            parts.append(f"DEFAULT {row['dflt_value']}")
    73|        col_defs.append(" ".join(parts))
    74|    if len(pk_cols) > 1:
    75|        col_defs.append("PRIMARY KEY (" + ", ".join(pk_cols) + ")")
    76|    conn.execute(f"DROP TABLE IF EXISTS {tmp}")
    77|    conn.execute(f"CREATE TABLE {tmp} (" + ", ".join(col_defs) + ")")
    78|    names = [row["name"] for row in cols]
    79|    select_exprs = [f"CAST({name} AS TEXT)" if name in text_columns else name for name in names]
    80|    conn.execute(f"INSERT INTO {tmp} ({', '.join(names)}) SELECT {', '.join(select_exprs)} FROM {table}")
    81|    conn.execute(f"DROP TABLE {table}")
    82|    conn.execute(f"ALTER TABLE {tmp} RENAME TO {table}")
    83|    if table == "crypto_holdings":
    84|        conn.execute("CREATE UNIQUE INDEX IF NOT EXISTS idx_crypto_holdings_asset_wallet ON crypto_holdings(asset_id, wallet_id)")
    85|
    86|
    87|def _create_broker_bank_mapping_tables(conn: Connection) -> None:
    88|    conn.executescript(
    89|        """
    90|        CREATE TABLE IF NOT EXISTS instrument_mappings (
    91|            mapping_id TEXT PRIMARY KEY,
    92|            source_name TEXT NOT NULL,
    93|            source_instrument_name TEXT,
    94|            source_isin TEXT,
    95|            source_ticker TEXT,
    96|            source_exchange TEXT,
    97|            source_currency TEXT,
    98|            instrument_id TEXT REFERENCES instruments(instrument_id),
    99|            mapping_status TEXT NOT NULL DEFAULT 'unmapped',
   100|            quality_flags_json TEXT,
   101|            notes TEXT,
   102|            created_at TEXT NOT NULL,
   103|            updated_at TEXT,
   104|            UNIQUE(source_name, source_isin, source_ticker, source_instrument_name, source_currency)
   105|        );
   106|
   107|        CREATE TABLE IF NOT EXISTS platform_account_mappings (
   108|            mapping_id TEXT PRIMARY KEY,
   109|            source_name TEXT NOT NULL,
   110|            source_platform_label TEXT,
   111|            source_account_label TEXT,
   112|            source_account_type TEXT,
   113|            source_currency TEXT,
   114|            platform_id TEXT REFERENCES platforms(platform_id),
   115|            account_id TEXT REFERENCES accounts(account_id),
   116|            mapping_status TEXT NOT NULL DEFAULT 'unmapped',
   117|            quality_flags_json TEXT,
   118|            notes TEXT,
   119|            created_at TEXT NOT NULL,
   120|            updated_at TEXT,
   121|            UNIQUE(source_name, source_platform_label, source_account_label, source_currency)
   122|        );
   123|
   124|        CREATE TABLE IF NOT EXISTS broker_import_dry_runs (
   125|            dry_run_id TEXT PRIMARY KEY,
   126|            source_name TEXT NOT NULL,
   127|            source_file_label TEXT NOT NULL,
   128|            source_file_type TEXT NOT NULL,
   129|            source_hash TEXT,
   130|            detected_sections_json TEXT NOT NULL,
   131|            field_coverage_json TEXT NOT NULL,
   132|            quality_flags_json TEXT NOT NULL,
   133|            rows_total INTEGER NOT NULL DEFAULT 0,
   134|            rows_position_candidates INTEGER NOT NULL DEFAULT 0,
   135|            rows_cash_candidates INTEGER NOT NULL DEFAULT 0,
   136|            created_at TEXT NOT NULL,
   137|            notes TEXT
   138|        );
   139|        """
   140|    )
   141|
   142|def _apply_compat_migrations(conn: Connection) -> None:
   143|    _add_missing_instrument_columns(conn)
   144|    for table, text_columns in TEXT_AFFINITY_COLUMNS.items():
   145|        _rebuild_table_with_text_columns(conn, table, text_columns)
   146|    _create_broker_bank_mapping_tables(conn)
   147|
   148|
   149|def apply_migrations(conn: Connection) -> None:
   150|    conn.executescript(INITIAL_SCHEMA_SQL)
   151|    existing_initial = conn.execute("SELECT 1 FROM schema_migrations WHERE version = 1").fetchone()
   152|    if not existing_initial:
   153|        conn.execute(
   154|            "INSERT INTO schema_migrations(version, name, applied_at, checksum) VALUES (?, ?, ?, ?)",
   155|            (1, "001_initial_schema", utc_now(), checksum_sql(INITIAL_SCHEMA_SQL)),
   156|        )
   157|    _apply_compat_migrations(conn)
   158|    existing = conn.execute("SELECT 1 FROM schema_migrations WHERE version = ?", (MIGRATION_VERSION,)).fetchone()
   159|    if not existing:
   160|        conn.execute(
   161|            "INSERT INTO schema_migrations(version, name, applied_at, checksum) VALUES (?, ?, ?, ?)",
   162|            (MIGRATION_VERSION, MIGRATION_NAME, utc_now(), checksum_sql(MIGRATION_NAME)),
   163|        )
   164|    conn.commit()
   165|


Explanation:
**Implementing robust solutions**

I need to focus on implementing robust solutions. It sounds like I should use an execution code approach to write modules and handle migration patches. I’ll also need to inspect the audit log for any issues and check the dashboard’s main page patterns to ensure everything is functioning smoothly. It’s essential to keep everything organized and efficient during this process so that I can achieve the best results!


     1|from __future__ import annotations
     2|
     3|import json
     4|import uuid
     5|from datetime import datetime, timezone
     6|from sqlite3 import Connection
     7|from typing import Any
     8|
     9|
    10|def utc_now() -> str:
    11|    return datetime.now(timezone.utc).isoformat()
    12|
    13|
    14|def record_audit_event(
    15|    conn: Connection,
    16|    *,
    17|    source: str,
    18|    action: str,
    19|    entity_type: str,
    20|    entity_id: str,
    21|    old_values: dict[str, Any] | None = None,
    22|    new_values: dict[str, Any] | None = None,
    23|    user_text_note: str | None = None,
    24|    confirmed: bool = True,
    25|    created_by: str = "system",
    26|) -> str:
    27|    audit_id = str(uuid.uuid4())
    28|    now = utc_now()
    29|    conn.execute(
    30|        """
    31|        INSERT INTO audit_log(
    32|            audit_id, timestamp, source, action, entity_type, entity_id,
    33|            old_values_json, new_values_json, user_text_note, confirmed,
    34|            confirmation_timestamp, created_by, created_at
    35|        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    36|        """,
    37|        (
    38|            audit_id,
    39|            now,
    40|            source,
    41|            action,
    42|            entity_type,
    43|            entity_id,
    44|            json.dumps(old_values or {}, sort_keys=True),
    45|            json.dumps(new_values or {}, sort_keys=True),
    46|            user_text_note,
    47|            1 if confirmed else 0,
    48|            now if confirmed else None,
    49|            created_by,
    50|            now,
    51|        ),
    52|    )
    53|    return audit_id
    54|


     1|from __future__ import annotations
     2|
     3|from pathlib import Path
     4|from sqlite3 import Connection
     5|
     6|from jarvis_finance.config.settings import ensure_runtime_dirs, find_repo_root, load_settings
     7|from jarvis_finance.dashboard.demo_data import load_synthetic_demo_data
     8|from jarvis_finance.dashboard.pages import PAGES
     9|from jarvis_finance.storage.database import connect
    10|from jarvis_finance.storage.migrations import apply_migrations
    11|
    12|
    13|def safe_connect_existing_db(db_path: Path) -> Connection | None:
    14|    if not db_path.exists():
    15|        return None
    16|    conn = connect(db_path)
    17|    apply_migrations(conn)
    18|    return conn
    19|
    20|
    21|def create_demo_db(db_path: Path) -> None:
    22|    db_path.parent.mkdir(parents=True, exist_ok=True)
    23|    conn = connect(db_path)
    24|    try:
    25|        load_synthetic_demo_data(conn)
    26|    finally:
    27|        conn.close()
    28|
    29|
    30|def run_app() -> None:
    31|    import streamlit as st
    32|
    33|    st.set_page_config(page_title="JARVIS Finance MVP", layout="wide")
    34|    st.sidebar.title("JARVIS Finance")
    35|    st.sidebar.caption("MVP: lokal, synthetisch, read-first")
    36|
    37|    try:
    38|        repo_root = find_repo_root()
    39|        settings = load_settings(repo_root=repo_root)
    40|    except Exception as exc:
    41|        st.error(f"Settings-/Runtime-Ladeprüfung fehlgeschlagen: {exc}")
    42|        st.info("Prüfe JARVIS_FINANCE_RUNTIME_DIR. Runtime muss ausserhalb des Repos liegen.")
    43|        return
    44|
    45|    ensure_runtime_dirs(settings)
    46|    db_path = settings.db_path
    47|    conn = safe_connect_existing_db(db_path)
    48|
    49|    if conn is None:
    50|        st.title("JARVIS Finance Dashboard")
    51|        st.error(f"Keine lokale SQLite-DB gefunden: {db_path}")
    52|        st.info("Das Dashboard erstellt oder verändert keine echte Datenbank automatisch.")
    53|        st.caption("Für lokale Entwicklung kann explizit eine synthetische Dummy-DB aus tests/fixtures geladen werden.")
    54|        if st.button("Synthetische Dummy-DB jetzt erstellen", type="primary"):
    55|            create_demo_db(db_path)
    56|            st.success("Synthetische Dummy-DB erstellt. Bitte Seite neu laden.")
    57|        return
    58|
    59|    try:
    60|        page_names = [page.PAGE_TITLE for page in PAGES]
    61|        selected = st.sidebar.radio("Navigation", page_names, index=0)
    62|        st.sidebar.write(f"DB: `{db_path}`")
    63|        st.sidebar.warning("Keine Live-API-Abfragen beim Seitenreload. Preise nur aus lokaler DB/Cache.")
    64|        page = PAGES[page_names.index(selected)]
    65|        page.render(st, conn)
    66|    finally:
    67|        conn.close()
    68|
    69|
    70|if __name__ == "__main__":
    71|    run_app()
    72|


     1|from __future__ import annotations
     2|
     3|from . import page_00_command_center
     4|from . import page_01_portfolio
     5|from . import page_02_platforms
     6|from . import page_03_equities_etfs
     7|from . import page_04_crypto
     8|from . import page_05_wallets
     9|from . import page_06_ledger
    10|from . import page_07_watchlist
    11|from . import page_08_reports
    12|from . import page_09_alerts
    13|from . import page_10_audit
    14|from . import page_11_settings_data_quality
    15|from . import page_12_crypto_manage
    16|from . import page_13_equity_manage
    17|
    18|PAGES = [
    19|    page_00_command_center,
    20|    page_01_portfolio,
    21|    page_02_platforms,
    22|    page_03_equities_etfs,
    23|    page_04_crypto,
    24|    page_05_wallets,
    25|    page_06_ledger,
    26|    page_07_watchlist,
    27|    page_08_reports,
    28|    page_09_alerts,
    29|    page_10_audit,
    30|    page_11_settings_data_quality,
    31|    page_12_crypto_manage,
    32|    page_13_equity_manage,
    33|]
    34|


     1|from __future__ import annotations
     2|
     3|from jarvis_finance.dashboard.components.tables import show_table
     4|from jarvis_finance.dashboard import data
     5|
     6|from pathlib import Path
     7|
     8|from jarvis_finance.config.settings import find_repo_root, load_settings
     9|
    10|PAGE_TITLE = "Settings/Data Quality"
    11|
    12|def render(st, conn) -> None:
    13|    st.title("Settings / Data Quality")
    14|    try:
    15|        settings = load_settings()
    16|        repo_root = find_repo_root()
    17|        runtime_ok = data.runtime_outside_repo(settings, repo_root)
    18|        rows = [
    19|            {"setting": "Runtime-Pfad", "value": str(settings.runtime_paths.base_dir)},
    20|            {"setting": "DB-Pfad", "value": str(settings.db_path)},
    21|            {"setting": "Runtime ausserhalb Repo", "value": str(runtime_ok)},
    22|            {"setting": "Basiswährung", "value": settings.base_currency},
    23|        ]
    24|        show_table(st, rows)
    25|    except Exception as exc:
    26|        st.error(f"Settings-Fehler: {exc}")
    27|    st.subheader("Data Quality")
    28|    show_table(st, data.get_data_quality(conn))
    29|


     1|from __future__ import annotations
     2|
     3|import json
     4|from decimal import Decimal, InvalidOperation, ROUND_HALF_UP
     5|from pathlib import Path
     6|from sqlite3 import Connection
     7|
     8|from jarvis_finance.config.settings import Settings
     9|from jarvis_finance.crypto.holdings import calculate_crypto_holdings
    10|from jarvis_finance.ledger.cash import calculate_cash_balances
    11|from jarvis_finance.ledger.positions import calculate_positions
    12|from jarvis_finance.market.providers import is_price_stale
    13|
    14|MONEY = Decimal("0.01")
    15|
    16|
    17|def d(value: object, default: str = "0") -> Decimal:
    18|    if value is None or str(value).strip() == "":
    19|        return Decimal(default)
    20|    try:
    21|        return Decimal(str(value))
    22|    except InvalidOperation:
    23|        return Decimal(default)
    24|
    25|
    26|def dec_text(value: Decimal | int | str | None, places: str | None = None) -> str:
    27|    if value is None:
    28|        return ""
    29|    value = d(value)
    30|    if places:
    31|        value = value.quantize(Decimal(places), rounding=ROUND_HALF_UP)
    32|    return format(value, "f")
    33|
    34|
    35|def money_text(value: Decimal | None, currency: str = "CHF") -> str:
    36|    if value is None:
    37|        return f"n/a {currency}"
    38|    return f"{dec_text(value, '0.01')} {currency}"
    39|
    40|
    41|def rowdicts(rows) -> list[dict[str, str]]:
    42|    return [{k: ("" if row[k] is None else str(row[k])) for k in row.keys()} for row in rows]
    43|
    44|
    45|def db_exists(path: str | Path) -> bool:
    46|    return Path(path).expanduser().exists()
    47|
    48|
    49|def runtime_outside_repo(settings: Settings, repo_root: Path) -> bool:
    50|    try:
    51|        settings.runtime_paths.base_dir.resolve().relative_to(repo_root.resolve())
    52|        return False
    53|    except ValueError:
    54|        return True
    55|
    56|
    57|def latest_crypto_price(conn: Connection, asset_id: str, currency: str = "CHF", *, max_age_seconds: int = 86_400) -> dict[str, str] | None:
    58|    row = conn.execute(
    59|        """
    60|        SELECT price, price_currency, fetched_at, provider_timestamp, quality_status, provider
    61|        FROM crypto_prices
    62|        WHERE asset_id=? AND price_currency=?
    63|        ORDER BY COALESCE(provider_timestamp, fetched_at, '') DESC, fetched_at DESC LIMIT 1
    64|        """,
    65|        (asset_id, currency.upper()),
    66|    ).fetchone()
    67|    if not row:
    68|        return None
    69|    item = dict(row)
    70|    ts = item.get("provider_timestamp") or item.get("fetched_at")
    71|    if item.get("quality_status") == "fresh" and is_price_stale(ts, max_age_seconds=max_age_seconds):
    72|        item["quality_status"] = "stale"
    73|    return item
    74|
    75|
    76|def get_crypto_overview(conn: Connection, *, price_max_age_seconds: int = 86_400) -> list[dict[str, str]]:
    77|    holdings = calculate_crypto_holdings(conn)
    78|    rows: list[dict[str, str]] = []
    79|    for asset_id, holding in sorted(holdings.total_by_asset.items()):
    80|        qty = holding.quantity
    81|        asset = conn.execute("SELECT * FROM crypto_assets WHERE asset_id=?", (asset_id,)).fetchone()
    82|        if not asset:
    83|            continue
    84|        price = latest_crypto_price(conn, asset_id, "CHF", max_age_seconds=price_max_age_seconds)
    85|        price_dec = d(price["price"]) if price and price["price"] else None
    86|        value = qty * price_dec if price_dec is not None else None
    87|        rows.append({
    88|            "asset_id": asset_id,
    89|            "coin_name": asset["coin_name"],
    90|            "symbol": asset["symbol"],
    91|            "coingecko_id": asset["coingecko_id"] or "",
    92|            "quantity": dec_text(qty),
    93|            "latest_price_chf": dec_text(price_dec, "0.01") if price_dec is not None else "",
    94|            "market_value_chf": dec_text(value, "0.01") if value is not None else "",
    95|            "price_quality": price["quality_status"] if price else "missing",
    96|            "fetched_at": price["fetched_at"] if price else "",
    97|        })
    98|    return rows
    99|
   100|
   101|def get_crypto_by_wallet(conn: Connection, *, price_max_age_seconds: int = 86_400) -> list[dict[str, str]]:
   102|    holdings = calculate_crypto_holdings(conn)
   103|    rows: list[dict[str, str]] = []
   104|    for (wallet_id, asset_id), holding in sorted(holdings.wallet_holdings.items()):
   105|        wallet = conn.execute("SELECT wallet_name, wallet_type, platform_provider FROM crypto_wallets WHERE wallet_id=?", (wallet_id,)).fetchone()
   106|        asset = conn.execute("SELECT coin_name, symbol, coingecko_id FROM crypto_assets WHERE asset_id=?", (asset_id,)).fetchone()
   107|        price = latest_crypto_price(conn, asset_id, "CHF", max_age_seconds=price_max_age_seconds)
   108|        price_dec = d(price["price"]) if price and price["price"] else None
   109|        value = holding.quantity * price_dec if price_dec is not None else None
   110|        rows.append({
   111|            "wallet_id": wallet_id,
   112|            "wallet_name": wallet["wallet_name"] if wallet else wallet_id,
   113|            "wallet_type": wallet["wallet_type"] if wallet else "",
   114|            "provider": wallet["platform_provider"] if wallet else "",
   115|            "coin_name": asset["coin_name"] if asset else asset_id,
   116|            "symbol": asset["symbol"] if asset else "",
   117|            "coingecko_id": asset["coingecko_id"] if asset else "",
   118|            "quantity": dec_text(holding.quantity),
   119|            "market_value_chf": dec_text(value, "0.01") if value is not None else "",
   120|            "wallet_value_chf": dec_text(value, "0.01") if value is not None else "",
   121|            "price_quality": price["quality_status"] if price else "missing",
   122|            "price_fetched_at": price["fetched_at"] if price else "",
   123|            "verification_status": holding.verification_status,
   124|            "last_verified_at": holding.last_verified_at or "",
   125|        })
   126|    return rows
   127|
   128|
   129|def get_wallets(conn: Connection) -> list[dict[str, str]]:
   130|    return rowdicts(conn.execute(
   131|        """
   132|        SELECT wallet_id, wallet_name, wallet_type, platform_provider AS provider,
   133|               network_chain, last_verified_at, is_active, notes
   134|        FROM crypto_wallets ORDER BY wallet_name
   135|        """
   136|    ).fetchall())
   137|
   138|
   139|def get_latest_crypto_transactions(conn: Connection, limit: int = 20) -> list[dict[str, str]]:
   140|    return rowdicts(conn.execute(
   141|        """
   142|        SELECT ct.transaction_datetime, ct.transaction_type, ca.symbol, ct.quantity, ct.fee_quantity,
   143|               fw.wallet_name AS from_wallet, tw.wallet_name AS to_wallet, ct.currency_original,
   144|               ct.amount_chf, ct.confirmation_status, ct.source
   145|        FROM crypto_transactions ct
   146|        LEFT JOIN crypto_assets ca ON ca.asset_id=ct.asset_id
   147|        LEFT JOIN crypto_wallets fw ON fw.wallet_id=ct.from_wallet_id
   148|        LEFT JOIN crypto_wallets tw ON tw.wallet_id=ct.to_wallet_id
   149|        ORDER BY ct.transaction_datetime DESC, ct.created_at DESC LIMIT ?
   150|        """, (limit,)
   151|    ).fetchall())
   152|
   153|
   154|def get_platform_overview(conn: Connection) -> list[dict[str, str]]:
   155|    return rowdicts(conn.execute(
   156|        """
   157|        SELECT p.name AS platform, p.platform_type, a.account_name, a.account_type, a.currency,
   158|               a.performance_included, a.is_active
   159|        FROM accounts a JOIN platforms p ON p.platform_id=a.platform_id
   160|        ORDER BY p.name, a.account_name
   161|        """
   162|    ).fetchall())
   163|
   164|
   165|def get_cash_overview(conn: Connection) -> list[dict[str, str]]:
   166|    result = calculate_cash_balances(conn)
   167|    rows: list[dict[str, str]] = []
   168|    for (account_id, currency), bal in sorted(result.balances.items()):
   169|        row = conn.execute(
   170|            "SELECT a.account_name, p.name AS platform FROM accounts a JOIN platforms p ON p.platform_id=a.platform_id WHERE a.account_id=?",
   171|            (account_id,),
   172|        ).fetchone()
   173|        rows.append({
   174|            "platform": row["platform"] if row else "",
   175|            "account_name": row["account_name"] if row else account_id,
   176|            "currency": currency,
   177|            "amount_original": dec_text(bal.amount_original, "0.01"),
   178|            "amount_chf": dec_text(bal.amount_chf, "0.01") if bal.amount_chf is not None else "",
   179|            "quality_status": bal.quality_status,
   180|        })
   181|    return rows
   182|
   183|
   184|def get_positions(conn: Connection) -> list[dict[str, str]]:
   185|    result = calculate_positions(conn)
   186|    rows: list[dict[str, str]] = []
   187|    for (account_id, instrument_id), pos in sorted(result.positions.items()):
   188|        row = conn.execute(
   189|            """
   190|            SELECT a.account_name, p.name AS platform, i.name, i.ticker, i.isin, i.asset_class, i.currency
   191|            FROM accounts a JOIN platforms p ON p.platform_id=a.platform_id
   192|            JOIN instruments i ON i.instrument_id=? WHERE a.account_id=?
   193|            """,
   194|            (instrument_id, account_id),
   195|        ).fetchone()
   196|        rows.append({
   197|            "platform": row["platform"] if row else "",
   198|            "account_name": row["account_name"] if row else account_id,
   199|            "asset_class": row["asset_class"] if row else "",
   200|            "name": row["name"] if row else instrument_id,
   201|            "ticker": row["ticker"] if row else "",
   202|            "isin": row["isin"] if row else "",
   203|            "quantity": dec_text(pos.quantity),
   204|            "cost_basis_chf": dec_text(pos.cost_basis_chf, "0.01"),
   205|            "market_value_chf": dec_text(pos.market_value_chf, "0.01") if pos.market_value_chf is not None else "",
   206|            "total_return_chf": dec_text(pos.total_return_chf, "0.01") if pos.total_return_chf is not None else "",
   207|            "data_quality_status": pos.data_quality_status,
   208|        })
   209|    return rows
   210|
   211|
   212|def get_ledger_transactions(conn: Connection, limit: int = 200) -> list[dict[str, str]]:
   213|    return rowdicts(conn.execute(
   214|        """
   215|        SELECT t.trade_date, p.name AS platform, a.account_name, t.transaction_type,
   216|               i.name AS instrument, i.ticker, t.quantity, t.gross_amount_original,
   217|               t.net_amount_original, t.currency_original, t.fx_status, t.quality_status,
   218|               t.source_type, t.source_id
   219|        FROM transactions t
   220|        LEFT JOIN accounts a ON a.account_id=t.account_id
   221|        LEFT JOIN platforms p ON p.platform_id=a.platform_id
   222|        LEFT JOIN instruments i ON i.instrument_id=t.instrument_id
   223|        ORDER BY t.trade_date DESC, t.created_at DESC LIMIT ?
   224|        """, (limit,)
   225|    ).fetchall())
   226|
   227|
   228|def get_watchlist(conn: Connection) -> list[dict[str, str]]:
   229|    return rowdicts(conn.execute("SELECT name, asset_class, reason, status, target_entry_price, target_entry_currency, investment_case, bear_case, next_review_date FROM watchlist ORDER BY name").fetchall())
   230|
   231|
   232|def get_alerts(conn: Connection) -> list[dict[str, str]]:
   233|    return rowdicts(conn.execute(
   234|        """
   235|        SELECT priority, status, category, entity_type, entity_id, rule_id, message,
   236|               occurrence_count, last_seen_at, resolved_at, muted_until, fingerprint, dedup_key, created_at
   237|        FROM alerts
   238|        ORDER BY CASE priority WHEN 'kritisch' THEN 0 WHEN 'warnung' THEN 1 ELSE 2 END, status, created_at DESC
   239|        """
   240|    ).fetchall())
   241|
   242|
   243|def get_audit_events(conn: Connection, limit: int = 200) -> list[dict[str, str]]:
   244|    rows = conn.execute(
   245|        """
   246|        SELECT timestamp, source, action, entity_type, entity_id, old_values_json, new_values_json,
   247|               confirmed, created_by, quality_status
   248|        FROM audit_log ORDER BY timestamp DESC LIMIT ?
   249|        """, (limit,)
   250|    ).fetchall()
   251|    out: list[dict[str, str]] = []
   252|    for row in rows:
   253|        item = {k: ("" if row[k] is None else str(row[k])) for k in row.keys()}
   254|        for key in ("old_values_json", "new_values_json"):
   255|            if item[key]:
   256|                try:
   257|                    item[key] = json.dumps(json.loads(item[key]), ensure_ascii=False, indent=2, sort_keys=True)
   258|                except json.JSONDecodeError:
   259|                    pass
   260|        out.append(item)
   261|

Use offset=261 to continue reading (showing 1-260 of 292 lines)


   260|        out.append(item)
   261|    return out
   262|
   263|
   264|def get_data_quality(conn: Connection) -> list[dict[str, str]]:
   265|    items: list[dict[str, str]] = []
   266|    for row in conn.execute("SELECT COUNT(*) AS c FROM transactions WHERE fx_status='missing'"):
   267|        items.append({"check": "fehlende FX", "count": str(row["c"]), "severity": "kritisch" if row["c"] else "ok"})
   268|    for row in conn.execute("SELECT COUNT(*) AS c FROM crypto_assets WHERE coingecko_id IS NULL OR coingecko_id='' "):
   269|        items.append({"check": "fehlende CoinGecko-ID", "count": str(row["c"]), "severity": "warnung" if row["c"] else "ok"})
   270|    for row in conn.execute("SELECT COUNT(*) AS c FROM crypto_assets ca LEFT JOIN crypto_prices cp ON cp.asset_id=ca.asset_id WHERE cp.crypto_price_id IS NULL"):
   271|        items.append({"check": "fehlende Crypto-Preise", "count": str(row["c"]), "severity": "warnung" if row["c"] else "ok"})
   272|    for row in conn.execute("SELECT COUNT(*) AS c FROM alerts WHERE status='active' AND priority='kritisch'"):
   273|        items.append({"check": "aktive kritische Alerts", "count": str(row["c"]), "severity": "kritisch" if row["c"] else "ok"})
   274|    return items
   275|
   276|
   277|def get_command_center_summary(conn: Connection) -> dict[str, str]:
   278|    cash_rows = get_cash_overview(conn)
   279|    crypto_rows = get_crypto_overview(conn)
   280|    cash_total = sum(d(r["amount_chf"]) for r in cash_rows if r["amount_chf"])
   281|    crypto_total = sum(d(r["market_value_chf"]) for r in crypto_rows if r["market_value_chf"])
   282|    alert_counts = conn.execute("SELECT priority, COUNT(*) AS c FROM alerts WHERE status='active' GROUP BY priority").fetchall()
   283|    quality = get_data_quality(conn)
   284|    return {
   285|        "base_currency": "CHF",
   286|        "cash_total_chf": dec_text(cash_total, "0.01"),
   287|        "crypto_total_chf": dec_text(crypto_total, "0.01"),
   288|        "total_value_chf": dec_text(cash_total + crypto_total, "0.01"),
   289|        "active_alerts": ", ".join(f"{r['priority']}: {r['c']}" for r in alert_counts) or "0",
   290|        "data_quality_status": "kritisch" if any(i["severity"] == "kritisch" and i["count"] != "0" for i in quality) else ("warnung" if any(i["severity"] == "warnung" and i["count"] != "0" for i in quality) else "ok"),
   291|        "crypto_price_data_as_of": max((r["fetched_at"] for r in crypto_rows if r.get("fetched_at")), default=""),
   292|    }
   293|


Explanation:
**Considering database migration**

I don't think I need to read any real files since the structure is already in place. I'll need to implement a layer and use `write_file`. It seems like there might be a migration to version 4, but version 3 has already been pushed. Adding new columns to existing mapping tables might be simpler by using ALTER for the missing columns. For the new database creation, I should ensure it includes those fields. I'll also need to check the Streamlit tests pattern.


     1|from __future__ import annotations
     2|
     3|from decimal import Decimal
     4|from pathlib import Path
     5|
     6|import pytest
     7|
     8|from jarvis_finance.config.settings import load_settings
     9|from jarvis_finance.dashboard import data as dashboard_data
    10|from jarvis_finance.dashboard.demo_data import load_synthetic_demo_data
    11|from jarvis_finance.dashboard.pages import (
    12|    page_00_command_center,
    13|    page_01_portfolio,
    14|    page_04_crypto,
    15|    page_05_wallets,
    16|    page_06_ledger,
    17|    page_09_alerts,
    18|    page_10_audit,
    19|    page_11_settings_data_quality,
    20|    page_12_crypto_manage,
    21|)
    22|from jarvis_finance.storage.database import connect_memory
    23|from jarvis_finance.storage.migrations import apply_migrations
    24|
    25|
    26|class FakeStreamlit:
    27|    def __init__(self) -> None:
    28|        self.calls: list[tuple[str, tuple[object, ...], dict[str, object]]] = []
    29|        self.sidebar = self
    30|
    31|    def __getattr__(self, name: str):
    32|        def recorder(*args, **kwargs):
    33|            self.calls.append((name, args, kwargs))
    34|            if name in {"button", "checkbox"}:
    35|                return False
    36|            if name in {"selectbox", "radio"}:
    37|                options = args[1] if len(args) > 1 else kwargs.get("options", [])
    38|                return list(options)[0] if options else None
    39|            if name == "tabs":
    40|                return [self for _ in (args[0] if args else [])]
    41|            if name == "columns":
    42|                count = args[0] if args else 1
    43|                if isinstance(count, int):
    44|                    return [self for _ in range(count)]
    45|                return [self for _ in count]
    46|            return None
    47|        return recorder
    48|
    49|    def __enter__(self):
    50|        return self
    51|
    52|    def __exit__(self, *args):
    53|        return False
    54|
    55|
    56|def setup_demo_conn():
    57|    conn = connect_memory()
    58|    apply_migrations(conn)
    59|    load_synthetic_demo_data(conn)
    60|    return conn
    61|
    62|
    63|def test_dashboard_modules_import_and_page_render_smoke() -> None:
    64|    conn = setup_demo_conn()
    65|    st = FakeStreamlit()
    66|    for module in [
    67|        page_00_command_center,
    68|        page_01_portfolio,
    69|        page_04_crypto,
    70|        page_05_wallets,
    71|        page_06_ledger,
    72|        page_09_alerts,
    73|        page_10_audit,
    74|        page_11_settings_data_quality,
    75|        page_12_crypto_manage,
    76|    ]:
    77|        module.render(st, conn)
    78|    assert any(call[0] in {"title", "header", "subheader"} for call in st.calls)
    79|
    80|
    81|def test_dashboard_reader_functions_return_expected_dummy_data() -> None:
    82|    conn = setup_demo_conn()
    83|    summary = dashboard_data.get_command_center_summary(conn)
    84|    assert summary["base_currency"] == "CHF"
    85|    assert Decimal(summary["cash_total_chf"]) > Decimal("0")
    86|    assert Decimal(summary["crypto_total_chf"]) > Decimal("0")
    87|    assert dashboard_data.get_platform_overview(conn)
    88|    assert dashboard_data.get_ledger_transactions(conn)
    89|    assert dashboard_data.get_audit_events(conn)
    90|
    91|
    92|def test_crypto_dashboard_reads_cached_db_prices_without_api_calls(monkeypatch: pytest.MonkeyPatch) -> None:
    93|    def explode(*args, **kwargs):
    94|        raise AssertionError("dashboard must not call CoinGecko/API provider on render")
    95|
    96|    monkeypatch.setattr("jarvis_finance.market.providers.CoinGeckoClient.get_crypto_price", explode)
    97|    conn = setup_demo_conn()
    98|    st = FakeStreamlit()
    99|    page_04_crypto.render(st, conn)
   100|    rows = dashboard_data.get_crypto_overview(conn)
   101|    assert rows
   102|    assert any(row["latest_price_chf"] for row in rows)
   103|
   104|
   105|def test_dashboard_preserves_decimal_crypto_quantity_format() -> None:
   106|    conn = setup_demo_conn()
   107|    wallet_rows = dashboard_data.get_crypto_by_wallet(conn)
   108|    quantities = {row["quantity"] for row in wallet_rows}
   109|    assert "0.0899" in quantities
   110|    for row in wallet_rows:
   111|        assert "E" not in row["quantity"].upper()
   112|
   113|
   114|def test_alert_lifecycle_data_loaded_readably() -> None:
   115|    conn = setup_demo_conn()
   116|    alerts = dashboard_data.get_alerts(conn)
   117|    assert alerts
   118|    assert {"status", "priority", "occurrence_count", "last_seen_at", "fingerprint", "dedup_key"} <= set(alerts[0])
   119|
   120|
   121|def test_dashboard_runtime_path_inside_repo_remains_blocked(tmp_path: Path) -> None:
   122|    repo = tmp_path / "repo"
   123|    repo.mkdir()
   124|    (repo / "pyproject.toml").write_text("[project]\nname='demo'\n")
   125|    with pytest.raises(ValueError, match="Runtime directory must be outside"):
   126|        load_settings(repo_root=repo, environ={"JARVIS_FINANCE_RUNTIME_DIR": str(repo / "runtime")})
   127|


Explanation:
**Fixing import issues**

I need to ensure that I'm importing smoke properly. The REQUIRED_TABLES schema doesn't need any special patches, which is a relief. I should set the migration version to 4 and provide a name for it. To patch everything smoothly, I think I might find that using execute_code is a better approach. It could simplify the process, making it easier for me to keep everything on track. I'll stay focused on making this work!

from pathlib import Path
p=Path('/home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/storage/migrations.py')
s=p.read_text()
s=s.replace('MIGRATION_VERSION = 3\nMIGRATION_NAME = "003_broker_bank_mapping_foundation"','MIGRATION_VERSION = 4\nMIGRATION_NAME = "004_import_wizard_mapping_layer"')
old='''def _create_broker_bank_mapping_tables(conn: Connection) -> None:\n    conn.executescript(\n        """\n        CREATE TABLE IF NOT EXISTS instrument_mappings (\n            mapping_id TEXT PRIMARY KEY,\n            source_name TEXT NOT NULL,\n            source_instrument_name TEXT,\n            source_isin TEXT,\n            source_ticker TEXT,\n            source_exchange TEXT,\n            source_currency TEXT,\n            instrument_id TEXT REFERENCES instruments(instrument_id),\n            mapping_status TEXT NOT NULL DEFAULT 'unmapped',\n            quality_flags_json TEXT,\n            notes TEXT,\n            created_at TEXT NOT NULL,\n            updated_at TEXT,\n            UNIQUE(source_name, source_isin, source_ticker, source_instrument_name, source_currency)\n        );\n\n        CREATE TABLE IF NOT EXISTS platform_account_mappings (\n            mapping_id TEXT PRIMARY KEY,\n            source_name TEXT NOT NULL,\n            source_platform_label TEXT,\n            source_account_label TEXT,\n            source_account_type TEXT,\n            source_currency TEXT,\n            platform_id TEXT REFERENCES platforms(platform_id),\n            account_id TEXT REFERENCES accounts(account_id),\n            mapping_status TEXT NOT NULL DEFAULT 'unmapped',\n            quality_flags_json TEXT,\n            notes TEXT,\n            created_at TEXT NOT NULL,\n            updated_at TEXT,\n            UNIQUE(source_name, source_platform_label, source_account_label, source_currency)\n        );\n\n        CREATE TABLE IF NOT EXISTS broker_import_dry_runs (\n            dry_run_id TEXT PRIMARY KEY,\n            source_name TEXT NOT NULL,\n            source_file_label TEXT NOT NULL,\n            source_file_type TEXT NOT NULL,\n            source_hash TEXT,\n            detected_sections_json TEXT NOT NULL,\n            field_coverage_json TEXT NOT NULL,\n            quality_flags_json TEXT NOT NULL,\n            rows_total INTEGER NOT NULL DEFAULT 0,\n            rows_position_candidates INTEGER NOT NULL DEFAULT 0,\n            rows_cash_candidates INTEGER NOT NULL DEFAULT 0,\n            created_at TEXT NOT NULL,\n            notes TEXT\n        );\n        """\n    )\n'''
new='''def _add_missing_columns(conn: Connection, table: str, columns: dict[str, str]) -> None:\n    existing = _table_columns(conn, table)\n    for name, col_type in columns.items():\n        if name not in existing:\n            conn.execute(f"ALTER TABLE {table} ADD COLUMN {name} {col_type}")\n\n\ndef _create_broker_bank_mapping_tables(conn: Connection) -> None:\n    conn.executescript(\n        """\n        CREATE TABLE IF NOT EXISTS instrument_mappings (\n            mapping_id TEXT PRIMARY KEY,\n            source_name TEXT,\n            source_platform TEXT NOT NULL,\n            source_account TEXT,\n            source_label TEXT NOT NULL,\n            normalized_name TEXT NOT NULL,\n            isin TEXT,\n            ticker TEXT,\n            exchange TEXT,\n            currency TEXT,\n            asset_class TEXT NOT NULL DEFAULT 'other',\n            source_instrument_name TEXT,\n            source_isin TEXT,\n            source_ticker TEXT,\n            source_exchange TEXT,\n            source_currency TEXT,\n            instrument_id TEXT REFERENCES instruments(instrument_id),\n            mapping_status TEXT NOT NULL DEFAULT 'needs_manual_review',\n            confidence TEXT NOT NULL DEFAULT '0',\n            quality_flags_json TEXT,\n            notes TEXT,\n            created_at TEXT NOT NULL,\n            updated_at TEXT\n        );\n\n        CREATE TABLE IF NOT EXISTS platform_account_mappings (\n            mapping_id TEXT PRIMARY KEY,\n            source_name TEXT,\n            source_platform TEXT NOT NULL,\n            source_platform_label TEXT,\n            source_account_label TEXT NOT NULL,\n            normalized_platform TEXT NOT NULL,\n            normalized_account_name TEXT NOT NULL,\n            source_account_type TEXT,\n            account_type TEXT NOT NULL DEFAULT 'other',\n            source_currency TEXT,\n            currency TEXT,\n            platform_id TEXT REFERENCES platforms(platform_id),\n            account_id TEXT REFERENCES accounts(account_id),\n            internal_platform_id TEXT REFERENCES platforms(platform_id),\n            internal_account_id TEXT REFERENCES accounts(account_id),\n            mapping_status TEXT NOT NULL DEFAULT 'needs_manual_review',\n            quality_flags_json TEXT,\n            notes TEXT,\n            created_at TEXT NOT NULL,\n            updated_at TEXT\n        );\n\n        CREATE TABLE IF NOT EXISTS broker_import_dry_runs (\n            dry_run_id TEXT PRIMARY KEY,\n            source_name TEXT,\n            source_platform TEXT NOT NULL,\n            source_file_label TEXT,\n            source_file_type TEXT NOT NULL,\n            source_hash TEXT,\n            source_filename_hash TEXT,\n            detected_snapshot_date TEXT,\n            snapshot_date_status TEXT NOT NULL DEFAULT 'missing',\n            detected_sections_json TEXT NOT NULL,\n            field_coverage_json TEXT,\n            rows_total INTEGER NOT NULL DEFAULT 0,\n            rows_position_candidates INTEGER NOT NULL DEFAULT 0,\n            rows_cash_candidates INTEGER NOT NULL DEFAULT 0,\n            candidate_positions INTEGER NOT NULL DEFAULT 0,\n            candidate_cash_rows INTEGER NOT NULL DEFAULT 0,\n            mapped_positions INTEGER NOT NULL DEFAULT 0,\n            blocked_positions INTEGER NOT NULL DEFAULT 0,\n            warnings_count INTEGER NOT NULL DEFAULT 0,\n            errors_count INTEGER NOT NULL DEFAULT 0,\n            quality_flags_json TEXT NOT NULL,\n            summary_json TEXT NOT NULL DEFAULT '{}',\n            created_at TEXT NOT NULL,\n            notes TEXT\n        );\n        """\n    )\n    _add_missing_columns(conn, "instrument_mappings", {\n        "source_name": "TEXT",\n        "source_platform": "TEXT",\n        "source_account": "TEXT",\n        "source_label": "TEXT",\n        "normalized_name": "TEXT",\n        "isin": "TEXT",\n        "ticker": "TEXT",\n        "exchange": "TEXT",\n        "currency": "TEXT",\n        "asset_class": "TEXT DEFAULT 'other'",\n        "confidence": "TEXT DEFAULT '0'",\n    })\n    _add_missing_columns(conn, "platform_account_mappings", {\n        "source_platform": "TEXT",\n        "normalized_platform": "TEXT",\n        "normalized_account_name": "TEXT",\n        "account_type": "TEXT DEFAULT 'other'",\n        "currency": "TEXT",\n        "internal_platform_id": "TEXT",\n        "internal_account_id": "TEXT",\n    })\n    _add_missing_columns(conn, "broker_import_dry_runs", {\n        "source_platform": "TEXT",\n        "source_filename_hash": "TEXT",\n        "detected_snapshot_date": "TEXT",\n        "snapshot_date_status": "TEXT DEFAULT 'missing'",\n        "candidate_positions": "INTEGER DEFAULT 0",\n        "candidate_cash_rows": "INTEGER DEFAULT 0",\n        "mapped_positions": "INTEGER DEFAULT 0",\n        "blocked_positions": "INTEGER DEFAULT 0",\n        "warnings_count": "INTEGER DEFAULT 0",\n        "errors_count": "INTEGER DEFAULT 0",\n        "summary_json": "TEXT DEFAULT '{}'",\n    })\n'''
if old not in s:
    raise SystemExit('old block not found')
p.write_text(s.replace(old,new))
print('patched migrations')


Output: patched migrations