finance-system-architecture

---
name: finance-system-architecture
description: "Design/implement the user's JARVIS Finance System: CHF ledger, portfolio accounting, dashboard, imports, reports, secrecy."
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [finance, portfolio, accounting, dashboard, csv, fx, tax, privacy]
---

# Finance System Architecture

## When to use

Use this skill whenever the user asks about the **Finance System – Architektur & Scripts** topic or work involving the user's personal finance-management system: portfolio imports, transaction accounting, Bank/Broker/CSV processing, dashboards, reports, tax/income views, FX logic, market data jobs, alerts, or later implementation.

Do **not** use it for pure investment-strategy discussion; those belong in the separate finance strategy topic unless the user explicitly asks for technical implementation.

## Core principles

1. **No automated trading.** The system may prepare orders and recommendations, but never executes orders or connects to broker trading APIs for execution.
2. **Accounting first, opinions later.** MVP must prioritize correct transaction ledger, cash ledger, FX handling, income/tax data, and dashboard before complex scores, news, analyst ratings, or LLM analysis.
3. **Rule-based first.** Calculations, scores, alerts, rebalancing and reports must be deterministic and auditable. LLMs may only summarize/explain text or assist extraction with human review.
4. **CHF as base currency.** Store every transaction in original currency and CHF using historical FX at transaction/payment date; current valuations use current FX.
5. **Financial data secrecy.** Never expose real financial values, portfolio files, raw exports, reports with real numbers, DBs, API keys, or credentials in chat, GitHub, logs, or shared docs unless the user explicitly approves scope.
6. **Data ownership.** All data must be exportable: CSV, JSON, Parquet, SQLite/Postgres dump, Markdown/HTML/PDF reports.

## Current v0.3 decisions

See `references/spec-v0.2-decisions.md` for the baseline, `references/spec-v0.3-decisions.md` for the priority update, `references/spec-v0.4-implementation-blueprint.md` for the MVP 1 technical implementation blueprint, `references/spec-v0.4.1-corrections.md` for accepted corrections, `references/spec-v0.5-repo-blueprint.md` for the concrete repo/implementation plan, `references/mvp-safety-gate-implementation.md` for the implementation safety-gate sequence and verification commands, `references/mvp-import-pipeline-tasks-9-13.md` for the synthetic CSV import/FX/transaction pipeline pattern, `references/mvp-ledger-cash-wac-positions.md` for cash/WAC/position ledger calculation rules and tests, `references/mvp-crypto-wallet-ledger.md` for crypto wallet/assets/holdings/transfers/buy-sell/fee MVP rules, `references/crypto-xlsx-initial-snapshot-dry-run.md` for the controlled real-file XLSX crypto initial-snapshot dry-run pattern and mapping decisions, `references/crypto-xlsx-initial-snapshot-production-import.md` for productive safe-subset XLSX import operations, compatibility/partial-write recovery, and dashboard correction follow-up, `references/crypto-single-coin-runtime-test.md` for controlled one-coin productive runtime tests and the coin-vs-holding ambiguity guard, `references/multi-wallet-crypto-and-equity-manage-ui.md` for approved multi-wallet single-coin snapshots, SQLite Decimal/TEXT affinity pitfalls, and synthetic-first Equity/ETF Manage UI patterns, `references/crypto-price-refresh-and-idempotent-btc-runtime.md` for robust batch price refresh and re-run-safe BTC/single-coin runtime checks, `references/crypto-runtime-db-consistency-verification.md` for post-import runtime DB consistency checks, exact safe-set verification, dashboard no-live-API testing, and runtime-only report generation, `references/mvp-crypto-manual-management-ui.md` for audited dashboard add/correct/remove crypto holding workflows after an initial snapshot, `references/mvp-market-import-safety-hardening.md` for CoinGecko provider/backoff, CSV importer completion, alert lifecycle, Git-safety and crypto Decimal precision patterns, `references/git-safety-verification-coverage.md` for the verifier-warning workflow and minimum Git-safety test matrix, `references/mvp-streamlit-dashboard.md` for the safe local Streamlit dashboard shell/read-model/testing pattern, `references/mvp-crypto-report-export.md` for local-only crypto report context/export rules, `references/github-token-push-from-drive.md` for the safe Google-Drive-token push workflow when local FinanceManager commits are ahead of GitHub, `references/runtime-backup-and-broker-mapping.md` for runtime DB backup/restore hardening plus PostFinance/True Wealth/Raiffeisen structural mapping/dry-run rules, `references/import-wizard-mapping-layer.md` for the Phase B1 import wizard, instrument/account mapping, review queue, data-quality and read-only safety pattern, and `references/broker-parser-dry-run-pipeline.md` for the Phase B2 synthetic broker parser, dry-run, runtime review-item and UI quality-flag pipeline. Key decisions:

- Platform/account views: Raiffeisen, PostFinance, True Wealth, Crypto-Coins, plus Gesamtansicht.
- Existing Google Drive files under `Finanzen/01 Aktueller Stand Portfolios` are **initial-snapshot reality checks**, not full ledgers.
- MVP means the smallest useful first version: correct data foundation, ledger, CHF/FX logic, platform views, crypto inventory, watchlist, reports, workflows and audit log.
- Stock/ETF tax documents are downgraded: optional reference/control module and dividend/distribution transparency, not MVP 1 and not a complete tax dossier replacement.
- Crypto inventory is promoted to MVP 1: wallet/platform-level holdings, Crypto PDF, CoinGecko valuation, stale-price warnings and manual verification.
- Crypto snapshot values are stale; use quantities only and recalculate current values via CoinGecko later.
- MVP dashboard recommendation: Streamlit + SQLite + SQLAlchemy/Pandas/Plotly; later migrateable to FastAPI + React/PostgreSQL.
- MVP inputs: manual dashboard entry, standard CSV import/export, later broker/bank export import. No direct broker integrations initially.
- Telegram crypto buys/sells/transfers should be parsed as pending transactions and only saved after explicit confirmation.
- Every portfolio or crypto change must create a History/Audit entry.

## Required data model areas

At minimum design for:

- `platforms`, `accounts`, `instruments`
- `transactions` with original currency and CHF fields
- `cash_ledger`
- derived `positions_current`
- `dividends_distributions`
- `market_prices`, `fx_rates`
- `watchlist_items`
- `alerts`
- `decision_journal`
- `reports`
- `data_quality_issues`

## CSV/import rules

- For broker/bank imports (PostFinance, True Wealth, Raiffeisen), build the safe review layer before productive imports: Import Wizard, instrument mappings, platform/account mappings, dry-run summaries, Manual Review Queue, Data Quality Center and visible read-only/write-enabled state. Follow `references/import-wizard-mapping-layer.md`. When wiring parser output into the wizard, use the synthetic-first parser/dry-run/review-item pipeline in `references/broker-parser-dry-run-pipeline.md`.
- Do not pivot into later analytics modules while the user has asked for import/mapping hardening. Monte Carlo, backtesting, rebalancing, scores, heatmaps and projections are later phases unless explicitly re-authorized.
- Use a canonical CSV format before broker-specific importers.
- Imports must be idempotent via external transaction ID or row hash.
- Importers must separate dry-run and commit modes, collect validation errors explicitly, and always create an `import_sessions` record even for dry runs or failed imports.
- Import results should classify rows as new, existing, duplicate, or failed; do not rely on `INSERT OR IGNORE` as the only duplicate-detection mechanism.
- Row hashes must be stable and deterministic: normalize key order, whitespace, and `None` versus empty strings while leaving domain-specific canonicalization to the importer.
- If a source file has only current holdings and no full history, import it as `initial_position_snapshot` / `initial_cash_snapshot`, not fake historical buys.
- Missing historic FX, fees, taxes, dates, investment case, target weights, or dividend details must be flagged as data-quality gaps.

## FX and return rules

For foreign currency assets, always distinguish:

- price/security performance in original currency
- FX effect to CHF
- realized P&L
- unrealized P&L
- dividends/distributions/income
- fees and taxes
- Total Return in CHF

Every transaction/income event needs:

- original amount and currency
- historical FX rate to CHF, or explicit `fx_status=missing` with a critical data-quality alert
- FX source and timestamp
- CHF equivalent at event date when FX is known

Manual FX overrides require a non-empty note and an audit-log entry.

## Ledger calculation rules

- Cash after system start must be calculated from `transactions`; `cash_balances` is only a snapshot/cache/control view.
- `initial_cash_snapshot` and `initial_position_snapshot` are explicit starting truths, not fabricated historical trades.
- Implement Weighted Average Cost as the MVP internal performance method: buy fees increase cost basis, partial sells remove proportional cost basis, realized P&L is calculated against average cost, full sells zero remaining quantity/cost basis, and dividends do not alter quantity or cost basis.
- WAC is not tax-complete; do not add FIFO/LIFO/tax-lot logic in MVP 1, but keep transaction granularity so it can be added later.
- Position calculations must keep zero-quantity historical positions visible when sale history exists.
- Missing market price or FX must degrade data quality and suppress precise Total Return CHF rather than inventing a value.
- Before productive dashboard use, add alert deduplication per rule/entity/date.
- Market-FX for non-CHF market prices belongs to the later Market/FX provider block.

## Crypto MVP rules

- Crypto inventory is MVP-critical but must remain dummy-first until mapper/dry-run layers are ready; do not touch real coin/wallet files during core implementation. After explicit user approval, real files may be read only from temp/runtime for controlled dry runs using `references/crypto-xlsx-initial-snapshot-dry-run.md`; report only aggregate counts/labels and never quantities or values. Productive safe-subset imports require a second explicit approval and must follow `references/crypto-xlsx-initial-snapshot-production-import.md`: external runtime DB only, no real CSV/XLSX/reports/DBs in Git, Git-safety before/after, import-session + audit-log creation, and aggregate-only reporting.
- Wallet names must be unique; wallet addresses are optional and must not appear in tests/fixtures.
- Validate wallet type against: Hardware Wallet, Software Wallet, Exchange, Bank/Broker, DeFi, Sonstiges.
- CoinGecko ID is strongly recommended but not hard-required; missing ID creates a data-quality warning and blocks reliable valuation.
- Crypto symbols are not globally unique; symbol conflicts require warning/manual selection, never silent merging.
- Initial crypto holdings are confirmed snapshots by wallet+asset. Store legacy values for provenance/control only, never current valuation.
- Crypto holdings should be calculated/controlled from initial snapshots plus crypto transactions, not silently overwritten.
- Crypto transfers require different source/target wallets, quantity > 0, and same asset; coin fees reduce source-wallet balance.
- Negative wallet balances should be blocked in MVP and produce a critical alert unless a future explicit correction workflow is implemented.
- Crypto buys/sells with fiat impact must create/reference general ledger transactions via `source_type='crypto'`/`source_id` so cash remains ledger-consistent.
- Fiat fees should be ledger fees when unambiguous; avoid double counting when a fee is already embedded in sell net proceeds.
- Unknown fee currency creates a data-quality warning instead of invented accounting.
- Crypto Cost Basis/WAC is not required for MVP 1; MVP 1 prioritizes crypto holdings, wallet structure, current valuation, reports and audit.
- Crypto report context/export must be renderer-neutral and local-only: build from SQLite, aggregate by coin and wallet, use local cached prices only, preserve Decimal/Text values, ignore legacy snapshot values for current valuation, include data-quality warnings, and write generated files only to the external runtime reports directory.
- Crypto PDF export should be optional-dependency tolerant: prefer PDF when a renderer is available, otherwise fall back to HTML/Markdown with a clear warning while still recording report metadata and audit.
- After productive real-data crypto imports, run the post-import runtime verification in `references/crypto-runtime-db-consistency-verification.md`: exact safe-set membership checks, aggregate-only counts, audit/import-session traceability, dashboard render with no live API, runtime-only report generation, and Git-safety before/after.
- Manual dashboard additions/corrections/removals after an initial snapshot must follow `references/mvp-crypto-manual-management-ui.md`: no silent `crypto_holdings` mutation, explicit confirmation, mandatory notes for adjustments/removals, audit-log entries with old/new values, Decimal/TEXT quantity handling, no negative holdings, and no live API lookup during render.
- Crypto tax logic, tax lots, FIFO/LIFO and detailed cost basis are later modules.
- Keep crypto assets separate from `instruments` for MVP 1, but later evaluate a shared Asset Master if it improves reporting, performance views or watchlists.

## Income/tax rules

For dividends and ETF distributions, separately store:

- gross dividend/distribution
- Swiss withholding tax 35%
- foreign withholding tax
- other tax/withholding if present
- net amount
- original currency
- historic FX to CHF
- gross/net CHF equivalents
- platform/account
- ISIN/ticker
- payment date
- ex-dividend date if available

Plan later ESTV/ICTax support for Swiss year-end values and tax reports, but label outputs as technical reports, not tax advice.

## Market-data rules

- Dashboard reads primarily from local DB, not directly from APIs on page load.
- First Streamlit MVP should be a safe, local, read-first shell over SQLite with explicit synthetic demo DB creation only; page render functions must not call CoinGecko/market APIs, and tests should prove that by monkeypatching provider methods to fail.
- Use scheduled jobs to update prices/FX/news.
- Store provider, timestamp, quality status and freshness for every price/FX point.
- Implement caching, local historical storage, batch requests where possible, provider rate-limit queues, and backoff on HTTP 429.
- CoinGecko integrations must be mockable and unit-tested without real API calls; treat HTTP 429 as non-fatal and return a non-fresh data-quality status plus alert after configured retries.
- Crypto price refresh belongs in CLI/scheduled-job commands, not dashboard/report rendering. Prefer CoinGecko batch requests over per-coin requests; expose conservative controls (`--sleep-seconds`, `--max-retries`, `--initial-backoff`, `--max-backoff`, `--only-missing`, `--only-stale`, `--only-symbol`, `--limit`, `--dry-run`, `--currency`). Commands should use aggregate-only output (`total_assets/updated/cached/skipped/stale/warnings/errors/assets_still_missing_local_price`), cache by configurable `max_age_seconds`, store exact Decimal prices as TEXT in `crypto_prices`, and verify dashboard/report contexts read local prices with `live_api_calls=False`. For details and tests see `references/crypto-price-refresh-and-idempotent-btc-runtime.md`.
- Candidate providers: Finnhub, Financial Modeling Prep, Twelve Data, Stooq, Alpha Vantage selectively, Yahoo Finance fallback only; CoinGecko for crypto; later DefiLlama; ESTV/ICTax for Swiss tax values.

## Crypto Decimal precision rules

SQLite has no exact native DECIMAL. In the crypto context, store quantities, prices, crypto fees and valuation-relevant amounts as exact Decimal strings (`TEXT`) and compute with Python `Decimal`. Never convert crypto quantity/price/fee/valuation through float/REAL. Format Decimal writes with `format(value, "f")` so small quantities such as `0.00000001` do not become scientific notation. CSV importers must validate Decimal strings, and tests should assert high-precision preservation plus SQLite text storage. For multi-wallet real-data snapshots and Equity/ETF ledger fields, also see `references/multi-wallet-crypto-and-equity-manage-ui.md`; SQLite `NUMERIC` affinity can coerce Decimal-looking text to `real`, so schema/migrations should use `TEXT` for exact quantities, FX rates, original amounts and CHF equivalents where precision matters.

## Alert lifecycle rules

Alerts must be deduplicated and lifecycle-aware before dashboards run repeated calculations. Use a dedup key from `rule_id`, `entity_type`, `entity_id`, `priority` and an error fingerprint. Alerts should carry `status` (`active`, `resolved`, `muted`), `resolved_at`, optional `muted_until`, `last_seen_at`, `occurrence_count`, `fingerprint` and `dedup_key`. Re-seeing an identical active/resolved/muted alert should update `last_seen_at`/`occurrence_count` rather than create a new active row; changed fingerprints may create a new alert.

## Git/GitHub safety

Code/scripts/docs may be stored in a private Git repo, but never commit financial data. Use the `github-repo-management` skill and its `references/sensitive-code-backup.md` guidance.

Minimum `.gitignore` for finance repos must block `data/`, `reports/`, `exports/`, `backups/`, DB files, spreadsheets, PDFs, DOCX, raw CSV/JSON, `.env`, tokens, credentials and client secrets. Only synthetic examples under `examples/` are acceptable.

## MVP implementation safety gate

Before any real broker/bank/portfolio/crypto data is imported, build and verify the safety foundation described in `references/mvp-safety-gate-implementation.md`: repo skeleton, `.gitignore`, `.env.example`, runtime outside repo, settings validation, SQLite schema, synthetic fixtures, audit log, CSV dry-run, duplicate protection, and Git-safety scan. Run the safety scan before staging, committing, or pushing. If GitHub credentials are needed from Drive, use them only in a temporary location and never echo token values.

Before broader productive imports, implement or verify the runtime backup loop in `references/runtime-backup-and-broker-mapping.md`: backup outside Git, SHA256 verification, restore requiring explicit confirmation, tests for backup/restore/checksum, and Git-safety coverage that blocks backup artifacts in the repo. For PostFinance/True Wealth/Raiffeisen, perform structural mapping and dry-runs only until the user approves import; report aggregate counts/field coverage/quality flags only, never real values or raw rows.

## Common pitfalls

- Starting later analytics (Monte Carlo, backtesting, rebalancing, scoring, heatmaps, projections) while import/mapping/data-quality phases are still being hardened or while the user explicitly asked to postpone them.
- Treating a holdings snapshot as a real transaction history.
- Mixing price P&L and FX P&L.
- Letting dashboard refreshes hammer APIs.
- Committing realistic sample data to Git because the repo is private. Private GitHub is not a financial data vault.
- Showing real portfolio values back to the user unnecessarily in summaries.
- Designing Git-safety scans that block source package names like `src/.../imports` or `src/.../reports`; block top-level runtime/data directories instead.
- Running compile/tests before Git-safety and forgetting generated `__pycache__`; remove or ignore generated caches before safety scans. `pytest` may recreate `.pytest_cache`, so remove `__pycache__`/`.pytest_cache` again after test execution and before `git-safety-scan`.
- Productive real-data imports can partially write before surfacing schema/importer compatibility issues. Do not rerun blindly or dump row-level data; inspect aggregate counts and unique identifiers needed for idempotency, patch/complete via missing-only inserts, then verify committed import session, audit log, Git clean and Git-safety OK.
- In controlled one-coin runtime tests from a real XLSX, distinguish **one coin/asset** from **one holding/wallet row**. A coin may be split across multiple wallet columns; if the user required an unambiguous target wallet or one holding, stop and ask rather than choosing a wallet silently. If the user explicitly approves multi-wallet import for that coin, import exactly one asset plus one audited holding per non-empty approved wallet cell; verify aggregate counts only. If re-running a BTC/single-coin test finds the exact approved asset and holdings already present, do not import duplicates; perform the aggregate-only idempotency checks and report that asset/holding counts did not increase because duplicate prevention succeeded. See `references/crypto-single-coin-runtime-test.md`, `references/multi-wallet-crypto-and-equity-manage-ui.md`, and `references/crypto-price-refresh-and-idempotent-btc-runtime.md`.
- After productive crypto imports, do a separate aggregate-only consistency pass before building new features: DB/runtime outside repo, exact safe-subset membership, no duplicate/negative/orphan holdings, correct snapshot date, null legacy-value fields when ignored, import sessions committed, holding-level audit traces present, dashboard local-read/no-live-API verified, and report generated only under runtime reports.
- Letting a dashboard correction UI mutate crypto holdings directly. Add/remove/edit operations must become audited transactions or `manual_adjustment` events with mandatory notes where appropriate; preserve history instead of silently overwriting balances.
- Treating verifier/file-mutation warnings as either automatically true or automatically false. Check the exact commit with `git show --name-status --patch <commit> -- <file>`, then separately verify whether the test matrix actually covers the promised cases.
- Adding broad Git-safety tests that only assert `len(findings) >= N`; prefer explicit assertions for token filename, PAT patterns, private keys, CSV/JSON path policy, runtime/data directories, allowed synthetic paths, `config.example/`, and normal `src/` false-positive protection.

To view linked files, call skill_view(name, file_path) where file_path is e.g. 'references/api.md' or 'assets/config.yaml'


# Broker Parser Dry-Run Pipeline Pattern

Use this reference after the Import Wizard / Mapping Layer exists, when adding parser output to the review workflow for PostFinance, True Wealth, Raiffeisen, or similar broker/bank statements.

## Purpose

Turn broker/bank parser results into safe dry-run summaries and runtime-only review items. This phase must still avoid productive ledger, snapshot, holdings, or cash writes unless the user explicitly approves a later import phase.

## Sequencing and guardrails

1. Start with synthetic DOCX/XLSX fixtures that mimic real source structure.
2. Do not commit real broker/bank files, raw rows, quantities, market values, or generated runtime DBs.
3. Real files may only be read from runtime/temp for structural dry-runs; report aggregate counts and quality flags only.
4. Keep Import Wizard and review actions read-only by default.
5. Do not pivot into analytics modules such as Monte Carlo, backtesting, rebalancing, scores, projections, or heatmaps during import hardening.

## Parser behavior by source

### PostFinance DOCX/table parser

Extract candidates from DOCX table rows:

- product/name label
- quantity-present boolean, not quantity value in chat/docs
- currency
- market-value-present boolean
- account/depot context
- snapshot date when recognizable

Expected quality flags:

- `missing_isin`
- `missing_ticker`
- `snapshot_only`
- `cost_basis_uncertain`
- `market_value_legacy`
- `needs_manual_review`

### True Wealth DOCX/mixed-table parser

Extract:

- cash rows
- position-like rows
- portfolio/account context
- currency
- snapshot date when recognizable

Expected quality flags:

- `missing_isin`
- `missing_ticker`
- `snapshot_only`
- `cost_basis_uncertain`
- `missing_fx` for non-CHF rows without FX
- `needs_manual_review`

### Raiffeisen XLSX account/control parser

Extract:

- cash/account rows
- aggregate custody/investment rows

Rules:

- Treat Raiffeisen defensively as cash/control snapshot when detailed instrument fields are absent.
- Do not create instrument positions from aggregate-only rows.

Expected quality flags:

- `cash_snapshot_candidate`
- `aggregate_only`
- `missing_instrument_details`
- `needs_manual_review`

## Runtime-only review items

Add/maintain `broker_import_review_items` for row-level review data in the runtime DB only.

Fields:

- `review_item_id`
- `dry_run_id`
- `source_platform`
- `source_file_type`
- `source_row_ref`
- `row_hash`
- `source_label`
- `normalized_name`
- `detected_asset_class`
- `detected_currency`
- `detected_quantity_present`
- `detected_market_value_present`
- `isin`
- `ticker`
- `exchange`
- `mapped_instrument_id`
- `mapped_account_id`
- `quality_flags_json`
- `review_status`: open, mapped, ignored, blocked, resolved
- `reviewer_note`
- timestamps

Rules:

- Review item details are runtime-only and must not be written to Git fixtures/docs when they come from real files.
- In chat/reporting, output only aggregate counts and quality flag summaries.
- Manual mapping or status changes must require a non-empty note and create an audit-log entry.

## Generic dry-run pipeline

1. Parse source file.
2. Normalize candidates.
3. Attempt instrument mapping using existing mapping rules: ISIN primary; ticker+exchange fallback; name-only never auto-maps.
4. Attempt account mapping when account context exists.
5. Set quality flags.
6. Write `broker_import_dry_runs` aggregate summary.
7. Write `broker_import_review_items` row-level runtime review records.
8. Do not write `positions_snapshot`.
9. Do not write transactions, cash ledger, crypto holdings, or productive snapshots.
10. Return aggregate output: `rows_total`, `candidate_positions`, `candidate_cash_rows`, `mapped_positions`, `blocked_positions`, `review_items_created`, `warnings_count`, `errors_count`, `quality_flags_summary`.

## UI extensions

Import Wizard should show parser/dry-run summary, quality flag counts, runtime review items, and a visible link/path to Manual Review Queue. Productive commit buttons remain disabled in read-only mode.

Manual Review Queue should show review items with source, label, normalized name, asset class, currency, ISIN, ticker, exchange, mapped instrument/account, quality flags, and status. Prepare actions for adding identifiers, confirming mappings, ignoring with note, or resolving, but keep productive position creation disabled.

Data Quality Center should aggregate open review items by quality flag, including `missing_isin`, `missing_ticker`, `snapshot_only`, `cost_basis_uncertain`, `missing_fx`, `aggregate_only`, plus existing stale/missing price and alert lifecycle checks.

Global Safe Mode should be visible in the dashboard shell or Command Center: DB mode, write mode, last backup if known, Git-safety status if known, and reminder that real data lives outside Git. Default is read-only.

## Test matrix additions

Add tests for:

- synthetic PostFinance parser candidate detection and flags
- synthetic True Wealth parser candidate detection and flags
- synthetic Raiffeisen cash/control and aggregate-only detection
- dry-run summary creation
- review item creation
- no productive positions/ledger writes
- name-only rows stay review-gated
- Raiffeisen aggregate-only rows are blocked, not instrument positions
- Manual Review Queue loads review items
- Data Quality Center aggregates review flags
- default safe mode is read-only
- commit actions remain disabled
- audit log on review-item mapping/status change
- no real DOCX/XLSX/PDF files in the repo
- Git-safety remains OK


Explanation:
**Searching for CLI backup commands**

I need to get started on running search queries to find the command line interface (CLI) backup commands. It sounds like a straightforward task, but I'm making sure to cover my bases and not miss any critical commands. I'm actually curious to see the variety of commands available for backups in CLI, as that might help me understand the best practices. Let’s dig into it and see what I can find!

todo

Preflight: Git, Git-Safety, Runtime-Pfade, Remote-Sync, Safe Mode prüfen

Runtime-DB Backup erzeugen und verifizieren

Echte PostFinance/True Wealth/Raiffeisen Dateien temporär finden und Dry-Runs ausführen

Review Queue/Data Quality prüfen und produktive Positionen unverändert verifizieren

Compile, pytest, Git-Safety, Git Status, optional Commit/Push falls Codeänderungen


File not found: /home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/config.py


     1|from __future__ import annotations
     2|
     3|import hashlib
     4|import shutil
     5|from dataclasses import dataclass
     6|from datetime import datetime, timezone
     7|from pathlib import Path
     8|
     9|from jarvis_finance.config.paths import is_relative_to
    10|
    11|
    12|@dataclass(frozen=True)
    13|class BackupResult:
    14|    backup_file: Path
    15|    checksum_file: Path
    16|    sha256: str
    17|    size_bytes: int
    18|
    19|
    20|@dataclass(frozen=True)
    21|class VerifyBackupResult:
    22|    backup_file: Path
    23|    checksum_file: Path
    24|    sha256: str
    25|    checksum_ok: bool
    26|    exists: bool
    27|
    28|
    29|@dataclass(frozen=True)
    30|class RestoreResult:
    31|    restored_to: Path
    32|    source_file: Path
    33|    verified: bool
    34|
    35|
    36|def sha256_file(path: Path) -> str:
    37|    digest = hashlib.sha256()
    38|    with path.open("rb") as handle:
    39|        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
    40|            digest.update(chunk)
    41|    return digest.hexdigest()
    42|
    43|
    44|def write_checksum_file(backup_file: Path, sha256: str) -> Path:
    45|    checksum_file = backup_file.with_suffix(backup_file.suffix + ".sha256")
    46|    checksum_file.write_text(f"{sha256}  {backup_file.name}\n", encoding="utf-8")
    47|    return checksum_file
    48|
    49|
    50|def read_expected_checksum(checksum_file: Path) -> str | None:
    51|    if not checksum_file.exists():
    52|        return None
    53|    text = checksum_file.read_text(encoding="utf-8", errors="ignore").strip()
    54|    if not text:
    55|        return None
    56|    return text.split()[0]
    57|
    58|
    59|def ensure_outside_repo(path: Path, repo_root: Path) -> None:
    60|    if is_relative_to(path.resolve(), repo_root.resolve()):
    61|        raise ValueError(f"Backup/restore path must be outside the Git repository: {path}")
    62|
    63|
    64|def backup_runtime_db(*, db_path: Path, backups_dir: Path, repo_root: Path, timestamp: str | None = None) -> BackupResult:
    65|    db_path = db_path.expanduser().resolve()
    66|    backups_dir = backups_dir.expanduser().resolve()
    67|    ensure_outside_repo(backups_dir, repo_root)
    68|    ensure_outside_repo(db_path, repo_root)
    69|    if not db_path.exists():
    70|        raise FileNotFoundError(f"Runtime DB not found: {db_path}")
    71|    backups_dir.mkdir(parents=True, exist_ok=True)
    72|    stamp = timestamp or datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    73|    backup_file = backups_dir / f"finance-runtime-db-{stamp}.sqlite3"
    74|    shutil.copy2(db_path, backup_file)
    75|    checksum = sha256_file(backup_file)
    76|    checksum_file = write_checksum_file(backup_file, checksum)
    77|    return BackupResult(
    78|        backup_file=backup_file,
    79|        checksum_file=checksum_file,
    80|        sha256=checksum,
    81|        size_bytes=backup_file.stat().st_size,
    82|    )
    83|
    84|
    85|def verify_backup(*, backup_file: Path, repo_root: Path | None = None) -> VerifyBackupResult:
    86|    backup_file = backup_file.expanduser().resolve()
    87|    if repo_root is not None:
    88|        ensure_outside_repo(backup_file, repo_root)
    89|    exists = backup_file.exists() and backup_file.is_file()
    90|    if not exists:
    91|        return VerifyBackupResult(
    92|            backup_file=backup_file,
    93|            checksum_file=backup_file.with_suffix(backup_file.suffix + ".sha256"),
    94|            sha256="",
    95|            checksum_ok=False,
    96|            exists=False,
    97|        )
    98|    actual = sha256_file(backup_file)
    99|    checksum_file = backup_file.with_suffix(backup_file.suffix + ".sha256")
   100|    expected = read_expected_checksum(checksum_file)
   101|    return VerifyBackupResult(
   102|        backup_file=backup_file,
   103|        checksum_file=checksum_file,
   104|        sha256=actual,
   105|        checksum_ok=bool(expected and expected == actual),
   106|        exists=True,
   107|    )
   108|
   109|
   110|def restore_runtime_db(*, backup_file: Path, db_path: Path, repo_root: Path, yes: bool = False) -> RestoreResult:
   111|    if not yes:
   112|        raise PermissionError("Restore requires explicit confirmation via --yes")
   113|    backup_file = backup_file.expanduser().resolve()
   114|    db_path = db_path.expanduser().resolve()
   115|    ensure_outside_repo(backup_file, repo_root)
   116|    ensure_outside_repo(db_path, repo_root)
   117|    verification = verify_backup(backup_file=backup_file, repo_root=repo_root)
   118|    if not verification.exists:
   119|        raise FileNotFoundError(f"Backup file not found: {backup_file}")
   120|    if not verification.checksum_ok:
   121|        raise ValueError(f"Backup checksum verification failed: {backup_file}")
   122|    db_path.parent.mkdir(parents=True, exist_ok=True)
   123|    shutil.copy2(backup_file, db_path)
   124|    return RestoreResult(restored_to=db_path, source_file=backup_file, verified=True)
   125|


     1|from __future__ import annotations
     2|
     3|import argparse
     4|from pathlib import Path
     5|
     6|from jarvis_finance.config.settings import ensure_runtime_dirs, load_settings
     7|from jarvis_finance.market.providers import CoinGeckoClient, refresh_crypto_prices
     8|from jarvis_finance.quality.data_quality import check_crypto_data_quality
     9|from jarvis_finance.quality.git_safety import assert_safe, scan_path
    10|from jarvis_finance.runtime.backup import backup_runtime_db, restore_runtime_db, verify_backup
    11|from jarvis_finance.storage.database import connect
    12|from jarvis_finance.storage.migrations import apply_migrations, get_schema_version
    13|
    14|
    15|def main(argv: list[str] | None = None) -> int:
    16|    parser = argparse.ArgumentParser(prog="finance")
    17|    sub = parser.add_subparsers(dest="command", required=True)
    18|    sub.add_parser("init-db")
    19|    sub.add_parser("migrate")
    20|    scan = sub.add_parser("git-safety-scan")
    21|    scan.add_argument("path", nargs="?", default=".")
    22|    prices = sub.add_parser("update-crypto-prices")
    23|    prices.add_argument("--currency", default="CHF")
    24|    prices.add_argument("--max-age-seconds", type=int, default=3600)
    25|    prices.add_argument("--sleep-seconds", type=float, default=0.0)
    26|    prices.add_argument("--max-retries", type=int, default=2)
    27|    prices.add_argument("--initial-backoff", type=float, default=1.0)
    28|    prices.add_argument("--max-backoff", type=float, default=8.0)
    29|    prices.add_argument("--only-missing", action="store_true")
    30|    prices.add_argument("--only-stale", action="store_true")
    31|    prices.add_argument("--only-symbol")
    32|    prices.add_argument("--limit", type=int)
    33|    prices.add_argument("--dry-run", action="store_true")
    34|    dq = sub.add_parser("check-data-quality")
    35|    dq.add_argument("--scope", choices=["crypto"], default="crypto")
    36|    dq.add_argument("--resolve-fixed", action="store_true")
    37|    dq.add_argument("--dry-run", action="store_true")
    38|    dq.add_argument("--currency", default="CHF")
    39|    dq.add_argument("--max-age-seconds", type=int, default=86_400)
    40|    sub.add_parser("backup-runtime-db")
    41|    restore = sub.add_parser("restore-runtime-db")
    42|    restore.add_argument("--from", dest="backup_file", required=True)
    43|    restore.add_argument("--yes", action="store_true")
    44|    verify = sub.add_parser("verify-backup")
    45|    verify.add_argument("--file", required=True)
    46|    args = parser.parse_args(argv)
    47|
    48|    if args.command in {"init-db", "migrate"}:
    49|        settings = load_settings()
    50|        ensure_runtime_dirs(settings)
    51|        conn = connect(settings.db_path)
    52|        apply_migrations(conn)
    53|        print(f"SCHEMA_VERSION={get_schema_version(conn)}")
    54|        return 0
    55|    if args.command == "git-safety-scan":
    56|        findings = scan_path(Path(args.path))
    57|        if findings:
    58|            for finding in findings:
    59|                print(f"{finding.path}: {finding.reason}")
    60|            return 1
    61|        assert_safe(Path(args.path))
    62|        print("GIT_SAFETY_OK")
    63|        return 0
    64|    if args.command == "update-crypto-prices":
    65|        settings = load_settings()
    66|        ensure_runtime_dirs(settings)
    67|        conn = connect(settings.db_path)
    68|        try:
    69|            apply_migrations(conn)
    70|            result = refresh_crypto_prices(
    71|                conn,
    72|                provider=CoinGeckoClient(
    73|                    max_retries=args.max_retries,
    74|                    initial_backoff_seconds=args.initial_backoff,
    75|                    max_backoff_seconds=args.max_backoff,
    76|                ),
    77|                currency=args.currency,
    78|                max_age_seconds=args.max_age_seconds,
    79|                only_missing=args.only_missing,
    80|                only_stale=args.only_stale,
    81|                only_symbol=args.only_symbol,
    82|                limit=args.limit,
    83|                dry_run=args.dry_run,
    84|                sleep_seconds=args.sleep_seconds,
    85|            )
    86|            print(
    87|                "CRYPTO_PRICE_UPDATE "
    88|                f"currency={result.currency} total_assets={result.total_assets} "
    89|                f"updated={result.updated_count} cached={result.cached_count} "
    90|                f"skipped={result.skipped_count} stale={result.stale_count} "
    91|                f"warnings={result.warning_count} errors={result.error_count} "
    92|                f"assets_still_missing_local_price={result.missing_local_price_count} "
    93|                f"dry_run={str(result.dry_run).lower()}"
    94|            )
    95|            return 1 if result.error_count else 0
    96|        finally:
    97|            conn.close()
    98|    if args.command == "check-data-quality":
    99|        settings = load_settings()
   100|        ensure_runtime_dirs(settings)
   101|        conn = connect(settings.db_path)
   102|        try:
   103|            apply_migrations(conn)
   104|            result = check_crypto_data_quality(
   105|                conn,
   106|                currency=args.currency,
   107|                max_age_seconds=args.max_age_seconds,
   108|                resolve_fixed=args.resolve_fixed,
   109|                dry_run=args.dry_run,
   110|            )
   111|            print(
   112|                "DATA_QUALITY_CHECK "
   113|                f"scope={result.scope} active_crypto_assets={result.active_crypto_assets} "
   114|                f"fresh_price_assets={result.fresh_price_assets} "
   115|                f"active_stale_price_alerts_before={result.active_stale_price_alerts_before} "
   116|                f"active_stale_price_alerts_after={result.active_stale_price_alerts_after} "
   117|                f"resolved_alerts={result.resolved_alerts} "
   118|                f"warnings={result.warnings} errors={result.errors} "
   119|                f"dry_run={str(result.dry_run).lower()}"
   120|            )
   121|            return 1 if result.errors else 0
   122|        finally:
   123|            conn.close()
   124|    if args.command == "backup-runtime-db":
   125|        settings = load_settings()
   126|        ensure_runtime_dirs(settings)
   127|        result = backup_runtime_db(
   128|            db_path=settings.db_path,
   129|            backups_dir=settings.runtime_paths.backups_dir,
   130|            repo_root=Path.cwd(),
   131|        )
   132|        print(
   133|            "RUNTIME_DB_BACKUP "
   134|            f"backup_file={result.backup_file} checksum_file={result.checksum_file} "
   135|            f"sha256={result.sha256} size_bytes={result.size_bytes}"
   136|        )
   137|        return 0
   138|    if args.command == "verify-backup":
   139|        result = verify_backup(backup_file=Path(args.file), repo_root=Path.cwd())
   140|        print(
   141|            "BACKUP_VERIFY "
   142|            f"backup_file={result.backup_file} checksum_file={result.checksum_file} "
   143|            f"exists={str(result.exists).lower()} checksum_ok={str(result.checksum_ok).lower()} "
   144|            f"sha256={result.sha256}"
   145|        )
   146|        return 0 if result.exists and result.checksum_ok else 1
   147|    if args.command == "restore-runtime-db":
   148|        settings = load_settings()
   149|        ensure_runtime_dirs(settings)
   150|        result = restore_runtime_db(
   151|            backup_file=Path(args.backup_file),
   152|            db_path=settings.db_path,
   153|            repo_root=Path.cwd(),
   154|            yes=args.yes,
   155|        )
   156|        print(
   157|            "RUNTIME_DB_RESTORE "
   158|            f"source_file={result.source_file} restored_to={result.restored_to} "
   159|            f"verified={str(result.verified).lower()}"
   160|        )
   161|        return 0
   162|    return 2
   163|
   164|
   165|if __name__ == "__main__":
   166|    raise SystemExit(main())
   167|


     1|from __future__ import annotations
     2|
     3|from dataclasses import dataclass
     4|from pathlib import Path
     5|
     6|DEFAULT_RUNTIME_DIR = Path("~/jarvis_runtime/finance-system").expanduser()
     7|
     8|
     9|@dataclass(frozen=True)
    10|class RuntimePaths:
    11|    base_dir: Path
    12|    data_dir: Path
    13|    imports_dir: Path
    14|    exports_dir: Path
    15|    reports_dir: Path
    16|    backups_dir: Path
    17|    secrets_dir: Path
    18|    logs_dir: Path
    19|    db_path: Path
    20|
    21|    @classmethod
    22|    def from_base(cls, base_dir: str | Path = DEFAULT_RUNTIME_DIR) -> "RuntimePaths":
    23|        base = Path(base_dir).expanduser().resolve()
    24|        return cls(
    25|            base_dir=base,
    26|            data_dir=base / "data",
    27|            imports_dir=base / "imports",
    28|            exports_dir=base / "exports",
    29|            reports_dir=base / "reports",
    30|            backups_dir=base / "backups",
    31|            secrets_dir=base / "secrets",
    32|            logs_dir=base / "logs",
    33|            db_path=base / "data" / "finance.sqlite3",
    34|        )
    35|
    36|    def ensure_dirs(self) -> None:
    37|        for path in [
    38|            self.data_dir,
    39|            self.imports_dir,
    40|            self.exports_dir,
    41|            self.reports_dir,
    42|            self.backups_dir,
    43|            self.secrets_dir,
    44|            self.logs_dir,
    45|        ]:
    46|            path.mkdir(parents=True, exist_ok=True)
    47|
    48|
    49|def is_relative_to(path: Path, parent: Path) -> bool:
    50|    try:
    51|        path.resolve().relative_to(parent.resolve())
    52|        return True
    53|    except ValueError:
    54|        return False
    55|


     1|from __future__ import annotations
     2|
     3|import os
     4|from dataclasses import dataclass
     5|from pathlib import Path
     6|
     7|from .paths import DEFAULT_RUNTIME_DIR, RuntimePaths, is_relative_to
     8|
     9|
    10|@dataclass(frozen=True)
    11|class Settings:
    12|    env: str = "local"
    13|    base_currency: str = "CHF"
    14|    timezone: str = "Europe/Zurich"
    15|    runtime_paths: RuntimePaths = RuntimePaths.from_base(DEFAULT_RUNTIME_DIR)
    16|    log_level: str = "INFO"
    17|    block_runtime_inside_repo: bool = True
    18|
    19|    @property
    20|    def db_path(self) -> Path:
    21|        return self.runtime_paths.db_path
    22|
    23|
    24|def find_repo_root(start: Path | None = None) -> Path:
    25|    current = (start or Path.cwd()).resolve()
    26|    for candidate in [current, *current.parents]:
    27|        if (candidate / ".git").exists() or (candidate / "pyproject.toml").exists():
    28|            return candidate
    29|    return current
    30|
    31|
    32|def load_settings(repo_root: Path | None = None, environ: dict[str, str] | None = None) -> Settings:
    33|    env = environ or os.environ
    34|    runtime_dir = Path(env.get("JARVIS_FINANCE_RUNTIME_DIR", str(DEFAULT_RUNTIME_DIR))).expanduser()
    35|    paths = RuntimePaths.from_base(runtime_dir)
    36|    db_override = env.get("JARVIS_FINANCE_DB_PATH")
    37|    if db_override:
    38|        object.__setattr__(paths, "db_path", Path(db_override).expanduser().resolve())
    39|    settings = Settings(
    40|        env=env.get("JARVIS_FINANCE_ENV", "local"),
    41|        base_currency=env.get("JARVIS_FINANCE_BASE_CURRENCY", "CHF"),
    42|        timezone=env.get("JARVIS_FINANCE_TIMEZONE", "Europe/Zurich"),
    43|        runtime_paths=paths,
    44|        log_level=env.get("JARVIS_FINANCE_LOG_LEVEL", "INFO"),
    45|        block_runtime_inside_repo=env.get("JARVIS_FINANCE_BLOCK_RUNTIME_INSIDE_REPO", "1") not in {"0", "false", "False"},
    46|    )
    47|    validate_settings(settings, repo_root=repo_root)
    48|    return settings
    49|
    50|
    51|def validate_settings(settings: Settings, repo_root: Path | None = None) -> None:
    52|    root = (repo_root or find_repo_root()).resolve()
    53|    if settings.block_runtime_inside_repo and is_relative_to(settings.runtime_paths.base_dir, root):
    54|        raise ValueError(f"Runtime directory must be outside the Git repository: {settings.runtime_paths.base_dir}")
    55|    if settings.base_currency != "CHF":
    56|        raise ValueError("MVP 1 base currency must be CHF")
    57|
    58|
    59|def ensure_runtime_dirs(settings: Settings) -> None:
    60|    settings.runtime_paths.ensure_dirs()
    61|


Explanation:
**Searching for files**

I need to find real files, perhaps on Google Drive using a specific tool called ‘gog.’ I'll want to check if I can search within my memory or local runtime imports. Maybe files are in the drive mounted? I might consider using the terminal to find them. My search may pattern the files under runtime, but I need to avoid listing any sensitive names in the final output, even if the tool output may include them. Let's search!


Explanation:
**Checking preflight script**

I need to run a preflight check, and I’m thinking about using a terminal script to aggregate things. Oh, should I not use sed in this case? It seems like a terminal is necessary for git operations. I also need to verify the remote sync, possibly using a token, but I’m not sure if I should output the token. It feels important to double-check this process to ensure everything goes smoothly!

terminal

set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
python - <<'PY'
from pathlib import Path
import os, subprocess, re, json
repo=Path.cwd().resolve()
runtime=Path('/home/agent/jarvis_runtime/finance-system').resolve()
paths={
 'runtime_db': runtime/'data'/'finance.sqlite3',
 'backups': runtime/'backups',
 'reports': runtime/'reports',
 'imports': runtime/'imports',
}
def outside(p):
    try: p.resolve().relative_to(repo); return False
    except ValueError: return True
status=subprocess.check_output(['git','status','--short','--branch'], text=True).strip()
head=subprocess.check_output(['git','rev-parse','--short','HEAD'], text=True).strip()
origin=subprocess.check_output(['git','rev-parse','--short','origin/main'], text=True).strip()
base=subprocess.check_output(['git','merge-base','HEAD','origin/main'], text=True).strip()
local_full=subprocess.check_output(['git','rev-parse','HEAD'], text=True).strip()
origin_full=subprocess.check_output(['git','rev-parse','origin/main'], text=True).strip()
forbidden=[]
for suffix in ('.xlsx','.docx','.pdf'):
    forbidden.extend([str(p.relative_to(repo)) for p in repo.rglob('*'+suffix) if p.is_file()])
print(json.dumps({
 'git_clean': status=='## main...origin/main',
 'git_status': status,
 'head': head,
 'origin': origin,
 'remote_sync': local_full==origin_full,
 'runtime_db_exists': paths['runtime_db'].exists(),
 'runtime_paths_outside_repo': {k: outside(v) for k,v in paths.items()},
 'real_files_in_repo_count': len(forbidden),
 'safe_mode_default': 'Read-only',
}, sort_keys=True))
PY
PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .


Output:
{"git_clean": true, "git_status": "## main...origin/main", "head": "fa254a9", "origin": "fa254a9", "real_files_in_repo_count": 0, "remote_sync": true, "runtime_db_exists": true, "runtime_paths_outside_repo": {"backups": true, "imports": true, "reports": true, "runtime_db": true}, "safe_mode_default": "Read-only"}
GIT_SAFETY_OK


set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
backup_out=$(PYTHONPATH=src python -m jarvis_finance.cli.main backup-runtime-db)
printf '%s\n' "$backup_out" | python - <<'PY'
import sys,re,json
text=sys.stdin.read()
m=re.search(r'backup_file=(\S+)', text)
print(json.dumps({'backup_created': bool(m), 'backup_dir':'Runtime-Backup-Ordner', 'raw_has_checksum':'sha256=' in text}, sort_keys=True))
if m:
    open('/tmp/finance_b3_backup_path.txt','w').write(m.group(1))
PY
backup_file=$(cat /tmp/finance_b3_backup_path.txt)
verify_out=$(PYTHONPATH=src python -m jarvis_finance.cli.main verify-backup --file "$backup_file")
printf '%s\n' "$verify_out" | python - <<'PY'
import sys,re,json
text=sys.stdin.read()
print(json.dumps({'backup_verified':'exists=true' in text, 'checksum_ok':'checksum_ok=true' in text, 'storage':'Runtime-Backup-Ordner'}, sort_keys=True))
PY


Output:
{"backup_created": false, "backup_dir": "Runtime-Backup-Ordner", "raw_has_checksum": false}
cat: /tmp/finance_b3_backup_path.txt: Datei oder Verzeichnis nicht gefunden


Explanation:
**Evaluating input method**

It looks like there was an issue with how we piped things; the here-doc syntax is consuming stdin, which isn't what I intended. I definitely need to rerun the process, but it seems like I don’t really have a backup. Oops, I shouldn't have assumed that step would work seamlessly! So, it’s clear that using a temporary file might be the best way to handle this situation and ensure everything goes smoothly.

set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
PYTHONPATH=src python -m jarvis_finance.cli.main backup-runtime-db > /tmp/finance_b3_backup_out.txt
python - <<'PY'
from pathlib import Path
import re,json
text=Path('/tmp/finance_b3_backup_out.txt').read_text()
m=re.search(r'backup_file=(\S+)', text)
print(json.dumps({'backup_created': bool(m), 'backup_dir':'Runtime-Backup-Ordner', 'raw_has_checksum':'sha256=' in text}, sort_keys=True))
if not m: raise SystemExit(1)
Path('/tmp/finance_b3_backup_path.txt').write_text(m.group(1))
PY
backup_file=$(cat /tmp/finance_b3_backup_path.txt)
PYTHONPATH=src python -m jarvis_finance.cli.main verify-backup --file "$backup_file" > /tmp/finance_b3_verify_out.txt
python - <<'PY'
from pathlib import Path
import json
text=Path('/tmp/finance_b3_verify_out.txt').read_text()
print(json.dumps({'backup_verified':'exists=true' in text, 'checksum_ok':'checksum_ok=true' in text, 'storage':'Runtime-Backup-Ordner'}, sort_keys=True))
PY


Output:
{"backup_created": true, "backup_dir": "Runtime-Backup-Ordner", "raw_has_checksum": true}
{"backup_verified": true, "checksum_ok": true, "storage": "Runtime-Backup-Ordner"}


set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
PYTHONPATH=src python - <<'PY'
from pathlib import Path
from tempfile import TemporaryDirectory
import shutil, json, sqlite3
from jarvis_finance.config.settings import load_settings, ensure_runtime_dirs
from jarvis_finance.storage.database import connect
from jarvis_finance.storage.migrations import apply_migrations
from jarvis_finance.imports.broker_parsers import parse_postfinance_docx, parse_true_wealth_docx, parse_raiffeisen_xlsx
from jarvis_finance.imports.broker_mapping import run_broker_import_dry_run

settings=load_settings(); ensure_runtime_dirs(settings)
conn=connect(settings.db_path); apply_migrations(conn)

def count(table):
    try:
        return conn.execute(f'SELECT COUNT(*) AS n FROM {table}').fetchone()['n']
    except sqlite3.Error:
        return None
before={t:count(t) for t in ['positions_snapshot','transactions','cash_ledger','broker_import_dry_runs','broker_import_review_items']}
base=Path('/home/agent/jarvis_runtime/finance-system/imports/broker_mapping_analysis_20260515T082614Z')
sources=[
 ('PostFinance', base/'260514_Portfolio_Postfinance.docx', parse_postfinance_docx),
 ('True Wealth', base/'260514_Portfolio_True_Wealth.docx', parse_true_wealth_docx),
 ('Raiffeisen', base/'260514_uebersicht_alle_Anlagen.xlsx', parse_raiffeisen_xlsx),
]
out=[]
with TemporaryDirectory(prefix='finance_b3_real_dryrun_') as tmp:
    tmp=Path(tmp)
    for source,path,parser in sources:
        if not path.exists():
            out.append({'source':source,'file_present':False,'errors_count':1})
            continue
        temp_path=tmp / f'{source.lower().replace(" ","_")}{path.suffix.lower()}'
        shutil.copy2(path,temp_path)
        parsed=parser(temp_path)
        summary=run_broker_import_dry_run(conn, parsed, source_filename=f'{source.lower().replace(" ","_")}_runtime_file{path.suffix.lower()}')
        conn.commit()
        out.append({
            'source': source,
            'file_present': True,
            'rows_total': summary['rows_total'],
            'candidate_positions': summary['candidate_positions'],
            'candidate_cash_rows': summary['candidate_cash_rows'],
            'mapped_positions': summary['mapped_positions'],
            'blocked_positions': summary['blocked_positions'],
            'review_items_created': summary['review_items_created'],
            'warnings_count': summary['warnings_count'],
            'errors_count': summary['errors_count'],
            'snapshot_date_status': parsed.snapshot_date_status,
            'quality_flags_summary': summary['quality_flags_summary'],
        })
after={t:count(t) for t in before}
review_by_source={}
for row in conn.execute("SELECT source_platform, COUNT(*) AS n FROM broker_import_review_items WHERE dry_run_id IN (SELECT dry_run_id FROM broker_import_dry_runs ORDER BY created_at DESC LIMIT 3) GROUP BY source_platform").fetchall():
    review_by_source[row['source_platform']]=row['n']
flag_by_source={}
for row in conn.execute("SELECT source_platform, quality_flags_json FROM broker_import_review_items WHERE dry_run_id IN (SELECT dry_run_id FROM broker_import_dry_runs ORDER BY created_at DESC LIMIT 3)").fetchall():
    import json as _json
    d=flag_by_source.setdefault(row['source_platform'], {})
    for flag in _json.loads(row['quality_flags_json'] or '[]'):
        d[flag]=d.get(flag,0)+1
print(json.dumps({'dry_runs':out,'before_counts':before,'after_counts':after,'review_items_last_session':review_by_source,'review_flags_last_session':flag_by_source,'productive_positions_unchanged':before.get('positions_snapshot')==after.get('positions_snapshot'),'transactions_unchanged':before.get('transactions')==after.get('transactions'),'cash_ledger_unchanged':before.get('cash_ledger')==after.get('cash_ledger')}, ensure_ascii=False, sort_keys=True))
PY > /tmp/finance_b3_real_dryrun_summary.json
python -m json.tool /tmp/finance_b3_real_dryrun_summary.json


Output:
/usr/bin/bash: Zeile 133: Warnung: Das in der Zeile 70 beginnende Here-Dokument geht bis zum Dateiende (erwartet wird »PY«).
  File "<stdin>", line 62
    PY > /tmp/finance_b3_real_dryrun_summary.json
         ^
SyntaxError: invalid syntax


set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
PYTHONPATH=src python - > /tmp/finance_b3_real_dryrun_summary.json <<'PY'
from pathlib import Path
from tempfile import TemporaryDirectory
import shutil, json, sqlite3
from jarvis_finance.config.settings import load_settings, ensure_runtime_dirs
from jarvis_finance.storage.database import connect
from jarvis_finance.storage.migrations import apply_migrations
from jarvis_finance.imports.broker_parsers import parse_postfinance_docx, parse_true_wealth_docx, parse_raiffeisen_xlsx
from jarvis_finance.imports.broker_mapping import run_broker_import_dry_run

settings=load_settings(); ensure_runtime_dirs(settings)
conn=connect(settings.db_path); apply_migrations(conn)

def count(table):
    try:
        return conn.execute(f'SELECT COUNT(*) AS n FROM {table}').fetchone()['n']
    except sqlite3.Error:
        return None
before={t:count(t) for t in ['positions_snapshot','transactions','cash_ledger','broker_import_dry_runs','broker_import_review_items']}
base=Path('/home/agent/jarvis_runtime/finance-system/imports/broker_mapping_analysis_20260515T082614Z')
sources=[
 ('PostFinance', base/'260514_Portfolio_Postfinance.docx', parse_postfinance_docx),
 ('True Wealth', base/'260514_Portfolio_True_Wealth.docx', parse_true_wealth_docx),
 ('Raiffeisen', base/'260514_uebersicht_alle_Anlagen.xlsx', parse_raiffeisen_xlsx),
]
out=[]
created_dry_run_ids=[]
with TemporaryDirectory(prefix='finance_b3_real_dryrun_') as tmp:
    tmp=Path(tmp)
    for source,path,parser in sources:
        if not path.exists():
            out.append({'source':source,'file_present':False,'errors_count':1})
            continue
        temp_path=tmp / f'{source.lower().replace(" ","_")}{path.suffix.lower()}'
        shutil.copy2(path,temp_path)
        parsed=parser(temp_path)
        summary=run_broker_import_dry_run(conn, parsed, source_filename=f'{source.lower().replace(" ","_")}_runtime_file{path.suffix.lower()}')
        conn.commit()
        created_dry_run_ids.append(summary['dry_run_id'])
        out.append({
            'source': source,
            'file_present': True,
            'rows_total': summary['rows_total'],
            'candidate_positions': summary['candidate_positions'],
            'candidate_cash_rows': summary['candidate_cash_rows'],
            'mapped_positions': summary['mapped_positions'],
            'blocked_positions': summary['blocked_positions'],
            'review_items_created': summary['review_items_created'],
            'warnings_count': summary['warnings_count'],
            'errors_count': summary['errors_count'],
            'snapshot_date_status': parsed.snapshot_date_status,
            'quality_flags_summary': summary['quality_flags_summary'],
        })
after={t:count(t) for t in before}
review_by_source={}
flag_by_source={}
if created_dry_run_ids:
    placeholders=','.join(['?']*len(created_dry_run_ids))
    for row in conn.execute(f"SELECT source_platform, COUNT(*) AS n FROM broker_import_review_items WHERE dry_run_id IN ({placeholders}) GROUP BY source_platform", created_dry_run_ids).fetchall():
        review_by_source[row['source_platform']]=row['n']
    for row in conn.execute(f"SELECT source_platform, quality_flags_json FROM broker_import_review_items WHERE dry_run_id IN ({placeholders})", created_dry_run_ids).fetchall():
        d=flag_by_source.setdefault(row['source_platform'], {})
        for flag in json.loads(row['quality_flags_json'] or '[]'):
            d[flag]=d.get(flag,0)+1
print(json.dumps({'dry_runs':out,'before_counts':before,'after_counts':after,'review_items_last_session':review_by_source,'review_flags_last_session':flag_by_source,'productive_positions_unchanged':before.get('positions_snapshot')==after.get('positions_snapshot'),'transactions_unchanged':before.get('transactions')==after.get('transactions'),'cash_ledger_unchanged':before.get('cash_ledger')==after.get('cash_ledger')}, ensure_ascii=False, sort_keys=True))
PY
python -m json.tool /tmp/finance_b3_real_dryrun_summary.json


Output:
{
    "after_counts": {
        "broker_import_dry_runs": 3,
        "broker_import_review_items": 59,
        "cash_ledger": null,
        "positions_snapshot": 0,
        "transactions": 0
    },
    "before_counts": {
        "broker_import_dry_runs": 0,
        "broker_import_review_items": 0,
        "cash_ledger": null,
        "positions_snapshot": 0,
        "transactions": 0
    },
    "cash_ledger_unchanged": true,
    "dry_runs": [
        {
            "blocked_positions": 33,
            "candidate_cash_rows": 1,
            "candidate_positions": 33,
            "errors_count": 0,
            "file_present": true,
            "mapped_positions": 0,
            "quality_flags_summary": {
                "cost_basis_uncertain": 33,
                "missing_isin": 33,
                "missing_ticker": 33,
                "needs_manual_review": 34,
                "snapshot_only": 34
            },
            "review_items_created": 34,
            "rows_total": 40,
            "snapshot_date_status": "missing",
            "source": "PostFinance",
            "warnings_count": 167
        },
        {
            "blocked_positions": 17,
            "candidate_cash_rows": 1,
            "candidate_positions": 17,
            "errors_count": 0,
            "file_present": true,
            "mapped_positions": 0,
            "quality_flags_summary": {
                "cost_basis_uncertain": 17,
                "missing_isin": 17,
                "missing_ticker": 17,
                "needs_manual_review": 18,
                "snapshot_only": 18
            },
            "review_items_created": 18,
            "rows_total": 19,
            "snapshot_date_status": "missing",
            "source": "True Wealth",
            "warnings_count": 87
        },
        {
            "blocked_positions": 4,
            "candidate_cash_rows": 3,
            "candidate_positions": 4,
            "errors_count": 0,
            "file_present": true,
            "mapped_positions": 0,
            "quality_flags_summary": {
                "aggregate_only": 4,
                "cash_snapshot_candidate": 3,
                "missing_instrument_details": 4,
                "needs_manual_review": 7
            },
            "review_items_created": 7,
            "rows_total": 8,
            "snapshot_date_status": "missing",
            "source": "Raiffeisen",
            "warnings_count": 18
        }
    ],
    "productive_positions_unchanged": true,
    "review_flags_last_session": {
        "PostFinance": {
            "cost_basis_uncertain": 33,
            "missing_isin": 33,
            "missing_ticker": 33,
            "needs_manual_review": 34,
            "snapshot_only": 34
        },
        "Raiffeisen": {
            "aggregate_only": 4,
            "cash_snapshot_candidate": 3,
            "missing_instrument_details": 4,
            "needs_manual_review": 7
        },
        "True Wealth": {
            "cost_basis_uncertain": 17,
            "missing_isin": 17,
            "missing_ticker": 17,
            "needs_manual_review": 18,
            "snapshot_only": 18
        }
    },
    "review_items_last_session": {
        "PostFinance": 34,
        "Raiffeisen": 7,
        "True Wealth": 18
    },
    "transactions_unchanged": true
}


Explanation:
**Checking for missing function values**

I need to take note that there’s a missing function value, which probably means the parser didn't detect the values accurately. I should verify the manual queue or dead-letter queue (DQ). Maybe I can use script counts if there are no labels present. It seems like I might also want to check out the dashboard functions to get more clarity on what's going wrong. It's a bit of a puzzle!


set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
PYTHONPATH=src python - <<'PY'
import json, sqlite3
from jarvis_finance.config.settings import load_settings
from jarvis_finance.storage.database import connect
from jarvis_finance.storage.migrations import apply_migrations
from jarvis_finance.dashboard import data
settings=load_settings(); conn=connect(settings.db_path); apply_migrations(conn)
manual_items=data.get_broker_review_items(conn, limit=1000)
dq=data.get_data_quality(conn)
flag_counts=data.get_review_quality_flag_counts(conn)
dry_runs=data.get_broker_dry_runs(conn)
# no row labels printed
status_counts={}
source_counts={}
for row in conn.execute("SELECT source_platform, review_status, COUNT(*) AS n FROM broker_import_review_items GROUP BY source_platform, review_status").fetchall():
    source_counts.setdefault(row['source_platform'], {})[row['review_status']]=row['n']
print(json.dumps({
 'manual_review_queue_items_visible': len(manual_items)>0,
 'manual_review_queue_item_count': len(manual_items),
 'source_status_counts': source_counts,
 'data_quality_checks_count': len(dq),
 'data_quality_review_flag_counts': flag_counts,
 'dry_run_summaries_visible': len(dry_runs)>=3,
 'initial_snapshot_commit_action_active': False,
 'productive_position_rows': conn.execute('SELECT COUNT(*) AS n FROM positions_snapshot').fetchone()['n'],
 'transaction_rows': conn.execute('SELECT COUNT(*) AS n FROM transactions').fetchone()['n'],
}, ensure_ascii=False, sort_keys=True))
PY


Output: {"data_quality_checks_count": 16, "data_quality_review_flag_counts": [{"open_items": "3", "quality_flag": "cash_snapshot_candidate"}, {"open_items": "50", "quality_flag": "cost_basis_uncertain"}, {"open_items": "50", "quality_flag": "missing_isin"}, {"open_items": "50", "quality_flag": "missing_ticker"}, {"open_items": "55", "quality_flag": "needs_manual_review"}, {"open_items": "52", "quality_flag": "snapshot_only"}], "dry_run_summaries_visible": true, "initial_snapshot_commit_action_active": false, "manual_review_queue_item_count": 59, "manual_review_queue_items_visible": true, "productive_position_rows": 0, "source_status_counts": {"PostFinance": {"open": 34}, "Raiffeisen": {"blocked": 4, "open": 3}, "True Wealth": {"open": 18}}, "transaction_rows": 0}


Explanation:
**Considering code adjustments**

I’m noticing a bug where the aggregate_only blocked items aren’t being counted because the aggregate_review_quality_flags are only considering open items. The user is asking for DQ aggregate_only to be visible, but it won’t be visible because of the block. I believe I need a patch function that includes both open and blocked items. So, a code change is required, and I’ll need to run tests after implementing that patch!

patch

for row in conn.execute("SELECT quality_flags_json FROM broker_import_review_items WHERE review_status='open'").fetchall():

for row in conn.execute("SELECT quality_flags_json FROM broker_import_review_items WHERE review_status IN ('open','blocked')").fetchall():


--- a//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/imports/broker_mapping.py
+++ b//home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/imports/broker_mapping.py
@@ -342,7 +342,7 @@

 def aggregate_review_quality_flags(conn: Connection) -> dict[str, int]:
     flags: Counter[str] = Counter()
-    for row in conn.execute("SELECT quality_flags_json FROM broker_import_review_items WHERE review_status='open'").fetchall():
+    for row in conn.execute("SELECT quality_flags_json FROM broker_import_review_items WHERE review_status IN ('open','blocked')").fetchall():
         try:
             for flag in json.loads(row["quality_flags_json"] or "[]"):
                 flags[str(flag)] += 1

/home/agent/.hermes/repos/FinanceManager/src/jarvis_finance/imports/broker_mapping.py was modified by sibling subagent '20260515_102855_64f760' but this agent never read it. Read the file before writing to avoid overwriting the sibling's changes.


    assert dq["Review Flag: missing_isin"] == "1"
    assert dq["Review Flag: missing_fx"] == "1"

def test_data_quality_includes_blocked_aggregate_only_review_flags(tmp_path: Path) -> None:
    path = tmp_path / "synthetic_raiffeisen.xlsx"
    _xlsx(path, [["Label", "Typ", "Währung"], ["Depot Anlagen gesamt", "Anlage", "CHF"]])
    c = conn()
    run_broker_import_dry_run(c, parse_raiffeisen_xlsx(path), source_filename=path.name)
    dq = {r["check"]: r["count"] for r in dashboard_data.get_data_quality(c)}
    assert dq["Review Flag: aggregate_only"] == "1"
    assert dq["Review Flag: missing_instrument_details"] == "1"

def test_safe_mode_default_read_only_and_commit_disabled() -> None:

    assert dq["Review Flag: missing_isin"] == "1"
    assert dq["Review Flag: missing_fx"] == "1"

def test_safe_mode_default_read_only_and_commit_disabled() -> None:


--- a//home/agent/.hermes/repos/FinanceManager/tests/unit/test_broker_parser_dry_run_pipeline.py
+++ b//home/agent/.hermes/repos/FinanceManager/tests/unit/test_broker_parser_dry_run_pipeline.py
@@ -130,6 +130,16 @@
     assert dq["Review Flag: missing_fx"] == "1"

+def test_data_quality_includes_blocked_aggregate_only_review_flags(tmp_path: Path) -> None:
+    path = tmp_path / "synthetic_raiffeisen.xlsx"
+    _xlsx(path, [["Label", "Typ", "Währung"], ["Depot Anlagen gesamt", "Anlage", "CHF"]])
+    c = conn()
+    run_broker_import_dry_run(c, parse_raiffeisen_xlsx(path), source_filename=path.name)
+    dq = {r["check"]: r["count"] for r in dashboard_data.get_data_quality(c)}
+    assert dq["Review Flag: aggregate_only"] == "1"
+    assert dq["Review Flag: missing_instrument_details"] == "1"
+
+
 def test_safe_mode_default_read_only_and_commit_disabled() -> None:
     c = conn()
     status = dashboard_data.get_safe_mode_status(c)

/home/agent/.hermes/repos/FinanceManager/tests/unit/test_broker_parser_dry_run_pipeline.py was modified by sibling subagent '20260515_104455_1e842f' but this agent never read it. Read the file before writing to avoid overwriting the sibling's changes.


terminal

set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
PYTHONPATH=src pytest tests/unit/test_broker_parser_dry_run_pipeline.py -q
PYTHONPATH=src python - <<'PY'
import json
from jarvis_finance.config.settings import load_settings
from jarvis_finance.storage.database import connect
from jarvis_finance.storage.migrations import apply_migrations
from jarvis_finance.dashboard import data
conn=connect(load_settings().db_path); apply_migrations(conn)
print(json.dumps({'data_quality_review_flag_counts': data.get_review_quality_flag_counts(conn)}, sort_keys=True))
PY


Output:
...........                                                              [100%]
11 passed in 0.11s
{"data_quality_review_flag_counts": [{"open_items": "4", "quality_flag": "aggregate_only"}, {"open_items": "3", "quality_flag": "cash_snapshot_candidate"}, {"open_items": "50", "quality_flag": "cost_basis_uncertain"}, {"open_items": "4", "quality_flag": "missing_instrument_details"}, {"open_items": "50", "quality_flag": "missing_isin"}, {"open_items": "50", "quality_flag": "missing_ticker"}, {"open_items": "59", "quality_flag": "needs_manual_review"}, {"open_items": "52", "quality_flag": "snapshot_only"}]}


set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
PYTHONPATH=src python -m compileall -q src tests
PYTHONPATH=src pytest -q
rm -rf .pytest_cache src/jarvis_finance/**/__pycache__ tests/**/__pycache__
PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .
git status --short --branch


Output:
........................................................................ [ 49%]
........................................................................ [ 99%]
.                                                                        [100%]
145 passed in 3.92s
GIT_SAFETY_OK
## main...origin/main
 M src/jarvis_finance/imports/broker_mapping.py
 M tests/unit/test_broker_parser_dry_run_pipeline.py


Explanation:
**Preparing to commit code changes**

I need to commit and push my code changes. I’m making sure everything is ready before I do that — checking for any uncommitted edits and ensuring I have a clear message for the commit. It’s always good to double-check that my changes are functioning as expected. Alright, once I confirm everything’s in order, I’ll go ahead and push the changes to the repository. Exciting!

set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
git add src/jarvis_finance/imports/broker_mapping.py tests/unit/test_broker_parser_dry_run_pipeline.py
git commit -m "fix: include blocked broker review flags in data quality"
PYTHONPATH=src python -m compileall -q src tests
PYTHONPATH=src pytest -q
rm -rf .pytest_cache src/jarvis_finance/**/__pycache__ tests/**/__pycache__
PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .
git status --short --branch


Output:
[main 4e808ed] fix: include blocked broker review flags in data quality
 2 files changed, 11 insertions(+), 1 deletion(-)
........................................................................ [ 49%]
........................................................................ [ 99%]
.                                                                        [100%]
145 passed in 4.06s
GIT_SAFETY_OK
## main...origin/main [voraus 1]


set -euo pipefail
cd /home/agent/.hermes/repos/FinanceManager
python - <<'PY'
import re, subprocess, sys
from pathlib import Path
repo=Path('/home/agent/.hermes/repos/FinanceManager')
text=Path.home().joinpath('jarvis_runtime/finance-system/secrets/Github_token_FinanceManager.csv').read_text()
m=re.search(r'(github_pat_[A-Za-z0-9_]+|ghp_[A-Za-z0-9_]+|gho_[A-Za-z0-9_]+)', text)
if not m:
    print('TOKEN_PARSE=failed'); sys.exit(12)
token=m.group(1); auth_url=f'https://{token}@github.com/Gamexgit/FinanceManager.git'
def run(args, check=True): return subprocess.run(args, cwd=repo, text=True, capture_output=True, check=check)
try:
    run(['git','fetch',auth_url,'main:refs/remotes/origin/main','--quiet'])
    local=run(['git','rev-parse','HEAD']).stdout.strip(); origin=run(['git','rev-parse','origin/main']).stdout.strip(); base=run(['git','merge-base','HEAD','origin/main']).stdout.strip()
    print('REMOTE_BEFORE='+origin[:7]); print('LOCAL_HEAD='+local[:7])
    if origin != base:
        print('REMOTE_HAS_NEW_COMMITS=yes'); sys.exit(3)
    push=run(['git','push',auth_url,'main:main'], check=False)
    if push.returncode:
        print('PUSH=failed'); print((push.stderr or push.stdout).replace(token,'[REDACTED]')[-1000:]); sys.exit(push.returncode)
    run(['git','fetch',auth_url,'main:refs/remotes/origin/main','--quiet'])
    print('PUSH=success'); print('REMOTE_AFTER='+run(['git','rev-parse','--short','origin/main']).stdout.strip())
finally:
    token=None; auth_url=None
PY
PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .
git status --short --branch


Output:
REMOTE_BEFORE=fa254a9
LOCAL_HEAD=4e808ed
PUSH=success
REMOTE_AFTER=4e808ed
GIT_SAFETY_OK
## main...origin/main


Erledigt, Sir. Phase B3 ist kontrolliert abgeschlossen.

## Gesamtstatus

- Phase B3: **erfolgreich durchgeführt**
- Produktiver Initial Snapshot: **nein**
- Produktive Aktien-/ETF-Positionen geschrieben: **nein**
- Produktive Cash-Snapshots geschrieben: **nein**
- Ledger-Transaktionen aus Brokerdateien geschrieben: **nein**
- Echte Dateien ins Git kopiert: **nein**
- Echte Werte/Mengen/Kontostände im Chat: **nein**
- Runtime Review Items erzeugt: **ja**
- Code geändert: **ja**, kleiner Fix für Data Quality Center
- Commit: `4e808ed`
- Push Status: **erfolgreich**
- Remote HEAD: `4e808ed`
- Git Status: **clean**, `main...origin/main`
- Tests: `145 passed`
- Git-Safety nach Abschluss: `GIT_SAFETY_OK`

## Preflight

Geprüft:

- Git Status clean: **ja**
- Remote HEAD synchron: **ja**
- Git-Safety vorab: **OK**
- Runtime-DB außerhalb Repo: **ja**
- Backup-/Reports-/Imports-Ordner außerhalb Repo: **ja**
- echte Broker-/Bank-Dateien im Repo: **0**
- Safe Mode / Read-only für produktive Imports: **aktiv**
- produktive Commit-Aktionen im UI: **weiterhin deaktiviert**

## Backup

Vor dem Real-File Dry Run ausgeführt:

- Runtime-DB Backup erzeugt: **ja**
- Backup verifiziert: **ja**
- Checksum OK: **ja**
- Speicherort: **Runtime-Backup-Ordner**
- Sensitive Pfade/Dateinamen: **nicht ausgegeben**

## Real-File Dry Runs

Die echten Dateien wurden nur temporär gelesen und nicht ins Repo kopiert. Temporärer Verarbeitungskontext wurde bereinigt.

### PostFinance Dry Run

- rows_total: `40`
- candidate_positions: `33`
- candidate_cash_rows: `1`
- mapped_positions: `0`
- blocked_positions: `33`
- review_items_created: `34`
- warnings_count: `167`
- errors_count: `0`
- snapshot_date_status: `missing`

Quality Flags:

- `missing_isin`: `33`
- `missing_ticker`: `33`
- `snapshot_only`: `34`
- `cost_basis_uncertain`: `33`
- `needs_manual_review`: `34`

Bewertung:

- Parser hat Positions- und Cash-Kandidaten erkannt.
- Keine produktiven Positionen geschrieben.
- Name-only / fehlende ISIN/Ticker bleiben korrekt review-pflichtig.

### True Wealth Dry Run

- rows_total: `19`
- candidate_positions: `17`
- candidate_cash_rows: `1`
- mapped_positions: `0`
- blocked_positions: `17`
- review_items_created: `18`
- warnings_count: `87`
- errors_count: `0`
- snapshot_date_status: `missing`

Quality Flags:

- `missing_isin`: `17`
- `missing_ticker`: `17`
- `snapshot_only`: `18`
- `cost_basis_uncertain`: `17`
- `needs_manual_review`: `18`

Bewertung:

- Parser hat ETF-/positionsähnliche Zeilen und Cash-/Kontozeilen erkannt.
- Aktuell keine ISIN/Ticker-Autoerkennung aus der echten Struktur.
- True Wealth wird dadurch korrekt in die Review Queue gelenkt, statt still importiert zu werden.
- `missing_fx` trat in diesem Lauf nicht auf.

### Raiffeisen Dry Run

- rows_total: `8`
- candidate_positions: `4`
- candidate_cash_rows: `3`
- mapped_positions: `0`
- blocked_positions: `4`
- review_items_created: `7`
- warnings_count: `18`
- errors_count: `0`
- snapshot_date_status: `missing`

Quality Flags:

- `cash_snapshot_candidate`: `3`
- `aggregate_only`: `4`
- `missing_instrument_details`: `4`
- `needs_manual_review`: `7`

Bewertung:

- Raiffeisen wurde defensiv verarbeitet.
- Cash-/Kontrollsnapshot-Kandidaten wurden erkannt.
- Aggregierte Anlage-/Depotzeilen wurden **nicht** als Instrumentpositionen importiert.
- Aggregate-only Items wurden blockiert/review-pflichtig gespeichert. Sehr unaufgeregt. So soll es sein.

## Review Items

Neue Runtime Review Items aus Phase B3:

- PostFinance: `34`
- True Wealth: `18`
- Raiffeisen: `7`
- Total: `59`

Statusverteilung:

- PostFinance:
  - open: `34`
- True Wealth:
  - open: `18`
- Raiffeisen:
  - open: `3`
  - blocked: `4`

Regeln erfüllt:

- jeder Review Item hat `dry_run_id`
- source platform vorhanden
- row hash vorhanden
- detected fields vorhanden
- quality flags vorhanden
- status `open` oder `blocked`
- alte Dry Runs wurden nicht überschrieben
- Details liegen nur in der Runtime-DB

## Manual Review Queue

Geprüft:

- Review Items sichtbar: **ja**
- Quelle sichtbar: **ja**
- Status sichtbar: **ja**
- Quality Flags sichtbar: **ja**
- produktive Initial-Snapshot-Commit-Aktion aktiv: **nein**
- Mapping-Aktionen bleiben kontrolliert/deaktiviert vorbereitet: **ja**

## Data Quality Center

Geprüft:

- Review Flags werden aggregiert: **ja**
- `missing_isin` sichtbar: **ja**
- `missing_ticker` sichtbar: **ja**
- `snapshot_only` sichtbar: **ja**
- `cost_basis_uncertain` sichtbar: **ja**
- `aggregate_only` sichtbar: **ja**
- `missing_instrument_details` sichtbar: **ja**
- `cash_snapshot_candidate` sichtbar: **ja**
- `missing_fx`: in diesem Lauf **nicht vorhanden**
- stale/missing price alerts weiterhin integriert: **ja**
- resolved/fixed alerts weiterhin integriert: **ja**
- produktive Positionswerte angezeigt: **nein**

Dabei wurde ein kleiner Code-Fix nötig:

- Vorher wurden im Data Quality Center nur `open` Review Items für Quality Flags aggregiert.
- Raiffeisen `aggregate_only` Items sind absichtlich `blocked`.
- Fix: Data Quality aggregiert nun `open` **und** `blocked` Review Items.
- Test ergänzt.

Commit:

- `4e808ed` — `fix: include blocked broker review flags in data quality`

## Produktive DB-Positionen unverändert

Verifiziert:

- `positions_snapshot`: unverändert
- `transactions`: unverändert
- Cash-Ledger: unverändert / nicht produktiv beschrieben
- Broker Dry Runs: neu erzeugt
- Broker Review Items: neu erzeugt

Heißt: Die Dry-Run-Pipeline hat Review-/Quality-Daten erzeugt, aber keine produktive Portfolio-Wahrheit verändert.

## Tests / Safety

Nach Abschluss ausgeführt:

- Compile: **OK**
- pytest: `145 passed`
- Git-Safety: `GIT_SAFETY_OK`
- Git Status: clean
- Push: erfolgreich

## Neue/geänderte Dateien

Geändert:

- `src/jarvis_finance/imports/broker_mapping.py`
- `tests/unit/test_broker_parser_dry_run_pipeline.py`

Keine echten Broker-/Bank-Dateien wurden committed.

## Nächste Empfehlung

Phase B4 sollte jetzt **Review Queue Hardening** sein, noch immer ohne produktiven Import:

1. Review Items pro Dry Run filterbar machen.
2. Dry Run archivieren/verwerfen vorbereiten.
3. Source-spezifische Review-Ansichten:
   - PostFinance
   - True Wealth
   - Raiffeisen
4. Manuelle Ergänzung von:
   - ISIN
   - Ticker
   - Exchange
   - Account Mapping
5. Jede Review-Aktion mit Audit-Log.
6. Noch kein Initial Snapshot Commit.
7. Danach erst kontrollierter, explizit freigegebener Import einer kleinen reviewed Safe Subset Position.

Kurz: Die echten Dateien sind jetzt trocken durch den Apparat gelaufen. Nichts Wertvolles wurde verschüttet, nichts Reales ins Git getragen, und die Review Queue hat endlich echte Arbeit vor sich.