# MVP import pipeline notes (Tasks 9–13)

Session learning from implementing the post-safety-gate FinanceManager MVP block. Use for future work on CSV importers, FX handling, and transaction ingestion.

## Scope and safety posture

- Continue with **synthetic CSVs only** until the user explicitly approves real-data import.
- Do not read/import the real crypto snapshot file during importer development; treat it only as a later reality check after dry-run, mapping, audit, and Git-safety are in place.
- Before each commit/push: remove generated caches, run compile, full pytest, Git-safety scan, and a staged secret-marker scan.
- If GitHub token is stored as a Google Sheet in Drive, download/export to a `mktemp` directory, parse token without printing it, use `GIT_ASKPASS` that reads the temp file, then `trap` cleanup/unset variables. Avoid exporting token values when possible.

## Importer pattern

1. Define `REQUIRED_COLUMNS` per importer.
2. Use a shared CSV reader/dialect detector.
3. Separate `commit=False` dry run from `commit=True` writes.
4. Collect validation errors; do not silently skip bad rows.
5. Always write an `import_sessions` record, including dry runs and failed imports.
6. Return a structured result with: `rows_total`, `rows_new`, `rows_existing`, `rows_failed`, `errors`, `status`, `import_session_id`.
7. Idempotency must classify rows as new/existing/duplicate/error before writing.

## Platforms/accounts importer

- Resolve deterministic IDs from canonical names (e.g. `stable_id("platform", platform_name)`, `stable_id("account", platform_name, account_name)`).
- Existing account key: platform + account name.
- `INSERT OR IGNORE` is acceptable only after explicit classification; do not use it as the only duplicate handling.

## Instruments importer

- Preferred existing checks: ISIN first, then ticker+name fallback.
- Required MVP fields: asset_class, name, currency. Ticker/ISIN may be absent for some assets later, so avoid over-tightening too early.

## Row hash / duplicate protection

- Row hash should be deterministic and normalize key order, whitespace, and `None` vs empty string.
- Keep value case-sensitive in the generic hash; importer-specific canonicalization belongs before storage/validation.
- Dry-run output should distinguish:
  - `new`: not in DB/source duplicates
  - `existing`: row hash or external ID already exists
  - `duplicate`: repeated in source file
  - `failed`: validation error
- General transaction imports should protect both `external_transaction_id` and `row_hash`.

## FX MVP rules

- CHF is base currency; `CHF/CHF = 1` without DB lookup.
- Foreign currency transactions need historical FX to CHF or must be stored with `fx_status=missing` and `quality_status=incomplete`.
- Missing FX creates a critical alert (`priority=kritisch`, category `fx`, rule `missing_fx`).
- Manual FX override must require a non-empty note and write an audit event.
- No multi-provider FX layer is needed at MVP stage; design DB/provider fields so it can be added later.

## Transactions importer MVP rules

- Required: transaction_type, platform/account, trade_date, quantity, original currency.
- Buy/sell/initial position snapshot require `quantity > 0`.
- `initial_position_snapshot` and `initial_cash_snapshot` must be explicitly marked in transaction_type and notes; never fake historical buys from a holdings snapshot.
- Confirmed imported transactions must write an audit log entry.
- Missing FX for foreign currency may be committed as incomplete if clearly flagged and alerted; do not pretend CHF amount is known.

## Gitignore pitfall

For finance repos, root runtime/data directories should be blocked with leading slash:

```gitignore
/data/
/imports/
/reports/
/exports/
/backups/
```

Do **not** use broad unanchored `imports/` or `reports/`; that can accidentally ignore source packages like `src/jarvis_finance/imports/` and `src/jarvis_finance/reports/`.

## Verification commands

```bash
find . -type d -name __pycache__ -prune -exec rm -rf {} +
PYTHONPATH=src python -m compileall src tests
PYTHONPATH=src pytest tests -q
find . -type d -name __pycache__ -prune -exec rm -rf {} +
PYTHONPATH=src python -m jarvis_finance.cli.main git-safety-scan .
git status --short
```

For staged secret marker check, print only OK/failed, never matched content.