schmidt.florian
e23df37a3f
ci / build-test (push) Successful in 1m12s
Reference-only secrets (SDD §6.4) meant adding a connector required editing a file on the server and restarting the service. In practice that leads to the token being pasted into the env-var *name* field, which fails with "environment variable '<token>' is not set" and gives no hint what went wrong. Add a second storage form, chosen per connector: type the secret in and it is encrypted via ASP.NET Core data protection before it is stored. The env-var reference stays as an equal alternative — this widens the choice rather than replacing it. Exactly one form survives a save, so a stale secret cannot linger and silently win; EndpointSecret.Resolve is the single resolution path. The guarantee that matters is preserved: no plaintext in the database, so pg_dump and JSON exports carry nothing usable. The trust boundary is stated plainly in §6.4 — the key ring is on disk, so this protects against leaked database content, not an attacker who already has the host, which is the same boundary an env var has. Details worth noting: - Key ring defaults to /var/lib/metervault/keys, outside the app directory, because the LXC updater republishes /opt/metervault on every update. Docker gets a named volume. Overridable via MeterVault__DataProtectionKeyPath. - Undecryptable ciphertext (key ring lost) falls back rather than throwing: an ingestion worker on a timer should degrade, not crash. - The stored secret is never sent to the browser; a blank field means "unchanged", not "cleared". - MQTT usernames are stored as-is — §6.4 covers tokens and passwords, and encrypting a username would only blank the field on every edit. - ExportService drops *_enc values: bound to the originating key ring, so useless where an export would be restored. Expect to re-enter after a restore. - HaConnectionTester now takes a resolved token, so the admin UI can test a token that has been typed but not yet saved. SDD §6.4 and §9 updated to describe both forms rather than contradict the code. Claude-Session: https://claude.ai/code/session_01V6joyergfvVLFEizH1hJLd
100 lines
12 KiB
Markdown
100 lines
12 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## What this repo is
|
||
|
||
MeterVault is a self-hosted, local-first energy & utility metering platform: it ingests meter data from Home Assistant, Tasmota and MQTT on a schedule, stores every reading timestamped and immutable, normalizes it into consumption, and turns it into cost dashboards. Energy types (electricity, water, heating oil, gas, …) and meters are **user-defined, never hardcoded**.
|
||
|
||
**Status: implemented (M0–M7) + SDD §8 panels.** The full solution is built and green — five projects, ~108 tests, working Docker deploy. `docs/SDD.md` remains the design reference; the milestone map (§12) matches the git history (M0…M7 commits). The dedicated **PV/Solar** (`/solar`), **Oil/consumable** (`/consumables`) and **meter-detail** (`/meters/{id}`) views (SDD §8.4–§8.6) are implemented as read models in `Infrastructure/Dashboard` (`SolarService`, `ConsumableService`, `MeterDetailService`) — PV meters are found by `Mode == GenerationCounter` and grid/load meters by a `role` tag in `Meter.Meta` (`MeterRoles`/`MeterMeta`), so nothing is hardcoded by name. **Admin write-CRUD** (SDD §8.7) is implemented as MudBlazor inline-dialog pages: energy types, meters (+ recompute on mode/baseline change), a meter's ingest sources (meter-detail Sources tab), tariffs, cost categories + members, and connectors (`ingestion_endpoint`, secrets by env-var reference only). `/admin/settings` is a read-only effective-config view (settings are env-driven and reproducible, not DB-stored). **Home Assistant reading** is configured here: an HA connector (`BaseUrl` + `TokenEnv`) + an HA source (entity id) drives `HomeAssistantWorker`'s REST poll, or — with the connector's **WebSocket push** toggle (`HaEndpointConfig.UseWebSocket`) — `HomeAssistantWebSocketWorker` holds a persistent `state_changed` subscription and ingests in real time (the poll worker skips WS endpoints, so each is served once; `HaWebSocketProtocol` is the pure, unit-tested handshake/parse logic). `HaConnectionTester` powers the connector "Test connection" button. **Meter topology & flow**: `MeterLink` (a directed `from→to` edge; a downstream meter is a *subsection* of an upstream one, multi-parent allowed) drives a per-energy-type page `/energy/{id}` with a hand-rolled SVG **Sankey** (`SankeyChart.razor`, since ApexCharts has no Sankey type) computed by `FlowService` (link value = downstream consumption, split proportionally across multiple parents; unaccounted remainder → an "Other" node). Upstream meters are wired cycle-safely in the meter editor; the nav lists a link per energy type. **CSV mapping wizard** (`/import/wizard`): upload an arbitrary CSV, map columns → meters/roles, dry-run preview, then commit as a revertible `import_batch` (the `/import` page lists batches with one-click revert). The `instant_rate` mode is normalized (`InstantRateNormalizer`: rate integrated over time, trapezoidal). Remaining refinement: full de-DE UI-string localization (data parsing is already de-DE) — noted at its commit. Set `MeterVault__SeedReferenceData=true` (compose: `METERVAULT_SEED=true`) for a one-command populated demo.
|
||
|
||
## Source of truth
|
||
|
||
`docs/SDD.md` is the authoritative spec and build brief — read it before implementing anything. Key protocol from §0 that governs all work here:
|
||
|
||
- **Build strictly in milestone order (§12, M0→M7).** Each milestone is independently runnable and testable; do not start Mn+1 until Mn's tests pass.
|
||
- **The four CSVs in `sampledata/` are golden fixtures.** Every parsing / consumption / cost rule must reconcile against them (§13). **If a computed number disagrees with the spreadsheet, the spreadsheet wins** unless the discrepancy is a deliberately documented correctness fix.
|
||
- When a design decision is ambiguous, check §14 (open questions): if listed, take the stated default and flag it; if not listed, ask before guessing.
|
||
- Keep the **domain layer free of infrastructure concerns** (the domain model and DB schema are UI-agnostic by design).
|
||
|
||
## Committed tech stack (do not re-litigate; see SDD §4.1)
|
||
|
||
.NET (current LTS — .NET 10, .NET 8 acceptable), C# · ASP.NET Core + **Blazor Server** · **MudBlazor** components · **ApexCharts** (Blazor-ApexCharts) · **MQTTnet** · **PostgreSQL + TimescaleDB** · **EF Core (Npgsql)** for schema/CRUD + **Dapper** for hot-path time-series reads · `BackgroundService` hosted services for ingestion/aggregation · **xUnit + Testcontainers** (Timescale image) · Docker Compose + GHCR.
|
||
|
||
## Project layout
|
||
|
||
```
|
||
/src/Core domain entities + enums; pure Normalization engine (mode strategies,
|
||
expression evaluator); Parsing (German dialect); Costing (TariffResolver)
|
||
/src/Infrastructure MeterVaultDbContext + migrations (relational + raw-SQL Timescale);
|
||
Import (CsvImporter, profiles, ImportService), Ingestion (MQTT/HA workers,
|
||
IngestionService), Normalization service, Costing/Dashboard/Backup services
|
||
/src/App ASP.NET Core host: Blazor Server UI (Components/), REST API (Api/), hosted
|
||
workers, Program.cs (Serilog, migrate+seed on startup, /healthz)
|
||
/tests/Core.Tests unit (no Docker): parsers, normalizers, swap→12, tariff resolver
|
||
/tests/Integration.Tests Testcontainers (Timescale): reconciliation vs the 4 fixtures,
|
||
import commit/revert, ingestion, cost, CAgg refresh, API, export, render
|
||
/deploy Dockerfile, docker-compose.yml (app + timescaledb), build-and-push.ps1, unraid-template.xml
|
||
```
|
||
|
||
Central package versions live in `Directory.Packages.props`; shared build/style in
|
||
`Directory.Build.props` + `.editorconfig`. Snake_case table/column mapping via
|
||
`UseSnakeCaseNamingConvention`. EF migrations are exempt from code-style enforcement (see `.editorconfig`).
|
||
|
||
## Commands
|
||
|
||
```powershell
|
||
dotnet build # build the solution
|
||
dotnet test # all tests (Integration.Tests needs Docker for Testcontainers)
|
||
dotnet test tests/Core.Tests # unit tests only (no Docker needed)
|
||
dotnet test tests/Integration.Tests --filter "FullyQualifiedName~Reconciliation" # one class/area
|
||
dotnet ef migrations add <Name> -p src/Infrastructure -s src/App -o Persistence/Migrations
|
||
dotnet run --project src/App # run app + workers locally (needs a Timescale DB)
|
||
docker compose -f deploy/docker-compose.yml up # app + TimescaleDB together
|
||
```
|
||
|
||
**Timescale-in-EF gotchas** (already handled — follow the pattern): hypertable/CAgg DDL lives in
|
||
raw-SQL migrations; continuous-aggregate creation + policies use `migrationBuilder.Sql(..., suppressTransaction: true)`, one statement each; CAgg policy `end_offset` must be ≥ one bucket. Tests
|
||
pause the compression job (historical fixture data would otherwise deadlock imports).
|
||
|
||
## Core architecture (the part that spans multiple files)
|
||
|
||
**Data pipeline — one direction, layered (SDD §4.2, §5, §7):**
|
||
|
||
```
|
||
sources (Tasmota/HA/MQTT/manual/CSV)
|
||
→ Ingestion workers write raw `reading` rows (immutable audit truth)
|
||
→ Normalization derives append-only `consumption` (deltas in base unit)
|
||
→ TimescaleDB continuous aggregates roll consumption to hourly/daily/monthly/yearly
|
||
→ Cost engine joins aggregates with time-ranged `tariff`
|
||
→ Blazor dashboard + REST API read aggregates + cost views
|
||
```
|
||
|
||
**Invariants that shape everything:**
|
||
|
||
- **Raw `reading` is immutable audit truth.** Everything derived (consumption, cost, balances, forecasts) is computed on top and must be reproducible. Never mutate readings to fix a derived number.
|
||
- **Dashboards and charts read aggregates only — never scan `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). Raw is kept for a bounded window (default 3y); `consumption` + aggregates are the long-term source of truth.
|
||
- **`meter.mode` (measurement mode) is the central abstraction** for how raw readings become consumption (SDD §5.2): `cumulative_counter`, `generation_counter`, `runtime_counter` (Δhours × rate), `consumable_balance` (tank: deliveries − usage + forecast), `direct_delta`, `instant_rate`, `virtual` (expression over other meters). New ingestion/normalization logic dispatches on mode.
|
||
- **Nothing domain-specific is hardcoded.** Energy types are data. Cost **categories are decoupled from energy types** (Heizung may be oil today, heat-pump tomorrow). PV self-consumption/savings/net are **virtual meters** with user-defined expressions, not special-cased code. Tariffs are time-ranged (price history), scoped global / per-type / per-meter.
|
||
|
||
**Timescale vs EF split (SDD §5.3):** EF Core migrations own the relational tables. **Timescale-specific DDL — `create_hypertable`, compression policies, continuous aggregates, retention — is not expressible via EF's model builder and must live in raw-SQL migrations.** `reading` and `consumption` are hypertables.
|
||
|
||
**Time & DST (SDD §10):** store UTC everywhere; bucket and display in the instance timezone (default `Europe/Berlin`). "Daily cost" boundaries are local-midnight — use `time_bucket(..., 'Europe/Berlin')`.
|
||
|
||
**Secrets (SDD §6.4):** broker/HA tokens are **never** stored in DB plaintext. Two forms, chosen per connector in the admin UI: a *reference* (`token_env`/`password_env` naming an env var or Docker secret path) resolved at runtime, or *encrypted at rest* (`token_enc`/`password_enc`) via `SecretProtector` over the ASP.NET Core data-protection key ring. Exactly one survives a save; `EndpointSecret.Resolve` is the single resolution path (encrypted wins). The key ring lives outside the app directory (`MeterVault__DataProtectionKeyPath`, default `/var/lib/metervault/keys`) because the LXC updater republishes `/opt/metervault`. `ExportService` drops `*_enc` values — they are bound to the originating key ring.
|
||
|
||
## Reference-data behaviours the code must reproduce (from `sampledata/`)
|
||
|
||
These CSVs are the German-dialect *Energiebilanz* spreadsheet export and define the minimum feature bar (SDD §2, Appendix A). When writing the importer or normalization, honour:
|
||
|
||
- **German number dialect:** decimal comma (`180,8244706`), thousands dot (`2.940,19`), trailing-`€` currency (`120,00 €`), **unit suffixes on values** (`411kWh`, `49` cm, `2287` L) — strip and validate.
|
||
- **Two date formats:** `Monat YYYY` (German month names, monthly tables) and `DD.MM.YYYY` (event rows).
|
||
- **Skip inline summary rows** (`Total`, `Heute`, `Seitbeginn Tage`, `Seit YYYY`) and **all-zero future placeholder rows** (e.g. Dec 2026) — do not ingest them. **Negatives are valid** (savings, grid balance).
|
||
- **Water register swaps mid-series** (…861 → 2 → 15): consumption must stay continuous across the boundary via a `meter_swap` event.
|
||
- **Electricity has 5 meters** (Haus, Netz, Auto, Solar 1, Solar 2) plus derived columns. Verified relations to reproduce: `Netz Einsparung = Haus − Netz`, `Ersparnis = Netz Einsparung × €/kWh`, `Kosten = Verbrauchskosten − Ersparnis` — but implement these as **user-definable virtual-meter expressions**, not hardcoded formulas.
|
||
- **Heating oil is the versatility stress test:** a consumable/tank model where consumption is derivable two ways — tank-level Δ, or burner runtime × rate (rate `fixed` from nozzle spec, or `empirical` = Δlevel ÷ Δhours). Early rows (1997–2004) carry deliveries only (no burner hours yet). Includes cm→litre dipstick calibration and forecast-to-empty.
|
||
|
||
## Git
|
||
|
||
Remote `origin` is `https://git.finalfactory.de/FinalFactory/MeterVault.git` (Gitea; default branch `master`). CI/release is **Gitea Actions** under `.gitea/workflows/` — edit `VERSION` on `master` to tag + publish the image to the Gitea container registry.
|