# MeterVault — Software Development Document

> **Working codename:** `MeterVault` (rename freely before publishing). Your existing spreadsheet system is called *Energiebilanz*; for a public GitHub repo an English name reaches more people, but `Energiebilanz` is also fine.
>
> **What this is:** a self-hosted, local-first energy & utility metering platform that pulls meter data from Home Assistant, Tasmota and MQTT on a schedule, stores every reading with a timestamp, and turns it into cost dashboards. Not limited to electricity/water/oil — energy types are user-defined.
>
> **Status:** design spec, pre-code. This document doubles as the build brief for Claude Code.

---

## 0. How to build this with Claude Code

This document is the source of truth. Suggested working protocol:

1. Drop this file in the repo root as `SDD.md`, and create a `CLAUDE.md` that references it (`See SDD.md for the full spec; work milestone by milestone; do not skip tests`).
2. Build strictly in the **milestone order** in §12. Each milestone is independently runnable and testable — do not start Mn+1 until Mn's tests pass.
3. The **four reference CSVs** (`Energiebilanz_-_*.csv`) are golden fixtures. Every parsing/consumption/cost rule in this doc must be validated against them (§13). If a computed number disagrees with the spreadsheet, the spreadsheet wins unless the discrepancy is documented as a deliberate correctness fix.
4. Prefer small, reviewable PRs per milestone. Keep the domain layer free of infrastructure concerns.
5. When a design decision is ambiguous, check §14 (open questions) — if it's listed, pick the stated default and flag it; if it isn't, ask before guessing.

---

## 1. Vision & scope

### 1.1 Problem

Utility/energy bookkeeping today lives in a hand-maintained Google Sheet (the *Energiebilanz*). It works but: readings are entered manually and monthly, there's no live pull from the sensors that already exist (Tasmota plugs, HA entities), the cost logic is buried in cell formulas, and it doesn't scale to fine-grained data or many meters.

### 1.2 Goals

- **Automatic ingestion** from HA, Tasmota and raw MQTT on a schedule, plus manual entry and CSV import.
- **Every reading timestamped** and preserved (auditable), with a normalized consumption/cost layer on top.
- **Versatile by design:** energy types (electricity, water, heating oil, gas, district heat, pool operation, …) and meters are user-defined, not hardcoded. Multiple meters of the same type are first-class (the reference data has **five** electricity meters).
- **Cost engine** with time-ranged tariffs (price history), per-type and per-category rollups.
- **Neat dashboard:** daily / monthly / yearly cost; period-over-period difference ("what cost more, what cost less"); PV savings; oil/tank forecasting; cost ranking.
- **Scale target:** up to ~1,000 meters, data retained up to 50 years.
- **Self-hosted, zero cloud dependency.** Runs on the existing homelab (Docker on Unraid, or Proxmox LXC).
- **Releasable OSS quality:** clean repo, Docker image, docs, CI, license.

### 1.3 Non-goals (v1)

- Not a smart-meter *reading* device (no P1/SML hardware decoding — that stays in HA/ESPHome/Tasmota upstream; we ingest the resulting values).
- Not a billing/invoicing system for third parties.
- Not multi-tenant SaaS. Single household/instance; optional lightweight multi-user, but not tenant isolation.
- No mobile native app (responsive web is enough).

---

## 2. What the reference data establishes

The four CSVs are the minimum feature bar. Summary of what each proves the app must support:

### 2.1 `Kosten` (cost overview) — monthly
Columns: `Datum, Jahreskosten, Kosten, Heizung, Strom, Wasser, Pool Betrieb`.
→ A **cost-category** rollup: monthly total plus a breakdown per category (Heizung/Strom/Wasser/Pool), and a yearly total (on December rows). "Heizung" is a *category* that may be fed by oil today but gas/heat-pump tomorrow — categories are decoupled from energy types. "Pool Betrieb" can be a **flat manual monthly cost** with no meter.

### 2.2 `Strom_Verbrauch` (electricity) — monthly, rich
Meters (cumulative registers): `Zähler Haus`, `Zähler Netz`, `Zähler Auto` (wallbox/EV), `Zähler Solar 1`, `Zähler Solar 2` → **5 electricity meters**.
Derived quantities: `Solar Erzeugung` (generation), `Netz Einsparung` (grid-balance), `Anlage Eigenverbrauch` (self-consumption), `Ersparnis` (savings), `Verbrauchskosten`, `Kosten` (= `Verbrauchskosten − Ersparnis`), plus yearly rollups.
Price `€/kWh` with **history** (0.16 → 0.44 → 0.31 → 0.27).
→ Must support: multiple registers per type; **generation** meters; **virtual/derived** meters computed from others via a formula; PV self-consumption & savings; register→consumption deltas; tariff-time-ranged cost. Verified relation: `Netz Einsparung = Haus − Netz`, `Ersparnis = Netz Einsparung × €/kWh`, `Kosten = Verbrauchskosten − Ersparnis`. The exact formula is user-domain — the app must let users **define** such derived metrics, not hardcode these.

### 2.3 `Wasser` (water) — monthly
`Zähler Wasser, Wasserverbrauch, €/m³, Kosten, Jahreskosten`. The register **swaps mid-series** (…861 → 2 → 15 → …).
→ Must support **meter swaps / counter resets** with consumption continuity across the boundary.

### 2.4 `Heizöl_Verbrauch` (heating oil) — the versatility stress test
Header KPIs: total delivered `64042`, tank size `7000`, consumption per month/day/year.
Table columns: `Betriebststunden` (cumulative burner hours), `Differenz Betrieb` (Δ hours), `Betrieb / Tag`, `Füllstand cm` (manual dipstick), `Tankfüllstand`/`Tank Aktuell` (litres), `Vorhersage`, `Lieferungmenge` (delivery litres), `Differenz Tank` (Δ level = monthly consumption), `Verbrauch / Tag`, `Vorraussichtliches Ende` (predicted empty), `Verbrauch / Betrieb Stunde` (L/h — **empirically derived**, e.g. 1.87, 1.94, 2.92 …), `€/100l`, `Monatskosten`.
Early rows (1997–2004) only carry **deliveries** (no burner hours — tracking started later).
→ Must support a **consumable/tank** model: deliveries add to a balance; consumption derivable **two ways** — (a) tank-level Δ, (b) burner **runtime × rate** — where the rate can be **fixed** (nozzle spec) or **empirical** (tank Δ ÷ hours Δ); physical level readings (cm) via a tank calibration curve; and a **forecast to empty**. This is the "not limited to oil" generalization: any consumable drawn from a store and/or consumed proportionally to a runtime signal.

### 2.5 Cross-cutting data facts (drive the CSV importer, §Appendix A)
- Decimal separator is **comma**; thousands separator **dot**; currency like `2.940,19 €`.
- Values carry **unit suffixes**: `411kWh`, `49` cm, `2287` L.
- Two date formats: `"September 2022"` (month tables) and `10.06.1997` / `13.07.2026` (`DD.MM.YYYY`, event rows).
- **Summary rows** exist inline (`Total`, `Heute`, `Seitbeginn Tage`, `Seit 2023`) and must be skipped, not ingested.
- **Placeholder future rows** (Dec 2026 all-zero) must be treated as no-data.
- Negative values are valid (savings, grid balance).

---

## 3. Functional requirements

| ID | Requirement |
|----|-------------|
| FR-1 | Define arbitrary **energy types** (key, display name, base unit, icon, colour). Ships with sensible defaults but nothing is hardcoded. |
| FR-2 | Define **meters** (≤ ~1000), each bound to an energy type and a **measurement mode** (§5.2). Multiple meters per type. Serial/model/install/retire metadata. |
| FR-3 | Attach one or more **data sources** to a meter: MQTT topic, Tasmota field, HA entity, manual, import, or virtual (formula). Per-source scale/offset, priority, enable flag, last-seen status. |
| FR-4 | **Ingest on schedule / on message** from MQTT (incl. Tasmota) and HA (WebSocket push or REST poll). Idempotent; guard cumulative registers against spurious decreases; sample/debounce high-frequency sources. |
| FR-5 | **Manual entry**: add a reading, delivery, tank level, swap, or correction from the UI. |
| FR-6 | **CSV import** with column mapping, saved mapping profiles, German dialect handling (Appendix A), dry-run preview, and revertible import batches. The four reference CSVs must import correctly. |
| FR-7 | **Consumption normalization**: convert raw readings (register/level/runtime/rate) into normalized consumption/generation in the base unit, handling deltas, swaps, resets, deliveries, runtime×rate. |
| FR-8 | **Virtual meters**: values computed from other meters via a user-defined expression (self-consumption, savings, net). |
| FR-9 | **Tariffs** with time ranges (price history): unit price, base/standing price, feed-in tariff, bonus, discount, tax. Scope: global, per energy type, or per meter. Support **monthly** pricing *and* **day-accurate proration** when price changes mid-period. |
| FR-10 | **Cost categories** (Heizung/Strom/Wasser/Pool …) mapping one or more meters/types → a category; plus **manual flat costs** (e.g. pool) with no meter. |
| FR-11 | **Aggregation**: hourly/daily/monthly/yearly consumption, generation and cost, per meter and per category. |
| FR-12 | **Dashboard** (§8): today/month/year cost KPIs with Δ vs previous period; cost breakdown & "what costs most"; period-over-period difference view; trends with granularity toggle and previous-year overlay; PV panel; oil/consumable panel; meter detail. |
| FR-13 | **PV analytics**: generation, self-consumption, autarky %, self-consumption %, savings. |
| FR-14 | **Consumable/tank analytics**: balance, deliveries log, burner runtime, effective L/h (fixed or empirical), forecast to empty. |
| FR-15 | **REST API + OpenAPI** for ingest (push), query, and automation; API-key auth. Lets HA *push* as an alternative to us *pulling*. |
| FR-16 | **Retention & storage** configurable to satisfy up to 50 years of data (§5.5). |
| FR-17 | **i18n**: English + German UI; locale-aware number/currency/date formatting. |
| FR-18 | **Deploy** via Docker Compose (app + TimescaleDB); Unraid template; healthcheck endpoint; backup guidance. |
| FR-19 | **Auth**: optional local accounts *and* reverse-proxy trust (honour `X-Forwarded-User` behind Authelia/Traefik). |

---

## 4. Architecture & tech stack

### 4.1 Stack (recommended, committed)

| Layer | Choice | Rationale |
|-------|--------|-----------|
| Runtime | **.NET (current LTS — .NET 10; .NET 8 acceptable)**, C# | Matches your toolchain (Rider, C#-first) and the MQTTower precedent (.NET Blazor + MQTT). |
| Web/UI | **ASP.NET Core + Blazor Server** | Same model as MQTTower; server-side keeps DB/time-series logic close, good for a homelab dashboard. |
| Component kit | **MudBlazor** | Mature, clean, good tables/cards/dialogs. |
| Charts | **ApexCharts (Blazor-ApexCharts)** | Solid for time-series, stacked bars, mixed cost/consumption. |
| MQTT | **MQTTnet** | The standard .NET MQTT lib; used for Tasmota + HA-published topics. |
| DB | **PostgreSQL + TimescaleDB** | Postgres you already know, with hypertables, native compression, and continuous aggregates — the right tool for 1000 meters × 50 years. |
| ORM / data | **EF Core (Npgsql)** for schema/migrations/CRUD; **Dapper** for hot-path time-series reads | EF for productivity; Dapper + raw SQL where Timescale features (hypertables, CAggs, `time_bucket`) need it. |
| Background work | **`BackgroundService` / hosted services** | MQTT subscriber, HA poller, aggregation refresh, forecast recompute. |
| Tests | **xUnit + Testcontainers (Timescale image)** | Real DB in integration tests; the 4 CSVs as fixtures. |
| Container | **Docker + docker-compose**; **GHCR** multi-arch (amd64 primary) via GitHub Actions | Homelab-native distribution. |

> The **domain model and DB schema are UI-agnostic.** If a future maintainer swaps Blazor for an SPA, everything from §5–§7 and the REST API in §9 is reusable.

### 4.2 Components & data flow

```
 ┌──────────── sources ────────────┐
 │  Tasmota  ──tele/<t>/SENSOR──┐   │
 │  HA (WS/REST)  ──────────────┤   │        ┌─────────────────────────────┐
 │  raw MQTT  ──────────────────┼──▶│ Ingestion │  normalize  │  TimescaleDB │
 │  Manual / CSV import ────────┘   │  workers  │  pipeline   │  (hypertables│
 └──────────────────────────────────┘          │  (deltas,   │   + CAggs)   │
                                                │  swaps,     └──────┬───────┘
                                                │  runtime×r) │       │
                                                └─────────────┘       │
                                          Cost engine (tariff join)   │
                                                     │                │
                                    ┌────────────────┴────────────────┴───┐
                                    │  Blazor dashboard   +   REST API     │
                                    └──────────────────────────────────────┘
```

Ingestion writes **raw `reading`** rows. A normalization step derives **`consumption`** (append-only, base unit). Continuous aggregates roll consumption up to hourly/daily/monthly/yearly. The cost engine joins aggregates with time-ranged tariffs. The dashboard and API read aggregates + cost views (never scan raw for charts).

---

## 5. Data model & database

Design principles: raw readings are immutable audit truth; everything derived (consumption, cost, balances, forecasts) is computed on top and reproducible; the big table is a Timescale hypertable; long-horizon retention is served by aggregates, not by keeping every raw row forever.

### 5.1 Entity overview

- `energy_type` — user-defined categories of measurement.
- `meter` — the "device" (≤1000). Has a **measurement mode**.
- `meter_source` — 0..n ingest bindings per meter.
- `reading` *(hypertable)* — raw timestamped values.
- `consumption` *(hypertable)* — normalized, append-only deltas in base unit.
- `meter_event` — discrete events: swap, reset, delivery, tank level, correction, note.
- `tank` — consumable store: capacity, cm→litre calibration, thresholds, cached balance.
- `tariff` — price components with validity ranges (history).
- `cost_category` + `cost_category_member` — reporting groups (Heizung/Strom/Wasser/Pool).
- `manual_cost` — flat costs with no meter (e.g. pool).
- `ingestion_endpoint` — broker / HA connection configs (secrets by reference).
- `import_batch` — provenance + revert for CSV/manual bulk loads.
- `app_setting` — currency, locale, timezone, retention, fallbacks.

### 5.2 Measurement modes (`meter.mode`)

| Mode | Meaning | Consumption derived by |
|------|---------|------------------------|
| `cumulative_counter` | Monotonic register (Zähler Haus/Netz/Auto, water) | Δ register between readings; handle swaps/resets. |
| `generation_counter` | Monotonic generation register (Solar 1/2) | Δ register → generation. |
| `runtime_counter` | Cumulative operating hours (burner) | Δ hours × rate (fixed or empirical) → consumption. |
| `consumable_balance` | Tank/bottle with deliveries + level | Deliveries add; usage from level-Δ and/or runtime×rate; forecast to empty. |
| `direct_delta` | Source already reports increments | Value *is* the increment. |
| `instant_rate` | Power/flow sensor (optional v1) | Integrate rate over time. |
| `virtual` | Computed from other meters | Evaluate `expression` over referenced meters. |

### 5.3 Schema sketch (PostgreSQL + TimescaleDB)

> Illustrative DDL; EF Core migrations own the relational tables, and **raw-SQL migrations** own the Timescale-specific DDL (`create_hypertable`, compression, continuous aggregates, retention). Timescale objects are *not* expressible through EF's model builder.

```sql
CREATE TABLE energy_type (
    id            SMALLSERIAL PRIMARY KEY,
    key           TEXT NOT NULL UNIQUE,          -- 'electricity','water','heating_oil','gas','heat','pool'
    display_name  TEXT NOT NULL,
    base_unit     TEXT NOT NULL,                 -- 'kWh','m3','L','h'
    default_mode  TEXT NOT NULL,                 -- see 5.2
    icon          TEXT,
    color_hex     TEXT,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE meter (
    id             INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    name           TEXT NOT NULL,
    energy_type_id SMALLINT NOT NULL REFERENCES energy_type(id),
    mode           TEXT NOT NULL,
    unit           TEXT NOT NULL,                -- defaults from energy_type.base_unit
    location       TEXT,
    serial_number  TEXT,
    model          TEXT,
    manufacturer   TEXT,
    installed_at   DATE,
    retired_at     DATE,
    is_active      BOOLEAN NOT NULL DEFAULT true,
    meta           JSONB NOT NULL DEFAULT '{}',  -- rate config, formula, tank ref, etc.
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ix_meter_type_active ON meter(energy_type_id, is_active);

CREATE TABLE meter_source (
    id           INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    meter_id     INT NOT NULL REFERENCES meter(id) ON DELETE CASCADE,
    source_type  TEXT NOT NULL,                  -- 'mqtt','tasmota','homeassistant','manual','import','virtual'
    endpoint_id  INT REFERENCES ingestion_endpoint(id),
    config       JSONB NOT NULL DEFAULT '{}',    -- topic / field path / entity_id / expression / poll interval
    value_kind   TEXT NOT NULL,                  -- 'register','delta','rate','level','runtime'
    scale        DOUBLE PRECISION NOT NULL DEFAULT 1,
    "offset"     DOUBLE PRECISION NOT NULL DEFAULT 0,
    priority     INT NOT NULL DEFAULT 0,
    is_enabled   BOOLEAN NOT NULL DEFAULT true,
    last_seen_at TIMESTAMPTZ,
    last_value   DOUBLE PRECISION,
    last_status  TEXT
);

-- BIG TABLE: raw readings
CREATE TABLE reading (
    time      TIMESTAMPTZ      NOT NULL,
    meter_id  INT              NOT NULL REFERENCES meter(id),
    value     DOUBLE PRECISION NOT NULL,        -- register value / level / hours / rate, in meter.unit
    source_id INT REFERENCES meter_source(id),
    quality   SMALLINT         NOT NULL DEFAULT 0, -- 0 measured,1 estimated,2 manual,3 imported,4 interpolated
    flags     INT              NOT NULL DEFAULT 0, -- bitmask: reset, anomaly, ...
    PRIMARY KEY (meter_id, time)
);
SELECT create_hypertable('reading', 'time',
       chunk_time_interval => INTERVAL '30 days');
ALTER TABLE reading SET (timescaledb.compress,
       timescaledb.compress_segmentby = 'meter_id',
       timescaledb.compress_orderby   = 'time DESC');
SELECT add_compression_policy('reading', INTERVAL '30 days');

-- Normalized consumption (append-only deltas in base unit)
CREATE TABLE consumption (
    time      TIMESTAMPTZ      NOT NULL,         -- end of the interval this delta covers
    meter_id  INT              NOT NULL REFERENCES meter(id),
    amount    DOUBLE PRECISION NOT NULL,         -- consumption(+) or generation(+) in base unit
    kind      SMALLINT         NOT NULL DEFAULT 0, -- 0 consumption, 1 generation
    quality   SMALLINT         NOT NULL DEFAULT 0,
    PRIMARY KEY (meter_id, time, kind)
);
SELECT create_hypertable('consumption', 'time',
       chunk_time_interval => INTERVAL '90 days');

CREATE TABLE meter_event (
    id          INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    meter_id    INT NOT NULL REFERENCES meter(id) ON DELETE CASCADE,
    time        TIMESTAMPTZ NOT NULL,
    event_type  TEXT NOT NULL,                   -- 'meter_swap','counter_reset','delivery','tank_level','correction','note'
    amount      DOUBLE PRECISION,                -- delivery litres / correction value
    prev_value  DOUBLE PRECISION,                -- swap: old register final
    new_value   DOUBLE PRECISION,                -- swap: new register initial
    unit        TEXT,
    notes       TEXT,
    meta        JSONB NOT NULL DEFAULT '{}'
);
CREATE INDEX ix_event_meter_time ON meter_event(meter_id, time);

CREATE TABLE tank (
    id              INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    meter_id        INT NOT NULL REFERENCES meter(id) ON DELETE CASCADE,
    capacity        DOUBLE PRECISION NOT NULL,   -- e.g. 7000
    unit            TEXT NOT NULL DEFAULT 'L',
    calibration     JSONB,                       -- cm→litre curve or geometry for level readings
    rate_mode       TEXT NOT NULL DEFAULT 'empirical', -- 'fixed' | 'empirical'
    fixed_rate      DOUBLE PRECISION,            -- L per runtime-hour when rate_mode='fixed'
    low_threshold   DOUBLE PRECISION,
    reorder_threshold DOUBLE PRECISION,
    cached_balance  DOUBLE PRECISION,            -- last computed Tank Aktuell
    cached_at       TIMESTAMPTZ
);

CREATE TABLE tariff (
    id          INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    scope_type  TEXT NOT NULL,                   -- 'global','energy_type','meter'
    scope_id    INT,                             -- energy_type.id or meter.id (null for global)
    component   TEXT NOT NULL,                   -- 'unit_price','base_price','feed_in','bonus','discount','tax'
    value       DOUBLE PRECISION NOT NULL,
    unit        TEXT NOT NULL,                   -- 'EUR/kWh','EUR/m3','EUR/100L','EUR/month'
    currency    TEXT NOT NULL DEFAULT 'EUR',
    valid_from  DATE NOT NULL,
    valid_to    DATE,                            -- null = open-ended
    notes       TEXT
);
CREATE INDEX ix_tariff_scope ON tariff(scope_type, scope_id, component, valid_from);

CREATE TABLE cost_category (
    id        INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    name      TEXT NOT NULL,
    color_hex TEXT,
    sort      INT NOT NULL DEFAULT 0
);
CREATE TABLE cost_category_member (
    category_id    INT NOT NULL REFERENCES cost_category(id) ON DELETE CASCADE,
    meter_id       INT REFERENCES meter(id) ON DELETE CASCADE,
    energy_type_id SMALLINT REFERENCES energy_type(id) ON DELETE CASCADE,
    CHECK (meter_id IS NOT NULL OR energy_type_id IS NOT NULL)
);

CREATE TABLE manual_cost (
    id          INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    category_id INT REFERENCES cost_category(id),
    meter_id    INT REFERENCES meter(id),
    period_start DATE NOT NULL,
    period_end   DATE NOT NULL,
    amount      DOUBLE PRECISION NOT NULL,
    currency    TEXT NOT NULL DEFAULT 'EUR',
    notes       TEXT
);

CREATE TABLE ingestion_endpoint (
    id         INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    type       TEXT NOT NULL,                    -- 'mqtt_broker','homeassistant'
    name       TEXT NOT NULL,
    config     JSONB NOT NULL DEFAULT '{}',      -- host/port/tls/base_url; secrets by *reference* only
    is_enabled BOOLEAN NOT NULL DEFAULT true,
    last_status TEXT,
    last_seen_at TIMESTAMPTZ
);

CREATE TABLE import_batch (
    id          INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    source_name TEXT,
    mapping     JSONB,
    row_count   INT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    reverted_at TIMESTAMPTZ
);

CREATE TABLE app_setting (
    key   TEXT PRIMARY KEY,
    value JSONB NOT NULL
);
```

### 5.4 Continuous aggregates & cost view

```sql
-- Daily normalized consumption per meter (local-tz buckets)
CREATE MATERIALIZED VIEW consumption_daily
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 day', time, 'Europe/Berlin') AS day,
       meter_id, kind,
       sum(amount) AS amount
FROM consumption
GROUP BY day, meter_id, kind;
-- + monthly and yearly CAggs the same way (bucket '1 month' / '1 year').
SELECT add_continuous_aggregate_policy('consumption_daily',
       start_offset => INTERVAL '3 days', end_offset => INTERVAL '1 hour',
       schedule_interval => INTERVAL '1 hour');
```

Cost is **not** baked into a continuous aggregate (tariffs are a slowly-changing dimension; a CAgg can't join them cleanly). Instead, compute cost in a regular SQL view / function on top of the consumption CAggs, resolving the active `unit_price`/`base_price`/`feed_in` for each bucket by date. Support two modes: **monthly price** (one price per calendar month, matching the spreadsheet) and **day-accurate proration** (split a bucket if a price change falls inside it).

### 5.5 Capacity & retention (the 1000×50y requirement)

Worst-case raw volume, 1000 meters:

| Ingest granularity | Rows/day | Rows/year | Rows/50y |
|--------------------|----------|-----------|----------|
| 1/min | 1.44 M | 525 M | **26.3 B** |
| 5/min (typical Tasmota TelePeriod 300) | 288 k | 105 M | **5.25 B** |
| daily (manual/legacy) | 1 k | 365 k | 18 M |

Normalized/rolled-up volumes are tiny regardless: daily consumption = 1000 × 365 × 50 = **18.25 M**, monthly **600 k**, yearly **50 k**.

**Strategy:**
- **Raw `reading`:** Timescale hypertable + columnar compression (`segmentby meter_id`), typically 10–20× on monotonic sensor data. Raw is kept for a **configurable window** (default: 3 years) — long enough for full-resolution drill-down.
- **`consumption` + continuous aggregates:** the long-term source of truth. Kept effectively **forever** (they're small), which is what actually satisfies "50 years of data" for dashboards and cost.
- If the user *insists* on 50 years of raw high-frequency data, it's still feasible on a homelab NAS (single-digit TB compressed) — expose raw retention as a setting, don't hardcode.
- Recommend `space` partitioning by `meter_id` only if meter count and query patterns justify it; start with time-only chunks.

> Design conclusion: 1000 meters × 50 years is comfortably within TimescaleDB on modest hardware **provided** dashboards read aggregates and raw retention is bounded. Don't let the UI scan `reading` for charts.

---

## 6. Ingestion

### 6.1 MQTT / Tasmota worker
- A hosted `BackgroundService` maintains a persistent MQTTnet connection per enabled `ingestion_endpoint` of type `mqtt_broker`.
- Subscribes to the union of topics from enabled `meter_source`s (and Tasmota patterns like `tele/+/SENSOR`).
- On message: resolve topic → source(s); extract value via JSON path/template from `config` (Tasmota energy under `ENERGY.Total`, `ENERGY.Today`, `ENERGY.Power`; generic sensors by path); apply `scale`/`offset`; write `reading` with a timestamp (prefer the payload's own time field, else Tasmota `Time`, else receive time).
- **Sampling/debounce:** per-source policy — store on-change and/or at most 1/min for chatty sources, to keep raw volume in check.
- **Idempotent** upsert on `(meter_id, time)`. For `cumulative_counter`/`generation_counter`, reject decreases unless an active `counter_reset`/`meter_swap` event explains it.

### 6.2 Home Assistant connector
- Prefer **WebSocket API** (`auth` with long-lived token → `subscribe_events` / `state_changed`) for push; fall back to **REST poll** `/api/states/<entity_id>` on a per-source interval.
- Extract `state` or a named `attribute`; parse units; write `reading`.
- Optional **backfill** on first connect / after downtime via `/api/history/period`.
- HA can also simply publish to MQTT — in which case it's an `mqtt`/`tasmota` source and no HA connector is needed.

### 6.3 Manual & CSV import
- UI quick-add for readings, deliveries, tank levels, swaps, corrections.
- CSV wizard: upload → detect dialect (Appendix A) → map columns to meters + row semantics → **dry-run preview** (computed consumption/cost shown) → commit as an `import_batch` (revertible).
- Ship the four reference CSVs as built-in example imports and as test fixtures.

### 6.4 Secrets
Tokens/passwords are **never** stored in plaintext in the DB. `ingestion_endpoint.config` holds a *reference* (env var name / Docker secret path); the app resolves at runtime. Document this clearly.

---

## 7. Consumption normalization & cost engine

### 7.1 Register → consumption
For `cumulative_counter`/`generation_counter`: for each new reading, `amount = value − previous_value`. Persist to `consumption`. Cross a `meter_swap` as `(old_final − prev) + (curr − new_initial)`; a `counter_reset` starts a fresh baseline. Ignore/annotate negative deltas that lack an explaining event (flag as anomaly).

### 7.2 Runtime → consumption (burner)
For `runtime_counter`: `amount = Δhours × rate`. `rate` comes from the linked `tank`: `fixed` (nozzle spec, L/h) or `empirical` (`Δlevel ÷ Δhours` measured between deliveries/level reads — reproduce the spreadsheet's 1.87/1.94/2.92 … behaviour). Expose both; default empirical when level data exists, else fixed.

### 7.3 Consumable/tank balance & forecast
`balance(t) = Σ deliveries(≤t) − Σ consumption(≤t)`, reconciled to physical `tank_level` events when present (cm → litres via calibration). Forecast to empty from a trailing consumption rate (e.g. last-30-day L/day) → `Vorraussichtliches Ende`. Surface low/reorder thresholds.

### 7.4 Virtual meters
For `virtual`: evaluate `config.expression` (whitelisted, sandboxed — a small safe expression evaluator over referenced meters' consumption/generation series), e.g. `self_consumption = generation − grid_feed_in`, `savings = self_consumption * unit_price`. Persist results to `consumption` (or compute on read — decide per §14). This is how PV self-consumption/savings and net figures are modelled without hardcoding.

### 7.5 Cost
`cost(bucket) = Σ(consumption_amount × active_unit_price) + base_price(prorated) − feed_in_credit − bonus`. Prices resolved by date from `tariff` (time-ranged). Currency from `app_setting`. Provide monthly-price and day-accurate-proration modes (§5.4). Categories roll costs up per `cost_category`; add `manual_cost` for meter-less categories (pool).

---

## 8. Dashboard & UX

### 8.1 Overview
- KPI cards: **Today**, **This month**, **This year** cost — each with Δ (absolute + %) vs the previous comparable period and an ↑/↓ indicator.
- "Cost now" total across all categories.

### 8.2 Cost breakdown / "what costs most"
- Stacked bar or donut by `cost_category` for a selectable period; ranked list (most → least).
- **Difference view** (explicitly requested): a table answering *"what cost more, what cost less this time"* — per category **and** per meter, **this month vs last month** and **this year vs last year**, columns `now | previous | Δ | Δ% | ↑/↓`, sorted by absolute impact.

### 8.3 Trends
- Consumption and cost over time; **granularity toggle** day/week/month/year; per-meter or per-category; **previous-year overlay**.

### 8.4 PV / Solar panel
- Generation, self-consumption, grid feed/draw, **savings (Ersparnis)**, **autarky %**, **self-consumption %**. Time-filtered.

### 8.5 Oil / consumable panel
- Tank level (cm + L), balance vs capacity gauge, deliveries log, burner runtime, effective **L/h** (fixed/empirical), **forecast to empty**, monthly cost.

### 8.6 Meter detail
- Raw readings, normalized consumption, source status (last-seen, last value), tariff timeline, events (swaps/deliveries/corrections), measured-vs-estimated markers.

### 8.7 Admin / config
- CRUD for energy types, meters, sources, tariffs, cost categories, connectors; retention & locale/currency settings; import wizard; API keys.

> **Legacy monthly history:** imported data is monthly-granular. Offer per-import choice: keep native monthly buckets, or **linearly interpolate to daily** (energietracker-style) so old and new data render on the same axes. Interpolated points are marked `quality = interpolated`.

---

## 9. REST API (v1)

OpenAPI/Swagger published. API-key auth for automation endpoints; UI uses the session.

| Method & path | Purpose |
|---------------|---------|
| `POST /api/v1/readings` | Ingest one/many readings (idempotent). Lets HA **push** instead of us pulling. |
| `GET  /api/v1/meters` · `POST/PUT/DELETE` | Meter CRUD. |
| `GET  /api/v1/energy-types` · CRUD | Energy-type CRUD. |
| `GET  /api/v1/consumption?meter=&from=&to=&bucket=` | Normalized consumption/generation. |
| `GET  /api/v1/cost?scope=&id=&from=&to=&bucket=` | Cost by meter/category/type. |
| `GET  /api/v1/dashboard/summary` | KPI cards + Δ payload. |
| `POST /api/v1/events` | Delivery, swap, tank level, correction. |
| `GET/POST /api/v1/tariffs` | Tariff CRUD (time-ranged). |
| `POST /api/v1/import` (multipart) | CSV import with a mapping profile. |
| `GET  /api/v1/sources/status` | Connector/source health. |
| `GET  /healthz` | Liveness/readiness (for Gatus). |

---

## 10. Non-functional

- **Time & DST:** store UTC; bucket and display in the instance timezone (default `Europe/Berlin`). "Daily cost" boundaries are local-midnight — use `time_bucket(..., 'Europe/Berlin')`.
- **Auth:** optional built-in local accounts; **reverse-proxy trust** mode honouring `X-Forwarded-User`/`Remote-User` behind Authelia/Traefik; API keys for machine access. Default: single admin user + one ingest API key.
- **Observability:** `/healthz`, structured logs (Serilog), optional Prometheus `/metrics`.
- **Config:** environment variables + a settings UI; secrets via env/Docker secrets (never in DB plaintext).
- **i18n:** `en` (default for OSS) + `de`; locale-aware number/currency/date. Ship a German locale that matches the source data conventions.
- **Backup:** document `pg_dump`/Timescale backup; provide a full **JSON export/import** for portability.
- **Performance:** dashboards read aggregates only; raw reads paginated and time-bounded.

---

## 11. Repository, CI, licensing

```
/                       CLAUDE.md, README.md, LICENSE, docker-compose.yml
/src
  /Core                 domain entities, enums, interfaces, expression eval
  /Infrastructure       EF Core + Npgsql, Dapper repos, Timescale SQL migrations,
                        MQTT client, HA client, CSV importer
  /App                  ASP.NET Core host: Blazor Server UI + REST API + hosted workers
/tests
  /Core.Tests           unit: deltas, swaps, tariff resolution, oil rate, CSV parsing
  /Integration.Tests    Testcontainers (Timescale): ingest→aggregate→cost e2e
  /fixtures             the 4 reference CSVs + expected outputs
/deploy
  Dockerfile            docker-compose.yml (app + timescaledb), unraid-template.xml
/docs                   architecture, setup, HA/Tasmota wiring, API, screenshots
```

- **CI (GitHub Actions):** build → test (spin Timescale) → publish Docker image to **GHCR** (amd64; add arm64 if desired) on tag.
- **License:** pick before release — **MIT** (max adoption; matches energietracker/your prior assets) or **AGPL-3.0** (keeps hosted forks open). Default suggestion: **MIT**, unless keeping SaaS forks open-source matters to you.
- **Docs:** a "wire up HA/Tasmota" guide is the highest-leverage doc for adoption.

---

## 12. Milestone roadmap (build order)

- **M0 — Scaffold.** Solution + 3 projects + tests; `docker-compose` with TimescaleDB; EF Core + first migration; `/healthz`. *Exit:* app boots against Timescale in Docker.
- **M1 — Domain & schema.** `energy_type`, `meter`, `meter_source`, `reading` hypertable, `consumption` hypertable + normalization pipeline (register/runtime/swap/reset), `tariff`, seed defaults. *Exit:* insert readings → correct `consumption`, unit-tested incl. swaps.
- **M2 — Manual entry & CSV import.** German-dialect importer (Appendix A), mapping profiles, dry-run + revertible batches; the 4 CSVs import and reconcile. *Exit:* importing the reference CSVs reproduces the spreadsheet's consumption/cost within tolerance (§13).
- **M3 — Live ingestion.** MQTTnet worker + Tasmota field mapping; HA connector (WebSocket + REST poll); idempotency; source status. *Exit:* a Tasmota plug and an HA entity land as readings automatically.
- **M4 — Aggregation & cost engine.** Continuous aggregates (hourly/daily/monthly/yearly); tariff-aware cost view (monthly + prorated); cost categories + manual costs. *Exit:* `GET /cost` and category rollups correct vs fixtures.
- **M5 — Dashboard.** Overview KPIs + Δ; cost breakdown + **difference view**; trends w/ granularity + prev-year overlay; PV panel; oil/consumable panel; meter detail. *Exit:* all §8 views render on imported + live data.
- **M6 — API & auth.** REST + OpenAPI; API keys; reverse-proxy trust. *Exit:* HA can push via `POST /readings`; Swagger published.
- **M7 — Release polish.** i18n (de/en); retention settings; JSON export/import; Unraid template; CI → GHCR; README + wiring guide. *Exit:* `docker compose up` from a clean host yields a working, documented instance.

---

## 13. Testing strategy

- **Unit:** consumption deltas incl. **water swap** (…861→2) and counter resets; **oil** empirical vs fixed L/h and forecast; tariff time-range resolution + mid-month proration; virtual-meter expression eval; **CSV parsing** of the exact reference dialect (decimal comma, unit suffixes, currency, `DD.MM.YYYY` vs `Monat YYYY`, summary-row skipping, zero-placeholder rows).
- **Integration (Testcontainers + Timescale):** end-to-end ingest → normalize → aggregate → cost; continuous-aggregate refresh; hypertable compression sanity.
- **Golden fixtures:** the four CSVs with expected monthly consumption/cost tables. A regression test asserts computed ≈ spreadsheet (define tolerance for rounding; the sheet rounds to cents / whole kWh).
- **Load smoke (optional):** synthetic 1000-meter × N-year generator to validate aggregate query latency and compression ratio.

---

## 14. Open questions & defaults

Pick the **default** and flag it if unsure; only ask when a question isn't listed here.

1. **Virtual meters: compute-on-write or compute-on-read?** *Default:* compute-on-read for dashboards, materialize to `consumption` only if a virtual meter is referenced by cost. (Avoids recompute storms; revisit if slow.)
2. **Legacy monthly import → daily interpolation or native monthly?** *Default:* offer both per import; interpolation off by default, points marked `interpolated`.
3. **Cost proration when price changes mid-month.** *Default:* day-accurate proration available, but the **displayed** monthly figure uses the month's dominant price to match the spreadsheet unless the user opts into proration.
4. **Instant-rate (power/flow) integration in v1?** *Default:* schema-supported, worker deferred to post-v1 (Tasmota already gives cumulative `ENERGY.Total`, so it's rarely needed).
5. **Multi-user?** *Default:* single admin + reverse-proxy trust; full accounts post-v1.
6. **.NET version pin.** *Default:* current LTS at implementation time; keep `TargetFramework` in one place.
7. **License.** *Default:* MIT unless you want AGPL's copyleft on hosted forks.
8. **Name.** `MeterVault` is a placeholder — decide before first public tag.

---

## Appendix A — CSV import dialect (from the reference files)

- **Field separator:** comma; fields quoted when they contain a comma.
- **Decimal separator:** comma (`180,8244706`). **Thousands separator:** dot (`2.940,19`).
- **Currency:** trailing `€` with a space (`120,00 €`); parse to `(amount, currency)`.
- **Unit suffixes on values:** strip and validate (`411kWh` → 411 kWh; `49` cm; `2287` L).
- **Dates:** `Monat YYYY` German month names for monthly tables; `DD.MM.YYYY` for event rows. Support both; store as timestamptz (month tables → first-of-month or period bucket).
- **Skip rows:** summary/label rows — `Total`, `Heute`, `Seitbeginn Tage`, `Seit YYYY`, and any embedded side-tables.
- **No-data rows:** all-zero future placeholders (e.g. Dec 2026) → ignore.
- **Negatives are valid** (savings, grid balance).
- **Column mapping is explicit** (a wizard), because these sheets pack multiple meters and derived columns side by side; ship saved mapping profiles for each of the four sheet shapes.

## Appendix B — Glossary (source ↔ model)

| Sheet term | Model concept |
|------------|---------------|
| Zähler (Haus/Netz/Auto/Solar) | `meter` (`cumulative_`/`generation_counter`) |
| Verbrauch / Erzeugung | `consumption.amount` (kind 0/1) |
| Solar Erzeugung / Eigenverbrauch / Ersparnis / Netz Einsparung | `virtual` meters via expressions |
| €/kWh, €/m³, €/100l | `tariff.unit_price` (time-ranged) |
| Grundpreis / Abschlag | `tariff.base_price` |
| Betriebststunden | `runtime_counter` meter |
| Verbrauch / Betrieb Stunde | derived L/h (`tank.rate_mode`) |
| Lieferungmenge | `meter_event` `delivery` |
| Füllstand cm / Tank Aktuell | `meter_event` `tank_level` / `tank.cached_balance` |
| Vorraussichtliches Ende | forecast-to-empty |
| Heizung / Strom / Wasser / Pool Betrieb | `cost_category` (Pool via `manual_cost`) |
| Zähler swap (…861→2) | `meter_event` `meter_swap` |