diff --git a/CLAUDE.md b/CLAUDE.md index 9250451..e72c413 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,7 +72,8 @@ sources (Tasmota/HA/MQTT/manual/CSV) **Invariants that shape everything:** -- **Raw `reading` is immutable audit truth.** Everything derived (consumption, cost, balances, forecasts) is computed on top and must be reproducible. Never mutate readings to fix a derived number. +- **Raw `reading` is immutable audit truth.** Everything derived (consumption, cost, balances, forecasts) is computed on top and must be reproducible. Never mutate readings to fix a derived number. Live ingestion recomputes the meter inline (`IngestionService.RenormalizeAsync`) — without it, polled readings never become consumption. +- **Long gaps are apportioned, short ones are not** (`GapAttribution`, SDD §7.1). An interval containing ≥2 whole calendar months is split across those months, proportional to elapsed time, marked `Estimated`. A monthly series contains exactly one and is untouched — that's what keeps the golden fixtures reconciling. `GapSplittingIsInertOnFixturesTests` asserts the rule declines to fire on the reference data, so this can't silently drift. - **Dashboards and charts read aggregates only — never scan `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). Raw is kept for a bounded window (default 3y); `consumption` + aggregates are the long-term source of truth. - **`meter.mode` (measurement mode) is the central abstraction** for how raw readings become consumption (SDD §5.2): `cumulative_counter`, `generation_counter`, `runtime_counter` (Δhours × rate), `consumable_balance` (tank: deliveries − usage + forecast), `direct_delta`, `instant_rate`, `virtual` (expression over other meters). New ingestion/normalization logic dispatches on mode. - **Nothing domain-specific is hardcoded.** Energy types are data. Cost **categories are decoupled from energy types** (Heizung may be oil today, heat-pump tomorrow). PV self-consumption/savings/net are **virtual meters** with user-defined expressions, not special-cased code. Tariffs are time-ranged (price history), scoped global / per-type / per-meter. diff --git a/docs/SDD.md b/docs/SDD.md index c8634bc..9a93c02 100644 --- a/docs/SDD.md +++ b/docs/SDD.md @@ -429,6 +429,10 @@ The key ring must be persisted outside the app directory (`MeterVault__DataProte ### 7.1 Register → consumption For `cumulative_counter`/`generation_counter`: for each new reading, `amount = value − previous_value`. Persist to `consumption`. Cross a `meter_swap` as `(old_final − prev) + (curr − new_initial)`; a `counter_reset` starts a fresh baseline. Ignore/annotate negative deltas that lack an explaining event (flag as anomaly). +**Gap attribution.** A delta is booked at the reading that closes it — correct at the reporting cadence, and what the reference sheets do. After a long unread stretch it misleads: 78 days of PV output arriving as one July row makes June look idle. So an interval containing **two or more complete calendar months** is apportioned across the months it covers, in proportion to elapsed time, and every row it yields is marked `quality = estimated` — the meter recorded a total, not a shape. + +The threshold is deliberately conservative. A monthly series contains exactly one whole month per interval and is never touched, which is what keeps the golden-fixture reconciliation (§13) measuring the normalizer rather than the splitter. Counting whole months *contained* rather than boundaries *crossed* keeps the rule stable when a reading lands hours late. Swap and reset amounts are never apportioned: they are explicit corrections booked at the event. Split points are UTC, so one can sit an hour or two from a displayed month edge (§10) — immaterial when dividing a multi-month gap, and the alternative is threading a timezone through an otherwise timezone-free engine. + ### 7.2 Runtime → consumption (burner) For `runtime_counter`: `amount = Δhours × rate`. `rate` comes from the linked `tank`: `fixed` (nozzle spec, L/h) or `empirical` (`Δlevel ÷ Δhours` measured between deliveries/level reads — reproduce the spreadsheet's 1.87/1.94/2.92 … behaviour). Expose both; default empirical when level data exists, else fixed. diff --git a/src/Core/Normalization/GapAttribution.cs b/src/Core/Normalization/GapAttribution.cs new file mode 100644 index 0000000..b300bf2 --- /dev/null +++ b/src/Core/Normalization/GapAttribution.cs @@ -0,0 +1,110 @@ +namespace MeterVault.Core.Normalization; + +/// +/// Spreads a register delta that spans several calendar months across the months it actually covers. +/// +/// +/// A counter delta is booked at the reading that closes it, which is right when readings arrive at +/// the reporting cadence: a monthly series books December's usage against the 1 January reading, and +/// that is what the reference spreadsheet does. It stops being right when a meter goes unread for a +/// long stretch — 78 days of PV generation arriving as a single July row makes June look idle and +/// July look extraordinary, when nothing unusual happened. +/// +/// Splitting is therefore deliberately conservative: an interval is divided only when it contains +/// two or more complete calendar months. A normal monthly series contains exactly one, so it +/// is left completely untouched and the golden-fixture reconciliation stands (SDD §13); a series that +/// skipped a month or more contains two or more, which is precisely where lumping misleads. +/// +/// Counting whole months contained, rather than boundaries crossed, is what makes this stable against +/// readings that do not land on midnight: a monthly reading arriving at 06:00 on the 1st still +/// contains one whole month, where a boundary count would tip over and hand the new month a sliver. +/// +/// The division is by elapsed time, so it assumes a flat rate across the gap. That is a guess — the +/// meter recorded a total, not a shape — so every row it produces is marked +/// . The sum is exact: the final segment absorbs any +/// rounding remainder, so a split never creates or destroys energy. +/// +/// Boundaries are UTC. The dashboard buckets in the instance timezone (SDD §10), so a split point +/// can sit an hour or two from the displayed month edge — immaterial for apportioning a multi-month +/// gap, and the alternative would be threading a timezone through the otherwise timezone-free engine. +/// +public static class GapAttribution +{ + /// + /// True when an interval contains two or more complete calendar months, and so would misattribute + /// a long gap to its closing month. + /// + public static bool ShouldSplit(DateTimeOffset start, DateTimeOffset end) => + end > start && WholeMonthsInside(start, end) >= 2; + + /// + /// Divides across the calendar months between the two instants, + /// proportionally to the time spent in each. + /// + /// + /// Each segment is stamped at its end, which keeps the existing convention that a + /// consumption row records the period ending at its timestamp — the same reason an unsplit delta + /// sits on its closing reading, and the reason the reference sheet's January row carries + /// December's usage. So the share covering May is stamped 1 June and buckets as June, exactly as + /// a May-to-June monthly reading pair already would. The last segment therefore keeps the closing + /// reading's own timestamp, and nothing shifts relative to how unsplit intervals are labelled. + /// + public static IReadOnlyList Split(DateTimeOffset start, DateTimeOffset end, double amount) + { + if (end <= start) + { + return [new GapSegment(end, amount)]; + } + + var total = end - start; + var segments = new List(); + var cursor = start; + var assigned = 0d; + + while (cursor < end) + { + var nextBoundary = NextMonthStart(cursor); + var segmentEnd = nextBoundary < end ? nextBoundary : end; + + if (segmentEnd >= end) + { + // Final segment takes the remainder, so the parts always sum to the original. + segments.Add(new GapSegment(end, amount - assigned)); + break; + } + + var share = amount * ((segmentEnd - cursor) / total); + segments.Add(new GapSegment(segmentEnd, share)); + assigned += share; + cursor = segmentEnd; + } + + return segments; + } + + private static int WholeMonthsInside(DateTimeOffset start, DateTimeOffset end) + { + // A month counts only if it lies entirely within the interval, so a partial month at either + // edge never tips the decision. + var cursor = MonthStart(start) == start.ToUniversalTime() ? MonthStart(start) : NextMonthStart(start); + var whole = 0; + while (cursor.AddMonths(1) <= end) + { + whole++; + cursor = cursor.AddMonths(1); + } + + return whole; + } + + private static DateTimeOffset MonthStart(DateTimeOffset instant) + { + var utc = instant.ToUniversalTime(); + return new DateTimeOffset(utc.Year, utc.Month, 1, 0, 0, 0, TimeSpan.Zero); + } + + private static DateTimeOffset NextMonthStart(DateTimeOffset instant) => MonthStart(instant).AddMonths(1); +} + +/// One month's share of a spread gap: the instant it closes and the amount attributed. +public sealed record GapSegment(DateTimeOffset Time, double Amount); diff --git a/src/Core/Normalization/Normalizers/CounterNormalizerBase.cs b/src/Core/Normalization/Normalizers/CounterNormalizerBase.cs index d4ce199..caf8cc4 100644 --- a/src/Core/Normalization/Normalizers/CounterNormalizerBase.cs +++ b/src/Core/Normalization/Normalizers/CounterNormalizerBase.cs @@ -12,7 +12,10 @@ namespace MeterVault.Core.Normalization.Normalizers; /// reconciles to 12), otherwise (oldFinal − prev) + (curr − newInitial); /// counter reset → baseline restarts at NewValue (default 0); /// unexplained decrease → 0 with an anomaly flagged (never a silent negative), rebaselined -/// to the current value. +/// to the current value; +/// a plain increase spanning two or more whole calendar months → apportioned across them +/// and marked estimated (), so an unread stretch does not land wholly +/// in its closing month. A monthly cadence never triggers this. /// /// public abstract class CounterNormalizerBase : IMeterNormalizer @@ -45,6 +48,7 @@ public abstract class CounterNormalizerBase : IMeterNormalizer var swap = FindEvent(swaps, previousTime, reading.Time); double amount; + var plainIncrease = false; if (swap is { EventType: MeterEventType.MeterSwap }) { amount = swap.Amount @@ -57,6 +61,7 @@ public abstract class CounterNormalizerBase : IMeterNormalizer else if (reading.Value >= previous) { amount = reading.Value - previous; + plainIncrease = true; } else { @@ -65,15 +70,40 @@ public abstract class CounterNormalizerBase : IMeterNormalizer quality = ReadingQuality.Estimated; } - yield return new Consumption + // Only a plain increase over an unread stretch is worth apportioning (SDD §7.1). A swap + // or reset amount is an explicit correction booked at its event; a rejected decrease + // contributes nothing; the first reading has no interval behind it; and fanning a zero + // out across three months just adds rows that say nothing. + var gapStart = plainIncrease && Math.Abs(amount) > 1e-9 ? previousTime : null; + + if (gapStart is { } start && GapAttribution.ShouldSplit(start, reading.Time)) { - MeterId = context.Meter.MeterId, - Time = reading.Time, - Amount = amount, - Kind = Kind, - Quality = quality, - ImportBatchId = reading.ImportBatchId, - }; + foreach (var segment in GapAttribution.Split(start, reading.Time, amount)) + { + yield return new Consumption + { + MeterId = context.Meter.MeterId, + Time = segment.Time, + Amount = segment.Amount, + Kind = Kind, + // The total is measured; only its distribution across the gap is inferred. + Quality = ReadingQuality.Estimated, + ImportBatchId = reading.ImportBatchId, + }; + } + } + else + { + yield return new Consumption + { + MeterId = context.Meter.MeterId, + Time = reading.Time, + Amount = amount, + Kind = Kind, + Quality = quality, + ImportBatchId = reading.ImportBatchId, + }; + } previous = reading.Value; previousTime = reading.Time; diff --git a/tests/Core.Tests/GapAttributionTests.cs b/tests/Core.Tests/GapAttributionTests.cs new file mode 100644 index 0000000..283ec96 --- /dev/null +++ b/tests/Core.Tests/GapAttributionTests.cs @@ -0,0 +1,208 @@ +using MeterVault.Core.Domain; +using MeterVault.Core.Normalization; +using static MeterVault.Core.Tests.TestData; + +namespace MeterVault.Core.Tests; + +/// +/// A counter delta is booked at the reading that closes it. That is correct at the reporting cadence +/// and wrong after a long outage, so a gap containing two or more whole months is apportioned. +/// The boundary between those two behaviours is what these pin down: a normal monthly series must +/// come out byte-for-byte unchanged, because it is what reconciles against the reference spreadsheet. +/// +public sealed class GapAttributionTests +{ + private readonly INormalizationEngine _engine = NormalizationEngine.CreateDefault(); + + [Fact] + public void A_monthly_cadence_is_never_split() + { + // One whole month per interval — the reference-data shape. Splitting here would move energy + // between months and break reconciliation (SDD §13). + Assert.False(GapAttribution.ShouldSplit(Month(2023, 1), Month(2023, 2))); + Assert.False(GapAttribution.ShouldSplit(Month(2023, 1), Month(2023, 2).AddDays(-1))); + + // A reading that lands hours late must not tip the rule and hand January a sliver. + Assert.False(GapAttribution.ShouldSplit(Month(2023, 12), Month(2024, 1).AddHours(6))); + + // Nor should a six-week interval, which still contains only one whole month. + Assert.False(GapAttribution.ShouldSplit(Month(2023, 1), Month(2023, 2).AddDays(14))); + } + + [Fact] + public void Sub_month_intervals_are_never_split() + { + Assert.False(GapAttribution.ShouldSplit(Month(2023, 5), Month(2023, 5).AddHours(1))); + Assert.False(GapAttribution.ShouldSplit(Month(2023, 5).AddDays(10), Month(2023, 5).AddDays(20))); + } + + [Fact] + public void A_skipped_month_is_split() + { + Assert.True(GapAttribution.ShouldSplit(Month(2023, 1), Month(2023, 3))); + Assert.True(GapAttribution.ShouldSplit(Month(2026, 5), new DateTimeOffset(2026, 7, 18, 15, 33, 0, TimeSpan.Zero))); + } + + [Fact] + public void Splitting_preserves_the_total_and_keeps_the_closing_timestamp() + { + var start = Month(2026, 5); + var end = new DateTimeOffset(2026, 7, 18, 15, 33, 0, TimeSpan.Zero); + + var segments = GapAttribution.Split(start, end, 714.5); + + // May, June, July. + Assert.Equal(3, segments.Count); + Assert.Equal(714.5, segments.Sum(s => s.Amount), 6); + Assert.Equal(end, segments[^1].Time); + Assert.Equal(Month(2026, 6), segments[0].Time); + Assert.Equal(Month(2026, 7), segments[1].Time); + } + + [Fact] + public void Each_month_gets_a_share_proportional_to_the_time_it_covers() + { + // Exactly two whole months: an even split, to the cent. + var segments = GapAttribution.Split(Month(2023, 1), Month(2023, 3), 620); + + Assert.Equal(2, segments.Count); + var januaryShare = 31d / 59d; // 2023 is not a leap year: Jan 31 + Feb 28. + Assert.Equal(620 * januaryShare, segments[0].Amount, 6); + Assert.Equal(620, segments.Sum(s => s.Amount), 6); + } + + [Fact] + public void A_gap_in_a_counter_series_is_spread_and_marked_estimated() + { + var ctx = new NormalizationContext + { + Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "kWh" }, + Readings = + [ + Reading(1, Month(2023, 1), 1000), + Reading(1, Month(2023, 4), 1900), // three months in one reading + ], + }; + + var result = _engine.Normalize(ctx).ToList(); + + // Baseline row for the first reading, then Jan/Feb/Mar shares of the 900 gap. + Assert.Equal(4, result.Count); + Assert.Equal(1000 + 900, result.Sum(c => c.Amount), 6); + + var spread = result.Skip(1).ToList(); + Assert.All(spread, c => Assert.Equal(ReadingQuality.Estimated, c.Quality)); + Assert.Equal(900, spread.Sum(c => c.Amount), 6); + } + + [Fact] + public void An_ordinary_monthly_series_produces_one_measured_row_per_reading() + { + // The regression that matters: this is the reference-data shape, and it must not gain rows + // or lose its quality markers. + var ctx = new NormalizationContext + { + Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "kWh" }, + Readings = + [ + Reading(1, Month(2022, 9), 0), + Reading(1, Month(2022, 10), 411), + Reading(1, Month(2022, 11), 1153), + Reading(1, Month(2022, 12), 1968), + ], + }; + + var result = _engine.Normalize(ctx).ToList(); + + Assert.Equal(4, result.Count); + Assert.DoesNotContain(result, c => c.Quality == ReadingQuality.Estimated); + Assert.Equal([0, 411, 742, 815], result.Select(c => c.Amount).ToArray()); + } + + [Fact] + public void The_observed_solar_gap_is_apportioned_across_the_months_it_covers() + { + // The case this exists for: Solar 1 read monthly to 1 May 2026, then a single live reading on + // 18 July. 714.5 kWh of generation arriving as one July row made June look like an outage. + var ctx = new NormalizationContext + { + Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.GenerationCounter, Unit = "kWh" }, + Readings = + [ + Reading(1, Month(2026, 4), 10308), + Reading(1, Month(2026, 5), 10731), + Reading(1, new DateTimeOffset(2026, 7, 18, 15, 33, 0, TimeSpan.Zero), 11445.5), + ], + }; + + var result = _engine.Normalize(ctx).ToList(); + var gap = result.Where(c => c.Time > Month(2026, 5)).ToList(); + + Assert.Equal(3, gap.Count); + Assert.Equal(714.5, gap.Sum(c => c.Amount), 6); + + // No single month swallows the whole gap any more. + Assert.All(gap, c => Assert.True(c.Amount < 714.5 * 0.75, $"{c.Time:yyyy-MM-dd} took {c.Amount:0.#}")); + + // Generation is preserved end to end: baseline 0 → 11445.5. + Assert.Equal(11445.5, result.Sum(c => c.Amount), 6); + } + + [Fact] + public void An_unchanged_register_across_a_long_gap_does_not_fan_out_into_empty_rows() + { + // Nothing was used. Three rows of zero say no more than one, and would dilute the + // measured/estimated ratio on the detail page. + var ctx = new NormalizationContext + { + Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "kWh" }, + Readings = [Reading(1, Month(2023, 1), 500), Reading(1, Month(2023, 5), 500)], + }; + + var result = _engine.Normalize(ctx).ToList(); + + Assert.Equal(2, result.Count); + Assert.Equal(0, result[^1].Amount, 6); + } + + [Fact] + public void A_rejected_decrease_across_a_long_gap_stays_a_single_row() + { + // The decrease branch already yields 0 and rebaselines; spreading that zero would invent + // rows for months the meter never reported. + var ctx = new NormalizationContext + { + Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "kWh" }, + Readings = [Reading(1, Month(2023, 1), 900), Reading(1, Month(2023, 5), 100)], + }; + + var result = _engine.Normalize(ctx).ToList(); + + Assert.Equal(2, result.Count); + Assert.Equal(0, result[^1].Amount, 6); + Assert.Equal(Month(2023, 5), result[^1].Time); + } + + [Fact] + public void A_swap_across_a_long_gap_keeps_its_explicit_amount_in_one_row() + { + // Swap amounts are corrections booked at the event (the water …861 → 2 case reconciles to + // 12). Apportioning one across the gap would silently rewrite a number the operator supplied. + var ctx = new NormalizationContext + { + Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "m³" }, + Readings = + [ + Reading(1, Month(2023, 1), 861), + Reading(1, Month(2023, 5), 15), + ], + Events = [Swap(1, Month(2023, 3), prevValue: 861, newValue: 2, amount: 12)], + }; + + var result = _engine.Normalize(ctx).ToList(); + + Assert.Equal(2, result.Count); + Assert.Equal(12, result[^1].Amount, 6); + Assert.NotEqual(ReadingQuality.Estimated, result[^1].Quality); + } +} diff --git a/tests/Integration.Tests/Reconciliation/GapSplittingIsInertOnFixturesTests.cs b/tests/Integration.Tests/Reconciliation/GapSplittingIsInertOnFixturesTests.cs new file mode 100644 index 0000000..f2392fb --- /dev/null +++ b/tests/Integration.Tests/Reconciliation/GapSplittingIsInertOnFixturesTests.cs @@ -0,0 +1,54 @@ +using MeterVault.Core.Domain; +using MeterVault.Core.Normalization; +using MeterVault.Infrastructure.Import; +using static MeterVault.Integration.Tests.Reconciliation.ReconciliationSupport; + +namespace MeterVault.Integration.Tests.Reconciliation; + +/// +/// Gap splitting apportions a long unread stretch across the months it covers. The reference sheets +/// are read monthly and must never trigger it, or their months would silently shift and the whole +/// golden-fixture oracle (SDD §13) would be measuring the splitter instead of the normalizer. +/// +/// +/// The reconciliation suites already compare month by month, so a spurious split would surface there +/// as a numeric failure. This asserts the mechanism directly instead of relying on that side effect: +/// it proves the rule was evaluated against real fixture cadence and declined to fire, rather than +/// the fixtures simply having no gaps to find. +/// +public sealed class GapSplittingIsInertOnFixturesTests +{ + [Theory] + [InlineData(ReferenceProfiles.Haus, MeterMode.CumulativeCounter)] + [InlineData(ReferenceProfiles.Netz, MeterMode.CumulativeCounter)] + [InlineData(ReferenceProfiles.Auto, MeterMode.CumulativeCounter)] + [InlineData(ReferenceProfiles.Solar1, MeterMode.GenerationCounter)] + [InlineData(ReferenceProfiles.Solar2, MeterMode.GenerationCounter)] + public void Electricity_meters_produce_exactly_one_row_per_reading(int meterId, MeterMode mode) + { + var staged = Stage(ReferenceProfiles.Electricity(), Electricity); + var readings = staged.Readings.Count(r => r.MeterId == meterId); + + var computed = Normalize(staged, new MeterConfig { MeterId = meterId, Mode = mode, Unit = "kWh" }); + + Assert.True(readings > 20, $"meter {meterId}: expected a real series, got {readings} readings."); + Assert.Equal(readings, computed.Count); + } + + [Fact] + public void No_fixture_interval_is_long_enough_to_split() + { + var staged = Stage(ReferenceProfiles.Electricity(), Electricity); + + foreach (var group in staged.Readings.GroupBy(r => r.MeterId)) + { + var times = group.Select(r => r.Time).OrderBy(t => t).ToList(); + for (var i = 1; i < times.Count; i++) + { + Assert.False( + GapAttribution.ShouldSplit(times[i - 1], times[i]), + $"meter {group.Key}: {times[i - 1]:yyyy-MM-dd} → {times[i]:yyyy-MM-dd} would be split."); + } + } + } +}