HeiKyu/Rendezvous
2
0
Fork 0
Code Issues 27 Pull Requests Actions Packages Projects Releases Wiki Activity

Establish capacity, soak, resilience, and availability topology gates #18

New Issue
Open
opened 2026-07-15 10:04:28 +02:00 by Kyuubi · 2 comments
Kyuubi commented 2026-07-15 10:04:28 +02:00
Owner
Copy Link
Copy Source

Parent: #1
Depends on: #15, #16, #17

Outcome

Turn the SLO/profile decisions from #2 into repeatable evidence and decide whether single-active remains sufficient.

Scope

  • Benchmark registration/renewal, browsing, join issuance, simultaneous punch attempts, expiry cleanup, revocation, and telemetry overhead.
  • Run long soaks for memory, handles, queues, replay caches, endpoint state, and cleanup.
  • Inject restart, clock skew, listener failure, secret rotation, dependency/store failure, overload, and graceful drain.
  • Publish supported and capacity profiles, latency/error budgets, bandwidth/amplification observations, and scaling signals.
  • Validate single-active restart/RTO behavior.
  • If SLOs require multiple instances, add shared TTL/CAS state plus deterministic mediator-node routing or source-preserving VIP behavior; prove two-instance atomic pairing.
  • Keep multi-region and relay out until separately justified.

Acceptance criteria

  • Target workload and SLOs are numeric, approved, and reproducible.
  • Soak tests show bounded stable state with no unbounded queue/cache growth.
  • Restart/drain behavior produces typed client failures and successful host re-registration.
  • Load shedding protects established work and the service recovers after saturation.
  • The chosen availability topology is documented with failure domains, RTO, rollout, and rollback.
  • Any multi-instance claim has a two-node integration test proving endpoint binding and attempt pairing cannot split/cross-wire.
  • Results are stored as versioned release evidence.
Parent: #1 Depends on: #15, #16, #17 ## Outcome Turn the SLO/profile decisions from #2 into repeatable evidence and decide whether single-active remains sufficient. ## Scope - Benchmark registration/renewal, browsing, join issuance, simultaneous punch attempts, expiry cleanup, revocation, and telemetry overhead. - Run long soaks for memory, handles, queues, replay caches, endpoint state, and cleanup. - Inject restart, clock skew, listener failure, secret rotation, dependency/store failure, overload, and graceful drain. - Publish supported and capacity profiles, latency/error budgets, bandwidth/amplification observations, and scaling signals. - Validate single-active restart/RTO behavior. - If SLOs require multiple instances, add shared TTL/CAS state plus deterministic mediator-node routing or source-preserving VIP behavior; prove two-instance atomic pairing. - Keep multi-region and relay out until separately justified. ## Acceptance criteria - Target workload and SLOs are numeric, approved, and reproducible. - Soak tests show bounded stable state with no unbounded queue/cache growth. - Restart/drain behavior produces typed client failures and successful host re-registration. - Load shedding protects established work and the service recovers after saturation. - The chosen availability topology is documented with failure domains, RTO, rollout, and rollback. - Any multi-instance claim has a two-node integration test proving endpoint binding and attempt pairing cannot split/cross-wire. - Results are stored as versioned release evidence.
Kyuubi added reference codex/issue-18-capacity-resilience 2026-07-16 15:04:00 +02:00
Kyuubi self-assigned this 2026-07-16 15:04:01 +02:00
Kyuubi commented 2026-07-16 15:04:10 +02:00
Author
Owner
Copy Link
Copy Source

Starting implementation on codex/issue-18-capacity-resilience, stacked on the completed #17 commit 08729ae.

I’ll first ground the numeric SLO/profile decisions and existing load/resilience harnesses, then add reproducible benchmark/soak/fault gates and versioned evidence. Single-active remains the default unless measured evidence proves it cannot meet the approved target; multi-instance/shared-state work will not be claimed without the required two-node atomic-pairing proof.

Starting implementation on `codex/issue-18-capacity-resilience`, stacked on the completed #17 commit `08729ae`. I’ll first ground the numeric SLO/profile decisions and existing load/resilience harnesses, then add reproducible benchmark/soak/fault gates and versioned evidence. Single-active remains the default unless measured evidence proves it cannot meet the approved target; multi-instance/shared-state work will not be claimed without the required two-node atomic-pairing proof.
Kyuubi commented 2026-07-16 16:11:58 +02:00
Author
Owner
Copy Link
Copy Source

Issue #18 implementation is complete on codex/issue-18-capacity-resilience and pushed through 07004cd.

Delivered:

  • bounded O(1)/heap-indexed ephemeral-state expiry and quota accounting, including reverse indexes for coincident listing/attempt/outcome cleanup;
  • queue-growth regressions for 10,000 lease/presence/revocation refreshes and extension-only principal-revocation semantics;
  • typed restart recovery through fresh registration, presence rebinding, and browser visibility;
  • production-process SIGTERM/socket release, same-endpoint replacement readiness under 15 seconds, and a real 10-second HTTP/UDP transport soak with handle/RSS bounds plus typed shedding/recovery;
  • reproducible quick/candidate capacity harness, CI gate, topology/RTO/failure-domain/rollout/rollback guidance, and truthful separation of core evidence from the #23 public-network canary;
  • exact versioned v2 candidate evidence at docs/evidence/capacity/v2/candidate-2cpu.json.

Formal clean-source candidate (cf14836, two exposed CPUs) passed:

  • 25,000 listings / 10,000 attempts;
  • 56.38% capacity-phase CPU;
  • 169,705,472-byte peak working set;
  • 29.69 ms coincident full-ceiling expiry cleanup;
  • 75,126,848 lifecycle cycles over 300 seconds;
  • seven scheduled expiry entries per live lifecycle, -257,288 managed-byte growth, +2 handles;
  • zero final retained state and no failures.

Verification:

  • Debug build + 287/287 tests;
  • Release build + 287/287 tests;
  • exact resilience/capacity gate: 17/17 selected tests plus quick profile;
  • branch audit/re-review: no remaining actionable findings;
  • formatter, shell syntax, systemd unit verification, API drift, JSON exactness, and diff checks clean.

The issue remains open while the stacked roadmap branch continues toward main; no multi-instance claim was made, so the two-node test is correctly not applicable.

Issue #18 implementation is complete on `codex/issue-18-capacity-resilience` and pushed through `07004cd`. Delivered: - bounded O(1)/heap-indexed ephemeral-state expiry and quota accounting, including reverse indexes for coincident listing/attempt/outcome cleanup; - queue-growth regressions for 10,000 lease/presence/revocation refreshes and extension-only principal-revocation semantics; - typed restart recovery through fresh registration, presence rebinding, and browser visibility; - production-process SIGTERM/socket release, same-endpoint replacement readiness under 15 seconds, and a real 10-second HTTP/UDP transport soak with handle/RSS bounds plus typed shedding/recovery; - reproducible quick/candidate capacity harness, CI gate, topology/RTO/failure-domain/rollout/rollback guidance, and truthful separation of core evidence from the #23 public-network canary; - exact versioned v2 candidate evidence at `docs/evidence/capacity/v2/candidate-2cpu.json`. Formal clean-source candidate (`cf14836`, two exposed CPUs) passed: - 25,000 listings / 10,000 attempts; - 56.38% capacity-phase CPU; - 169,705,472-byte peak working set; - 29.69 ms coincident full-ceiling expiry cleanup; - 75,126,848 lifecycle cycles over 300 seconds; - seven scheduled expiry entries per live lifecycle, -257,288 managed-byte growth, +2 handles; - zero final retained state and no failures. Verification: - Debug build + 287/287 tests; - Release build + 287/287 tests; - exact resilience/capacity gate: 17/17 selected tests plus quick profile; - branch audit/re-review: no remaining actionable findings; - formatter, shell syntax, systemd unit verification, API drift, JSON exactness, and diff checks clean. The issue remains open while the stacked roadmap branch continues toward main; no multi-instance claim was made, so the two-node test is correctly not applicable.
Sign in to join this conversation.
codex/issue-18-capacity-resilience
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Kyuubi schmidt.florian (Florian Schmidt)
No Assignees Kyuubi
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: HeiKyu/Rendezvous#18
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?