diff --git a/README.md b/README.md
index b65d428..fd6b838 100644
--- a/README.md
+++ b/README.md
@@ -100,6 +100,8 @@ defined in [hostile-input and overload protection](docs/security/abuse-protectio
Health semantics, bounded telemetry, alerting, audit privacy, and the authenticated
operator controls are defined in the
[observability and operator runbook](docs/operations/observability-and-operator-runbook.md).
+Concrete detect/contain/recover/verify procedures are in the
+[incident and change runbooks](docs/operations/incident-runbooks.md).
The pinned non-root container, production topology, graceful drain, Linux
hardening, smoke procedure, and recovery lifecycle are documented in
[secure single-active Linux deployment](docs/deployment/linux.md).
@@ -111,6 +113,9 @@ signing, staged promotion, rollback, and migration are defined in
[releases and compatibility](docs/releases/README.md).
The scriptable host/browser/join diagnostic and its stable automation contract are
documented in the [TestClient integration guide](docs/integration/test-client.md).
+The package, gameplay-socket, host-admission, provisioning, metadata, key rotation,
+versioning, and secure rollout seams are in the
+[game integration guide](docs/integration/sdk-seams.md).
The always-on three-party scenarios, optional Linux namespace topology, and
simulation limits are documented in the
[deterministic topology harness](docs/integration/topology-harness.md).
diff --git a/docs/deployment/linux.md b/docs/deployment/linux.md
index 27fe280..a805c41 100644
--- a/docs/deployment/linux.md
+++ b/docs/deployment/linux.md
@@ -177,6 +177,9 @@ dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
For the local Compose profile, the script derives a ten-minute diagnostic
publisher credential from the ignored local key without printing either secret.
+The fixed-scope helper used by the smoke can also support the manual
+[TestClient local flow](../integration/test-client.md); it is deliberately not a
+production issuer.
For production, do not copy the signing key to the smoke host. Instead inject a
short-lived, region-scoped credential through
`RENDEZVOUS_PUBLISHER_CREDENTIAL`, and set the external endpoints:
diff --git a/docs/integration/sdk-seams.md b/docs/integration/sdk-seams.md
new file mode 100644
index 0000000..0973552
--- /dev/null
+++ b/docs/integration/sdk-seams.md
@@ -0,0 +1,227 @@
+# Game integration seams
+
+Tracking: #20
+
+Use the [TestClient start-to-finish guide](test-client.md) before integrating a
+game. It proves the service and network path without engine or game code. This
+page documents only the seams that the diagnostic cannot choose for a game:
+package/version policy, ownership of the gameplay socket, host admission,
+metadata, credential custody, and deployment compatibility.
+
+## Packages and compatibility
+
+Consume `FinalFactory.Rendezvous.Client` and
+`FinalFactory.Rendezvous.Contracts` from the approved Gitea NuGet source and pin
+both to the same exact released version. Do not use a floating version range.
+The current release matrix is machine-readable in
+[`compatibility.json`](../releases/compatibility.json); the same window is
+available from authenticated `GET /v1/operator/status`.
+
+The authoritative package feed is
+`https://git.finalfactory.de/api/packages/HeiKyu/nuget/index.json`. Add it to the
+consumer's `NuGet.config` and map only Rendezvous packages to it; retain the
+consumer's existing NuGet.org mapping for other dependencies:
+
+```xml
+
+
+
+
+
+
+
+
+
+
+
+```
+
+When the feed is anonymously readable, no reader credential is needed. If
+registry policy requires authentication, use the platform's NuGet credential
+provider or a protected per-user/CI NuGet configuration populated by the secret
+manager. Never put a registry token in the project file, repository,
+package-source URL, or `dotnet` command argument.
+
+```xml
+
+
+
+
+```
+
+Run `dotnet restore`, then `dotnet list package --include-transitive` and verify
+that Client and Contracts resolve to the same exact version and LiteNetLib to the
+release matrix version before compiling the game.
+
+Release 1.0.0 targets `netstandard2.1`, requires LiteNetLib `2.1.4`, speaks HTTP,
+UDP, and connection-ticket contract version `1`, and requires an exact
+tenant-configured gameplay protocol match. A package patch does not silently
+change a wire version. Follow the [release and migration policy](../releases/README.md)
+when changing any dimension, and validate the generated
+[OpenAPI v1 document](../api/rendezvous-v1.json) rather than hand-building HTTP.
+
+## One caller-owned gameplay socket
+
+Create the game's LiteNetLib manager through `RendezvousNetListener`; do not open
+a separate NAT socket. The game owns start, stop, and disposal. A coordinator
+owns polling while it is active, so call its `Poll()` once from the game/network
+thread and do not also call `NetManager.PollEvents()` during that period.
+
+```csharp
+RendezvousNetListener networkEvents = new();
+NetManager gameplayNetwork = networkEvents.CreateManager();
+if (!gameplayNetwork.Start(gameplayPort))
+{
+ throw new InvalidOperationException("Gameplay UDP socket could not start.");
+}
+
+using RendezvousHostCoordinator host = new(
+ gameplayNetwork,
+ networkEvents,
+ mediatorEndPoint,
+ publishedSession,
+ joinClient);
+
+host.Poll(); // call each game frame while this coordinator owns polling
+```
+
+Register normal game callbacks on `networkEvents.GameplayEvents`. Rendezvous
+reserves only its authenticated direct requests and forwards other callbacks.
+The same socket sends host presence, punches through the mediator, establishes
+the peer, and then carries gameplay. A NAT introduction is not success; accept a
+peer only after the coordinator reports the typed `Connected` outcome.
+
+`Poll()` does not fetch new invitations. Schedule
+`RefreshJoinAttemptsAsync` repeatedly for the entire hosting lifetime using a
+bounded caller-owned timer (the diagnostic uses 250 ms), never allow two refreshes
+to overlap, and inspect each typed result. The refresh performs HTTP work and
+queues a snapshot; it does not call the LiteNetLib manager. Continue calling
+`Poll()` on the manager's owning thread so the queued snapshot, presence traffic,
+and callbacks are processed. Run the lease maintainer concurrently and cancel
+both loops before disposing the coordinator.
+
+For example, start one sequential refresh loop when hosting begins and await it
+during shutdown:
+
+```csharp
+static async Task RefreshInvitationsAsync(
+ RendezvousHostCoordinator host,
+ CancellationToken cancellationToken)
+{
+ using PeriodicTimer timer = new(TimeSpan.FromMilliseconds(250));
+ do
+ {
+ RendezvousClientResult result =
+ await host.RefreshJoinAttemptsAsync(cancellationToken);
+ if (!result.IsSuccess)
+ {
+ ObserveBoundedHostRefreshFailure(result.Error);
+ }
+ }
+ while (await timer.WaitForNextTickAsync(cancellationToken));
+}
+```
+
+On the joining side, create an attempt through `RendezvousJoinClient`, then give
+the issued attempt to `RendezvousClientCoordinator` using the same manager and
+listener. Cancellation, outcome reporting, bounded deadlines, fallback, and
+lease-maintainer examples are in the packaged
+[`FinalFactory.Rendezvous.Client` README](../../src/FinalFactory.Rendezvous.Client/README.md).
+
+## Host admission remains game-owned
+
+The coordinator privately validates and consumes the signed one-time connection
+ticket before accepting the LiteNetLib transport request. Do not create a second
+`ConnectionTicketValidator` beside it: the coordinator deliberately does not
+expose the expected or presented ticket. A connected transport proves only that
+Rendezvous authorized one attempt; it does not prove player identity,
+entitlement, capacity, ban status, or gameplay compatibility.
+
+Treat `AttemptCompleted` with a successful outcome and non-null `Peer` as the
+start of game-owned admission. Keep that peer outside authoritative gameplay
+until the game's normal authentication and admission exchange succeeds; disconnect
+it on rejection or timeout:
+
+```csharp
+host.AttemptCompleted += (_, completed) =>
+{
+ if (!completed.Outcome.IsSuccess || completed.Peer is null)
+ {
+ return;
+ }
+
+ BeginBoundedGameAuthentication(
+ completed.Peer,
+ onAccepted: AdmitToAuthoritativeGameplay,
+ onRejected: peer => peer.Disconnect());
+};
+```
+
+Revoke an attempt when the game cancels it. Never log a ticket or capability. A
+successful Rendezvous check must not bypass the game's authentication or
+authoritative server rules. `ConnectionTicketValidator` is a lower-level
+primitive for a custom transport integration that owns the complete request
+acceptance path; it is not an extra gate for `RendezvousHostCoordinator`.
+
+## Provision each game and environment
+
+Provision game/environment scope before issuing credentials. The policy fixes
+enabled regions, exact gameplay protocols, visibility and publisher trust modes,
+metadata schema and byte budgets, quotas, and whether a dedicated fallback may
+be published. Unknown or disabled scope fails closed. Follow
+[game provisioning and signing-key lifecycle](../security/provisioning.md) for
+the complete schema, principal kinds, secret providers, overlap, and revocation.
+
+Dedicated publisher credentials belong only on trusted hosting infrastructure.
+Never ship one in a player build, repository, image layer, appsettings file, URL,
+argument, log, crash report, or analytics event. Issue a short-lived credential
+scoped to one game/environment and its allowed regions from the trusted
+deployment boundary. Player-host grants are issued to an authenticated player
+session at runtime and are never embedded in the build. Player-host grants,
+dedicated publishers, anonymous unlisted hosts, and operators are separate
+principal kinds; do not interchange them.
+
+Rotate signing keys with an overlap:
+
+1. install a new authorized key inside its `NotBefore`/`SignUntil` window;
+2. begin issuing with it while the old key remains verify-only;
+3. wait at least the maximum credential lifetime plus allowed clock skew;
+4. retire the old verifier after `VerifyUntil` and preserve custody records.
+
+A suspected compromise is not routine rotation: stop issuance, revoke the exact
+key through the protected operator route, remove or replace it in provisioning,
+invalidate affected credentials, and follow the
+[key-compromise runbook](../operations/incident-runbooks.md#signing-key-or-issuer-compromise).
+
+## Metadata is public and policy-owned
+
+Treat listing metadata as untrusted public input. Define a small allowlist in
+each provisioned game's `MetadataValueMaxBytes`, set `RequiredMetadataKeys`, and
+keep `MetadataMaxKeys` and `MetadataMaxBytes` to the smallest useful values. Values
+must be display data only—for example a bounded map or ruleset identifier. Never
+publish player identity, free-form chat, secrets, access tokens, internal
+addresses, world state, or data needed for authoritative gameplay.
+
+The platform contract caps metadata at 32 keys, 256 UTF-8 bytes per value, and
+4096 encoded bytes total; tenant policy can and should be smaller. Build version
+and display name are separately bounded public fields. Games must escape metadata
+for their UI and must not infer trust from a listing being present.
+
+## Local, staging, and production path
+
+Use the checked-in Compose profile only for the local TestClient guide. For a
+real environment:
+
+1. provision the game/environment policy and externally held signing keys;
+2. deploy one active service behind the source-preserving HTTPS/UDP topology in
+ [secure single-active Linux deployment](../deployment/linux.md);
+3. install matching exact package versions in the game and set its service and
+ mediator endpoints through environment-specific configuration;
+4. pass the TestClient health/publish/browse/punch/direct-traffic smoke using a
+ short-lived diagnostic credential;
+5. run the topology harness and representative consumer-network trials; and
+6. monitor typed outcomes and bounded metrics before broad rollout.
+
+Rendezvous v1 has no relay, account system, matchmaking engine, server-browser
+UI, gameplay authority, or durable session database. A game owns player-facing
+recovery and an explicit fallback. Do not describe direct traversal as guaranteed.
diff --git a/docs/integration/test-client.md b/docs/integration/test-client.md
index 5ac8d69..83a9429 100644
--- a/docs/integration/test-client.md
+++ b/docs/integration/test-client.md
@@ -1,97 +1,222 @@
-# Diagnostic TestClient integration guide
+# Start-to-finish TestClient guide
-Tracking: #25
+Tracking: #20, #25
-`FinalFactory.Rendezvous.TestClient` is the smallest supported public-SDK consumer.
-It exists for integration development, CI smoke checks, deployment verification,
-and operator diagnosis. It is intentionally not a production game client, game
-server, matchmaking UI, or relay.
+`FinalFactory.Rendezvous.TestClient` is the supported executable proof that a
+consumer can publish, browse, authorize, punch, connect, exchange direct traffic,
+and diagnose a failure using only the public Client and Contracts packages. It is
+intentionally thin: a polished server browser and player-facing connection UI
+belong in each game repository.
-The automated scenario matrix, privileged Linux namespace run, and topology
-limitations are documented in the [deterministic topology harness](topology-harness.md).
+> **Traversal boundary:** Rendezvous v1 is not a relay and cannot guarantee a
+> connection through symmetric NAT, carrier-grade NAT, restrictive firewalls,
+> VPNs, or platform policy. It provides no accounts, social system, skill-based
+> matchmaking, gameplay server, gameplay authority, or gameplay transport.
-## Prerequisites
+The automated scenario matrix, privileged Linux namespace run, and simulation
+limits are in the [deterministic topology harness](topology-harness.md). The
+[SDK seam guide](sdk-seams.md) covers the few integration details that this
+executable cannot show.
-Start a configured Rendezvous service and note both its HTTP base URL and UDP
-mediator endpoint. The host needs a tenant-scoped publisher credential from the
-deployment secret boundary. Put it in an environment variable and pass only that
-variable's name when the default is unsuitable:
+## First local connection from a clean checkout
+
+Prerequisites are the pinned .NET SDK, Docker with Compose, OpenSSL, Python 3,
+`curl`, and `jq`. Run these commands from the repository root. The generated key
+and credential are disposable local fixtures, not production provisioning.
```bash
-export RENDEZVOUS_PUBLISHER_CREDENTIAL=''
+install -d -m 0700 deploy/compose/secrets
+umask 077
+openssl rand -out deploy/compose/secrets/signing-key 32
+export RENDEZVOUS_UID="$(id -u)"
+export RENDEZVOUS_GID="$(id -g)"
+test "$RENDEZVOUS_UID" -ne 0
+docker compose -f deploy/compose/compose.yaml up --build --detach
+ready=false
+for attempt in {1..45}; do
+ if curl --fail --silent http://127.0.0.1:8080/health/ready >/dev/null; then
+ ready=true
+ break
+ fi
+ sleep 1
+done
+test "$ready" = true
+curl --fail http://127.0.0.1:8080/health/live
+curl --fail http://127.0.0.1:8080/health/ready
+dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
```
-Never put the credential in a command argument, URL, checked-in configuration,
-shell trace, or captured test fixture. The development server's signing material
-is process-ephemeral; credentials from a prior development process are invalid.
-
-## Manual three-terminal flow
-
-Start the host:
+Only the host terminal needs a publisher credential. Disable shell tracing before
+capturing it; the helper prints the credential on stdout so command substitution
+can place it directly in the environment without writing it to disk.
```bash
-dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
- host --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
- --game space-game --environment development --region local --protocol 1
+set +x
+export RENDEZVOUS_PUBLISHER_CREDENTIAL="$(./scripts/mint-local-publisher-credential.sh)"
```
-Browse from another terminal:
+The helper accepts no arguments, reads the ignored `0600` local Compose key, and
+mints only `space-game` / `smoke` / `local` / protocol `1` for ten minutes. It is
+not a reusable issuer or an example for production. Never put the result in a
+command argument, URL, shell history, log, screenshot, support ticket, captured
+fixture, or source file.
+
+In terminal 1, publish a host. It stays alive for at most 60 seconds and exits
+after a joining peer completes the authenticated echo exchange:
```bash
-dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
- browse --service http://127.0.0.1:5000/ \
- --game space-game --environment development --region local --protocol 1
+dotnet run --project src/FinalFactory.Rendezvous.TestClient \
+ --configuration Release --no-build -- \
+ host --service http://127.0.0.1:8080/ --mediator 127.0.0.1:9050 \
+ --game space-game --environment smoke --region local --protocol 1 \
+ --display-name "Local diagnostic" --timeout-seconds 60 --run-seconds 60 \
+ --exit-after-echo
```
-Join from a third terminal. Omit `--listing` for an interactive choice:
+Copy the public listing ID printed by the host, or discover it from terminal 2:
```bash
-dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
- join --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
- --game space-game --environment development --region local --protocol 1 \
- --listing 00000000-0000-0000-0000-000000000000
+dotnet run --project src/FinalFactory.Rendezvous.TestClient \
+ --configuration Release --no-build -- \
+ browse --service http://127.0.0.1:8080/ \
+ --game space-game --environment smoke --region local --protocol 1
```
-Replace the sample UUID with the public listing ID printed by host or browse.
-Host and join each create one caller-owned LiteNetLib manager. That same socket
-sends presence/punch traffic, establishes the authenticated direct connection,
-and carries the ping/echo/ack/completion payload. The final completion confirms
-that the host received the reliable acknowledgement; none of this traffic passes through the HTTP
-service or UDP mediator.
+In terminal 3, either omit `--listing` and select interactively, or provide the
+copied ID for deterministic selection:
-## CI and deployment smoke flow
+```bash
+dotnet run --project src/FinalFactory.Rendezvous.TestClient \
+ --configuration Release --no-build -- \
+ join --service http://127.0.0.1:8080/ --mediator 127.0.0.1:9050 \
+ --game space-game --environment smoke --region local --protocol 1 \
+ --listing REPLACE_WITH_LISTING_UUID --timeout-seconds 30
+```
-Use `--script --json`, set `--listing` when deterministic selection matters, and
-check the documented process exit code. `--timeout-seconds` bounds each startup,
-traversal, or direct-traffic stage; a script host also uses it as its total runtime
-unless `--run-seconds` is explicit. A host can add `--exit-after-echo` so it
-terminates after the joining peer acknowledges direct traffic and receives the
-host's completion confirmation. Every wait is
-bounded by coordinator state and `--timeout-seconds`; no orchestration should use
-an unbounded sleep.
+Success means the joiner prints `join.connected` and verified direct traffic,
+and the host prints verified direct traffic before deregistering. The host and
+joiner each create one caller-owned LiteNetLib manager. The same UDP socket sends
+presence and punch traffic, accepts the authenticated peer, and carries the
+ping/echo/ack/completion payload; direct traffic does not pass through the HTTP
+service or mediator.
-The normal test suite contains a real process gate that starts the built Server,
-host TestClient, and join TestClient, waits for readiness and versioned events,
-and verifies direct traffic, cleanup, JSON shape, and secret canaries. Process
-trees are force-terminated in the test cleanup path if normal shutdown fails.
+Clean up secrets and the disposable service when finished:
-Useful success events are:
+```bash
+unset RENDEZVOUS_PUBLISHER_CREDENTIAL
+docker compose -f deploy/compose/compose.yaml down
+rm deploy/compose/secrets/signing-key
+```
-- `host.registered`, `host.ready`, `host.direct-traffic`, and `host.deregistered`;
-- `browse.completed` and `browse.session`; and
-- `join.connected`, `join.direct-traffic`, and `join.outcome-report`.
+## Script and JSON automation
-Failure events preserve stable typed phases and outcomes. When a terminal outcome
-contains a configured dedicated endpoint, `join.fallback` reports `available`
-with endpoint type `dedicated`; no raw address is printed and no fallback is
-started implicitly.
+`--script` forbids prompts and selects the first compatible listing unless
+`--listing UUID` fixes the choice. `--json` emits one JSON object per line with
+`version: 1`. New optional properties may be added, but event names and exit
+codes are stable automation contracts. Informational events use stdout and
+failures use stderr.
-## What the proof does and does not establish
+The deployment smoke performs the full health, publish, join, mediation, direct
+traffic, outcome-report, and cleanup flow using bounded waits:
-The deterministic loopback test proves the complete service/host/client protocol,
-ticket admission, and peer-to-peer payload path. Loopback is not evidence that all
-consumer routers, carrier-grade NATs, symmetric NATs, firewalls, VPNs, IPv6 paths,
-or platform policies permit hole punching. Same-LAN, separated observed endpoints,
-network namespaces/containers, mediator restart, and adverse topology coverage
-belong to the topology harness tracked by #14. Production rollout still requires
-tests from representative networks and a game-owned fallback policy.
+```bash
+dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
+./scripts/smoke-deployment.sh
+```
+
+For custom automation, capture JSON and preserve the process status separately:
+
+```bash
+set +e
+dotnet run --project src/FinalFactory.Rendezvous.TestClient \
+ --configuration Release --no-build -- \
+ browse --service http://127.0.0.1:8080/ \
+ --game space-game --environment smoke --region local --protocol 1 \
+ --script --json >browse.jsonl
+status=$?
+set -e
+jq -e 'select(.version == 1 and .event == "browse.completed")' browse.jsonl
+test "$status" -eq 0
+```
+
+Never use an unbounded sleep to orchestrate processes. Wait for versioned events
+such as `host.ready` and apply a deadline. Useful success events are
+`host.registered`, `host.ready`, `host.direct-traffic`, `host.deregistered`,
+`browse.completed`, `browse.session`, `join.connected`, `join.direct-traffic`,
+and `join.outcome-report`.
+
+| Exit | Meaning |
+| ---: | --- |
+| `0` | Requested diagnostic flow completed successfully |
+| `2` | Invalid command or options |
+| `3` | Missing or invalid local configuration |
+| `10` | HTTP, registration, browser, lease, or socket failure |
+| `11` | No compatible session was available or selected |
+| `12` | Authorization or traversal reached a typed terminal failure |
+| `13` | Direct connection succeeded but the direct traffic proof failed |
+| `130` | Caller cancellation or Ctrl+C |
+
+## Observe a safe failure
+
+Run this after the protocol-1 browse in terminal 2 and before the terminal-3
+join (or restart terminal 1 first). The preceding browse proves that one
+protocol-1 host is present. Now browse for deliberately incompatible protocol
+`999`. The command emits a successful directory response with
+`browse.completed`, `count: 0`, then exits `11` to distinguish compatibility
+from a service outage:
+
+```bash
+set +e
+dotnet run --project src/FinalFactory.Rendezvous.TestClient \
+ --configuration Release --no-build -- \
+ browse --service http://127.0.0.1:8080/ \
+ --game space-game --environment smoke --region local --protocol 999 \
+ --script --json >incompatible.jsonl
+status=$?
+set -e
+jq -e 'select(.event == "browse.completed" and .phase == "directory" and .count == 0)' \
+ incompatible.jsonl
+test "$status" -eq 11
+```
+
+This is a diagnostic failure drill, not a bypass: unknown tenant scope and
+protocols still fail closed, and the local helper cannot mint a credential for
+them.
+
+## Diagnose by phase, not by guesswork
+
+Start with the exit code, then the last versioned event and its `phase`, `status`,
+and typed `outcome`. Endpoint categories may be
+reported as `loopback`, `private`, or `public`; raw endpoints, credentials,
+capabilities, metadata, and player identities are never emitted.
+
+| Symptom or last event | Distinction | Check next |
+| --- | --- | --- |
+| `host.configuration`, exit `3` | Local credential variable is missing or malformed before any request | Confirm the named environment variable exists, tracing is off, and the credential has not expired |
+| `host.registration`, exit `10` | Publisher authentication, tenant policy, metadata, quota, or HTTP failure | Use the typed status; compare credential scope with game/environment/region and the provisioned policy, then correlate protected server telemetry by operation and time |
+| `browse.sessions`, exit `10` | Directory request failed | Check HTTP reachability, `/health/ready`, rate limiting, and contract compatibility |
+| `browse.completed` count `0`, or `join.selection` empty, exit `11` | Healthy directory but no compatible visible listing | Match game, environment, region, and exact gameplay protocol; then confirm a host lease is still active |
+| Exact `join.selection` failure, exit `10` | Listing disappeared, is hidden, or scope no longer matches | Browse again; do not retry an old listing ID forever |
+| `join.authorization`, exit `12` | Service rejected the attempt before NAT traversal | Inspect typed category/outcome for policy, capacity, stale host, or active-attempt limits |
+| `join.punch` / `join.traversal`, exit `12` | Mediation or NAT traversal did not establish a peer | Confirm UDP endpoint/reply path, host presence, clocks, firewall/NAT behavior, and topology; use a game-owned fallback if policy supplies one |
+| `join.direct-connect`, exit `12` | Introduction occurred but authenticated direct admission failed | Confirm host is polling the same socket, the one-time ticket is current, and game admission did not reject capacity, identity, or bans |
+| `join.connected` followed by exit `13` | Peer connected but the direct gameplay-like echo did not finish | Inspect the peer lifecycle and caller polling; this is not an HTTP/directory failure |
+
+Stopping a host without deregistration may leave its listing visible only until
+the bounded lease expires. During that window, a join can produce a typed stale
+host or traversal outcome; it must not be interpreted as a healthy host. Restarting
+the single-active service intentionally loses all ephemeral listings and attempts,
+so hosts re-register and clients browse again.
+
+If a terminal outcome reports an authoritative dedicated fallback,
+`join.fallback` exposes only availability and endpoint type. TestClient never
+connects to it automatically. The game owns the decision, authentication, and
+connection policy. If no fallback is present, Rendezvous v1 offers no relay.
+
+## Production use
+
+Do not copy a production signing key to a diagnostic host. Supply a short-lived,
+least-scope publisher credential from the deployment secret boundary and set the
+external service, mediator, and matching scope variables described in the
+[secure Linux deployment smoke](../deployment/linux.md#http-and-udp-smoke).
+Run representative external-network tests; loopback success is not NAT coverage.
diff --git a/docs/operations/incident-runbooks.md b/docs/operations/incident-runbooks.md
new file mode 100644
index 0000000..b6a33c2
--- /dev/null
+++ b/docs/operations/incident-runbooks.md
@@ -0,0 +1,339 @@
+# Incident and change runbooks
+
+Tracking: #20
+
+These runbooks supplement the [signal and operator reference](observability-and-operator-runbook.md).
+Every procedure has four explicit gates: detect, contain, recover, and verify.
+Record timestamps, the release digest, bounded aggregates, audit fingerprints,
+and `X-Rendezvous-Correlation-ID` values. Never copy credentials, capabilities,
+connection tickets, signing material, player identity, raw IP addresses,
+endpoints, listing metadata, or full request bodies into an incident record.
+
+Operator routes must be reachable only from an allowed management source. Use a
+short-lived, least-permission operator credential minted outside Rendezvous.
+Pass it to an approved operator client through protected stdin or a secret agent,
+not a URL, command argument, environment-wide process launcher, shell trace, or
+ticket. All request shapes and responses are defined by the generated
+[OpenAPI v1 document](../api/rendezvous-v1.json).
+
+Before an incident, keep these protected records available without depending on
+the affected service: current and previous image digests, matching configuration,
+key IDs and lifecycle windows (not raw key values), the game owner/on-call map,
+capacity baselines, collector destinations, and a separately authorized
+break-glass operator key. Test management-source allowlisting and credential
+permissions at least once per release.
+
+Use the exact versioned action shapes below. Confirmation fields deliberately
+repeat the target so a stale UI selection or copy error fails closed. Responses
+do not echo targets.
+
+| Operation | JSON body |
+| --- | --- |
+| `POST /v1/operator/listings/revoke` | `{"listingId":"","confirmListingId":""}` |
+| `POST /v1/operator/principals/revoke` | `{"subject":"","confirmSubject":"","lifetimeSeconds":60}` |
+| `POST /v1/operator/keys/revoke` | `{"keyId":"","confirmKeyId":""}` |
+| `POST /v1/operator/drain` | `{"confirmation":"DRAIN"}` |
+
+## Abuse or authentication spike
+
+### Detect
+
+- Alert on a baseline-relative increase in `rendezvous.limiter.drops`, HTTP/UDP
+ request rate, `rendezvous.operator.authentication` rejected/forbidden results,
+ registration requests by authentication status, queue depth, or p95/p99 latency.
+- Check `/health/live`, `/health/ready`, `rendezvous.store.available`, and
+ authenticated `GET /v1/operator/status`. Separate public-source rejection,
+ publisher credential failure, operator probing, and ordinary capacity growth.
+- Use only bounded operation/result dimensions and correlation IDs. Do not group
+ by raw address, token, subject, listing ID, or metadata.
+
+### Contain
+
+- Preserve the dedicated operator partition. Do not raise public limits during
+ an active spike. Apply source-preserving edge rate controls only when their
+ collateral effect is understood and UDP source address/port remains intact.
+- For one abusive session, call `POST /v1/operator/listings/revoke` with identical
+ `listingId` and `confirmListingId`. For a confirmed publisher subject, call
+ `POST /v1/operator/principals/revoke` with identical `subject` and
+ `confirmSubject` and a 1–600 second lifetime.
+- Revoke a signing key only when compromise evidence implicates that issuer;
+ broad key revocation invalidates every credential signed by it. Drain only if
+ the process itself must be isolated.
+
+### Recover
+
+- Correct the source integration, edge rule, leaked principal grant, or tenant
+ budget under change control. Let a bounded principal revocation expire only
+ after the owner confirms remediation; a repeated shorter revocation never
+ shortens the original deadline.
+- Restore normal limits gradually. If saturation caused state churn, allow leases
+ and attempts to expire naturally rather than deleting arbitrary state.
+
+### Verify
+
+- Require limiter drops, authentication result ratios, queue depth, latency, and
+ direct-connect outcomes to return to the same-region baseline for the agreed
+ observation window.
+- Confirm readiness stayed healthy or recovered, operator audit contains the
+ intended action/result fingerprint, revoked resources cannot create new work,
+ and unaffected tenants can still publish, browse, and connect.
+
+## Signing key or issuer compromise
+
+### Detect
+
+- Treat secret-manager access alerts, unexpected issuance, credentials outside
+ the expected region/kind, a signing-key expiry alarm, or unexplained publisher
+ authentication growth as compromise until disproved.
+- Identify the non-secret key ID, allowed credential kinds, game/environment
+ binding, `NotBefore`, `SignUntil`, and `VerifyUntil`. Do not retrieve or paste
+ raw material merely to compare it.
+
+### Contain
+
+- Stop the affected external issuer and deny further access to its secret.
+- From a separate uncompromised break-glass operator key with `RotateKeys`, call
+ `POST /v1/operator/keys/revoke` with identical `keyId` and `confirmKeyId`.
+ Runtime revocation is immediate but process-local.
+- Remove or mark the key revoked in authoritative provisioning before any
+ restart. Revoke affected principals/listings when narrower evidence supports
+ it. Do not drain automatically unless the running instance cannot be trusted.
+
+### Recover
+
+- Generate replacement material in the approved secret boundary, use a new key
+ ID, bind it to the exact credential kind and tenant, and deploy configuration
+ referencing the secret—never the secret value.
+- Resume issuance with short lifetimes. Reissue only to authenticated workloads.
+ When confidentiality is lost, do not use normal overlap to keep compromised
+ credentials valid; document the intentional invalidation window.
+- Rotate any release, registry, or operator credential exposed by the same
+ incident through its owning system; Rendezvous key revocation cannot revoke
+ unrelated systems.
+
+### Verify
+
+- Confirm `GET /v1/operator/status` shows the compromised key revoked and the
+ replacement signing, old credentials fail, new exact-scope credentials work,
+ and the result survives a controlled restart from updated provisioning.
+- Pass TestClient registration, browse, authenticated mediation, and direct
+ traffic with the replacement; monitor authentication and audit results through
+ at least the maximum newly issued credential lifetime.
+
+## Targeted listing or publisher revocation
+
+### Detect
+
+- Validate the abuse report against game-owned records and bounded Rendezvous
+ evidence. Determine whether the target is one listing or an authenticated
+ publisher subject. Do not use display name, metadata, or a raw address as
+ identity.
+- Confirm current aggregate state through `GET /v1/operator/status` and record
+ the correlation IDs that justified action.
+
+### Contain
+
+- Revoke one listing with `POST /v1/operator/listings/revoke`; the exact listing
+ UUID must appear in both confirmation fields.
+- Revoke a publisher with `POST /v1/operator/principals/revoke`; the exact subject
+ must appear in both confirmation fields and `lifetimeSeconds` must be 1–600.
+ This removes that principal's active listings and attempts and blocks new ones
+ for the bounded lifetime.
+- Choose the narrowest action. Do not revoke a tenant key for a single listing.
+
+### Recover
+
+- The game owner resolves the ban, account, workload, or configuration issue in
+ the authoritative game system. Rendezvous does not own user accounts or bans.
+- After the original revocation deadline, permit a newly authenticated publisher
+ to register. There is no un-revoke endpoint and no recovery of removed
+ ephemeral listings; the host creates a new listing.
+
+### Verify
+
+- Confirm the old listing is no longer browsable or joinable, the principal
+ cannot publish during its lifetime, and the audit action/result is present
+ without the raw target.
+- Confirm unrelated publishers in the same tenant and another tenant still pass
+ publish/browse/join/direct-traffic checks.
+
+## Planned restart or crash recovery
+
+### Detect
+
+- Planned restart begins with a recorded change and a healthy current baseline.
+ Crash recovery begins when liveness/process state fails or both TCP 8080 and
+ UDP 9050 stop answering. Distinguish dependency/readiness failure from a dead
+ process; liveness deliberately remains healthy for some recoverable failures.
+- Record active listing/lease/attempt aggregates. They are informational only:
+ v1 has no durable runtime database to restore.
+
+### Contain
+
+- For a planned stop, call `POST /v1/operator/drain` with confirmation exactly
+ `DRAIN`. Require readiness `503`, liveness `200`, and removal from new traffic.
+ Allow the bounded drain deadline to finish, then send SIGTERM.
+- Never start a second active instance while the old process owns the advertised
+ HTTP/UDP endpoints. On crash, fence the old process/host and verify both sockets
+ are released before replacement.
+
+### Recover
+
+- Start exactly one instance from the recorded immutable image digest and matching
+ reviewed configuration/key references. A restart intentionally loses listings,
+ observed endpoints, attempts, replay markers, and runtime-only revocations.
+- Ensure any emergency key revocation is also present in authoritative
+ provisioning. Hosts must re-register; clients must browse and start new
+ attempts. Do not restore stale ephemeral state from logs or backups.
+
+### Verify
+
+- Require live and ready health, UDP bind, store availability, and one active
+ target. Run the full deployment smoke and confirm host re-registration begins.
+- Verify no pre-restart listing or capability is accepted, runtime revocations
+ that should persist are configuration-backed, and latency/outcomes stabilize.
+
+## Release rollback
+
+### Detect
+
+- Trigger rollback from a predeclared objective: readiness loss, failed deployment
+ smoke, contract/package incompatibility, security regression, direct-success
+ regression beyond threshold, or sustained resource regression. Record the new
+ and previous digests and the evidence; do not move a tag.
+
+### Contain
+
+- Stop promotion and new rollout work. Drain and stop the faulty single active
+ instance, then verify both public sockets are released. Revoke affected keys or
+ principals only when the defect creates an authorization risk.
+- Preserve logs, artifacts, provenance, signatures, and the faulty release record.
+ Never overwrite or delete an immutable package/image to reuse its version.
+
+### Recover
+
+- Deploy the previous known-good image by digest with its compatible configuration
+ and key set. Do not run old and new concurrently. If configuration changed,
+ apply its reviewed down-migration before starting.
+- Publish a corrected build under a new SemVer after diagnosis; mark faulty release
+ notes withdrawn when appropriate.
+
+### Verify
+
+- Check the running image digest, live/ready health, one active target, UDP source
+ preservation, and the complete TestClient deployment smoke.
+- Confirm package/server compatibility from `GET /v1/operator/status`, hosts
+ re-register, and the rollback objective returns to baseline for the observation
+ window.
+
+## Capacity saturation
+
+### Detect
+
+- Page when `rendezvous.queue.depth` remains above 90% of the configured attempt
+ limit, lease-critical work is shed, `rendezvous.store.available` is zero, or no
+ ready instance remains. Warn at 70%, sustained `rendezvous.limiter.drops`, or
+ p95 latency above objective.
+- Compare CPU, memory, file descriptors, UDP errors, expiry churn, HTTP operation
+ rate, and typed connection outcomes with the measured
+ [capacity profile](capacity-and-resilience.md). Distinguish legitimate growth,
+ attack traffic, downstream telemetry pressure, and a regression.
+
+### Contain
+
+- Preserve lease-critical and operator reserves. Shed new browse/join work with
+ the existing typed `429`/`Retry-After` behavior; do not add an unbounded queue.
+- Apply per-tenant/source controls at the appropriate trusted boundary. If the
+ process is unstable, drain new work and recover on one replacement rather than
+ adding a second active replica; v1 state is process-local.
+
+### Recover
+
+- Remove the causal load or deploy a tested higher single-instance resource and
+ budget profile. Change CPU/memory and server limits together, using the numeric
+ gate and accelerated soak before production.
+- Long-term horizontal scaling requires a designed shared directory, replay, and
+ attempt authority. A generic load balancer is not that design.
+
+### Verify
+
+- Re-run the capacity/resilience gate at the chosen profile, then require queue,
+ limiter drops, expiry churn, latency, store health, and direct-success ratio to
+ remain within objectives through the production observation window.
+- Confirm termination still completes within `DrainDeadlineSeconds + 5` and the
+ public and operator partitions behave independently.
+
+## Privacy or telemetry incident
+
+### Detect
+
+- Trigger on any credential, token, capability, player identity, raw IP/endpoint,
+ listing ID, metadata, or caller-reported exact connection duration tied to an
+ event or identity found in logs, metrics, traces, crash reports, support systems,
+ or analytics. Aggregate HTTP/UDP duration histograms with bounded operation tags
+ are expected telemetry. Also trigger when audit data exceeds its approved 30-day
+ retention without an incident hold.
+- Identify the producing version, sink, access population, retention/replication
+ path, and time window without copying the exposed value into a new system.
+
+### Contain
+
+- Stop or filter the offending export and restrict access to affected sinks.
+ Preserve the minimum evidence under the incident process; do not take broad
+ diagnostic dumps that amplify exposure.
+- Revoke exposed reusable credentials/keys through their owning boundary. Listing
+ IDs and endpoints are not authentication secrets, but remove affected listings
+ if continued exposure creates risk. Notify privacy/security owners according to
+ applicable policy and law.
+
+### Recover
+
+- Patch the producer to the allowlisted telemetry model, test canary redaction
+ across logs/metrics/traces/output, and deploy through the immutable release
+ path. Delete or age out affected data from every sink according to approved
+ retention and legal-hold direction.
+- Replace exposed credentials and re-register hosts when necessary. Do not claim
+ that a service restart deletes copies already exported to collectors.
+
+### Verify
+
+- Search new telemetry using non-secret synthetic canaries and confirm no canary
+ or prohibited field crosses the boundary. Verify audit records contain only
+ fixed fields and fingerprints and that retention/eviction is operating.
+- Security/privacy owners confirm sink cleanup, access review, notification, and
+ monitoring closure before the incident is resolved.
+
+## Dependency or base-image upgrade
+
+### Detect
+
+- Open a reviewed change for an advisory, end-of-support date, pinned-digest
+ refresh, or planned package update. Record affected package/image, current and
+ proposed exact version/digest, advisory severity, exploitability, and required
+ deadline. Never float to `latest` as remediation.
+
+### Contain
+
+- For an actively exploited critical issue, restrict exposure or stop the service
+ under incident authority while building the fix. Revoking publisher keys does
+ not repair a vulnerable runtime. Otherwise keep the known-good release running
+ while the candidate is tested.
+
+### Recover
+
+- Update the SDK/base-image digest, lock files, license/advisory evidence, SBOM,
+ compatibility matrix, and release notes together. For LiteNetLib or a wire/API
+ change, apply the explicit version/migration policy rather than silently
+ replacing compatible bytes.
+- Run locked restore, formatting, Debug and Release builds/tests, public contract
+ and package gates, real consumer restores, reproducible artifact/image builds,
+ vulnerability scan, signatures, topology/deployment smoke, and capacity checks
+ proportional to the change. Promote the exact tested digest.
+
+### Verify
+
+- Verify signatures, provenance, checksums, SBOM contents, running digest, and
+ absence of the advisory in the shipped artifact—not merely the build host.
+- Require live/ready health, TestClient direct traffic, real consumer compatibility,
+ and normal latency/outcomes. Keep the previous digest and compatible config for
+ rollback until the observation window closes.
diff --git a/docs/operations/observability-and-operator-runbook.md b/docs/operations/observability-and-operator-runbook.md
index c77ac14..7a657d8 100644
--- a/docs/operations/observability-and-operator-runbook.md
+++ b/docs/operations/observability-and-operator-runbook.md
@@ -1,4 +1,4 @@
-# Observability and operator runbook
+# Observability and operator reference
This runbook defines the production signals and privileged controls for the
Rendezvous service. The service emits `System.Diagnostics.Metrics` instruments
@@ -6,6 +6,10 @@ from the `FinalFactory.Rendezvous` meter and distributed-tracing activities from
`FinalFactory.Rendezvous.Server`. Connect those sources to the deployment's
OpenTelemetry or equivalent collector. Do not add identifiers to metric labels.
+Concrete detect/contain/recover/verify procedures for abuse, key compromise,
+targeted revocation, restart, rollback, saturation, privacy incidents, and
+dependency upgrades are in the [incident and change runbooks](incident-runbooks.md).
+
## Health and readiness
- `GET /health/live` proves that the HTTP process can answer. It deliberately
diff --git a/scripts/mint-local-publisher-credential.sh b/scripts/mint-local-publisher-credential.sh
new file mode 100755
index 0000000..330d385
--- /dev/null
+++ b/scripts/mint-local-publisher-credential.sh
@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+LOCAL_KEY="${RENDEZVOUS_SMOKE_LOCAL_KEY:-$ROOT/deploy/compose/secrets/signing-key}"
+
+if (( $# != 0 )); then
+ printf 'This helper accepts no arguments and mints only the fixed local Compose smoke scope.\n' >&2
+ exit 2
+fi
+
+command -v python3 >/dev/null || {
+ printf 'Missing required command: python3\n' >&2
+ exit 2
+}
+
+# This is deliberately a local-fixture tool, not a general credential issuer.
+# Python reads the raw key from the protected file; key material never appears in
+# a child process argument, environment value, temporary file, or command output.
+python3 - "$LOCAL_KEY" <<'PY'
+import base64
+import hashlib
+import hmac
+import json
+import os
+import secrets
+import stat
+import sys
+import time
+
+key_path = sys.argv[1]
+try:
+ metadata = os.lstat(key_path)
+except FileNotFoundError:
+ raise SystemExit(f"Local Compose smoke key does not exist: {key_path}")
+
+if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode):
+ raise SystemExit(f"Local Compose smoke key must be a regular non-symlink file: {key_path}")
+parent_path = os.path.dirname(os.path.abspath(key_path))
+parent = os.lstat(parent_path)
+if stat.S_ISLNK(parent.st_mode) or not stat.S_ISDIR(parent.st_mode):
+ raise SystemExit(f"Local Compose secret directory must be a non-symlink directory: {parent_path}")
+if parent.st_uid != os.geteuid() or parent.st_mode & 0o077:
+ raise SystemExit(f"Local Compose secret directory must be owned by this user with mode 0700: {parent_path}")
+if metadata.st_uid != os.geteuid() or metadata.st_mode & 0o022 or metadata.st_nlink != 1:
+ raise SystemExit(f"Local Compose smoke key must be owned by this user, single-linked, and not group/world writable: {key_path}")
+
+with open(key_path, "rb") as key_file:
+ key = key_file.read(33)
+if len(key) != 32:
+ raise SystemExit(f"Local Compose smoke key must be exactly 32 bytes: {key_path}")
+
+now = int(time.time())
+payload = {
+ "version": 1,
+ "issuer": "final-factory-rendezvous-smoke",
+ "audience": "rendezvous-service",
+ "subject": "local-smoke-host",
+ "kind": "dedicatedPublisher",
+ "gameId": "space-game",
+ "environmentId": "smoke",
+ "regions": ["local"],
+ "permissions": [],
+ "issuedAtUnixSeconds": now,
+ "notBeforeUnixSeconds": now,
+ "expiresAtUnixSeconds": now + 600,
+ "nonce": secrets.token_hex(16),
+}
+
+def base64url(value: bytes) -> str:
+ return base64.urlsafe_b64encode(value).rstrip(b"=").decode("ascii")
+
+encoded = base64url(json.dumps(payload, separators=(",", ":")).encode("utf-8"))
+signed = f"rv1.local-smoke-1.{encoded}"
+signature = base64url(hmac.new(key, signed.encode("ascii"), hashlib.sha256).digest())
+print(f"{signed}.{signature}")
+PY
diff --git a/scripts/smoke-deployment.sh b/scripts/smoke-deployment.sh
index d0d17e6..d931339 100755
--- a/scripts/smoke-deployment.sh
+++ b/scripts/smoke-deployment.sh
@@ -13,7 +13,7 @@ ENVIRONMENT_ID="${RENDEZVOUS_SMOKE_ENVIRONMENT_ID:-smoke}"
REGION="${RENDEZVOUS_SMOKE_REGION:-local}"
PROTOCOL_VERSION="${RENDEZVOUS_SMOKE_PROTOCOL_VERSION:-1}"
-for command in curl date dotnet jq mktemp od openssl tail tr wc; do
+for command in curl dotnet jq mktemp tail; do
command -v "$command" >/dev/null || {
printf 'Missing required command: %s\n' "$command" >&2
exit 2
@@ -35,39 +35,14 @@ for scoped_value in "$GAME_ID" "$ENVIRONMENT_ID" "$REGION"; do
fi
done
-base64url() {
- openssl base64 -A | tr '+/' '-_' | tr -d '='
-}
-
local_credential() {
- if [[ ! -f "$LOCAL_KEY" ]] || [[ "$(wc -c < "$LOCAL_KEY")" -ne 32 ]]; then
- printf 'Local Compose smoke key must be exactly 32 bytes: %s\n' "$LOCAL_KEY" >&2
+ if [[ "$GAME_ID" != space-game || "$ENVIRONMENT_ID" != smoke \
+ || "$REGION" != local || "$PROTOCOL_VERSION" != 1 ]]; then
+ printf 'The local credential helper supports only space-game/smoke/local protocol 1. Supply RENDEZVOUS_PUBLISHER_CREDENTIAL for any other scope.\n' >&2
exit 2
fi
-
- local now expires nonce payload encoded signed hex signature
- now="$(date +%s)"
- expires="$((now + 600))"
- nonce="$(openssl rand -hex 16)"
- payload="$(jq -cn \
- --arg issuer final-factory-rendezvous-smoke \
- --arg audience rendezvous-service \
- --arg subject local-smoke-host \
- --arg kind dedicatedPublisher \
- --arg gameId "$GAME_ID" \
- --arg environmentId "$ENVIRONMENT_ID" \
- --arg region "$REGION" \
- --arg nonce "$nonce" \
- --argjson now "$now" \
- --argjson expires "$expires" \
- '{version:1,issuer:$issuer,audience:$audience,subject:$subject,kind:$kind,gameId:$gameId,environmentId:$environmentId,regions:[$region],permissions:[],issuedAtUnixSeconds:$now,notBeforeUnixSeconds:$now,expiresAtUnixSeconds:$expires,nonce:$nonce}')"
- encoded="$(printf '%s' "$payload" | base64url)"
- signed="rv1.local-smoke-1.$encoded"
- hex="$(od -An -v -tx1 "$LOCAL_KEY" | tr -d ' \n')"
- signature="$(printf '%s' "$signed" \
- | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$hex" -binary \
- | base64url)"
- printf '%s.%s' "$signed" "$signature"
+ RENDEZVOUS_SMOKE_LOCAL_KEY="$LOCAL_KEY" \
+ "$ROOT/scripts/mint-local-publisher-credential.sh"
}
credential="${RENDEZVOUS_PUBLISHER_CREDENTIAL:-}"
diff --git a/src/FinalFactory.Rendezvous.TestClient/README.md b/src/FinalFactory.Rendezvous.TestClient/README.md
index 3826b64..16f4f11 100644
--- a/src/FinalFactory.Rendezvous.TestClient/README.md
+++ b/src/FinalFactory.Rendezvous.TestClient/README.md
@@ -13,13 +13,16 @@ authenticated direct peer, and answers a bounded ping/echo/ack/completion exchan
LiteNetLib socket, proves direct traffic, reports the typed outcome, and exits.
Run `dotnet run --project src/FinalFactory.Rendezvous.TestClient -- --help` for
-the complete option reference. A typical script-mode invocation is:
+the complete option reference. The repository's
+[start-to-finish guide](../../docs/integration/test-client.md) provides an
+executable local Compose setup, safe failure drill, JSON automation, and a
+phase-by-phase diagnostic table. A typical deployment invocation is:
```bash
export RENDEZVOUS_PUBLISHER_CREDENTIAL=''
dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
- host --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
- --game space-game --environment development --region local --protocol 1 \
+ host --service https://rendezvous.example/ --mediator rendezvous.example:9050 \
+ --game space-game --environment production --region eu-central --protocol 1 \
--script --json --exit-after-echo
```
diff --git a/tests/FinalFactory.Rendezvous.Tests/Deployment/ProductionProcessTests.cs b/tests/FinalFactory.Rendezvous.Tests/Deployment/ProductionProcessTests.cs
index 82191b5..9410a34 100644
--- a/tests/FinalFactory.Rendezvous.Tests/Deployment/ProductionProcessTests.cs
+++ b/tests/FinalFactory.Rendezvous.Tests/Deployment/ProductionProcessTests.cs
@@ -207,10 +207,16 @@ public sealed class ProductionProcessTests
string root = RepositoryRoot();
int httpPort = ReserveTcpPort();
int udpPort = ReserveUdpPort();
- string secretPath = Path.Combine(
+ string secretDirectory = Path.Combine(
Path.GetTempPath(),
$"rendezvous-smoke-secret-{Guid.NewGuid():N}");
+ Directory.CreateDirectory(secretDirectory);
+ File.SetUnixFileMode(
+ secretDirectory,
+ UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute);
+ string secretPath = Path.Combine(secretDirectory, "signing-key");
await File.WriteAllBytesAsync(secretPath, RandomNumberGenerator.GetBytes(32));
+ File.SetUnixFileMode(secretPath, UnixFileMode.UserRead | UnixFileMode.UserWrite);
Process? server = null;
Process? smoke = null;
try
@@ -294,6 +300,7 @@ public sealed class ProductionProcessTests
}
File.Delete(secretPath);
+ Directory.Delete(secretDirectory);
}
}
diff --git a/tests/FinalFactory.Rendezvous.Tests/Documentation/DocumentationContractTests.cs b/tests/FinalFactory.Rendezvous.Tests/Documentation/DocumentationContractTests.cs
new file mode 100644
index 0000000..5cafdb9
--- /dev/null
+++ b/tests/FinalFactory.Rendezvous.Tests/Documentation/DocumentationContractTests.cs
@@ -0,0 +1,213 @@
+using System.Text.Json;
+using System.Text.RegularExpressions;
+using System.Xml.Linq;
+
+namespace FinalFactory.Rendezvous.Tests.Documentation;
+
+public sealed partial class DocumentationContractTests
+{
+ private static readonly string[] IncidentScenarios =
+ [
+ "Abuse or authentication spike",
+ "Signing key or issuer compromise",
+ "Targeted listing or publisher revocation",
+ "Planned restart or crash recovery",
+ "Release rollback",
+ "Capacity saturation",
+ "Privacy or telemetry incident",
+ "Dependency or base-image upgrade",
+ ];
+
+ [Fact]
+ public void TestClientGuideDocumentsTheExecutableSuccessAndFailureContracts()
+ {
+ string root = FindRepositoryRoot();
+ string guide = File.ReadAllText(Path.Combine(root, "docs", "integration", "test-client.md"));
+
+ Assert.Contains("space-game --environment smoke --region local --protocol 1", guide, StringComparison.Ordinal);
+ Assert.Contains("mint-local-publisher-credential.sh", guide, StringComparison.Ordinal);
+ Assert.Contains("join.connected", guide, StringComparison.Ordinal);
+ Assert.Contains("join.direct-traffic", guide, StringComparison.Ordinal);
+ Assert.Contains("browse.completed", guide, StringComparison.Ordinal);
+ Assert.Contains("--script --json", guide, StringComparison.Ordinal);
+ Assert.Contains("--run-seconds 60", guide, StringComparison.Ordinal);
+ Assert.Contains("for attempt in {1..45}", guide, StringComparison.Ordinal);
+ Assert.Contains("before the terminal-3", guide, StringComparison.Ordinal);
+ Assert.Contains("test \"$status\" -eq 11", guide, StringComparison.Ordinal);
+ Assert.Contains("no relay", guide, StringComparison.OrdinalIgnoreCase);
+ Assert.Contains("cannot guarantee", guide, StringComparison.OrdinalIgnoreCase);
+ Assert.DoesNotMatch(ReusableCredential(), guide);
+ }
+
+ [Fact]
+ public void LocalCredentialHelperIsFixedScopeAndSmokeDelegatesToIt()
+ {
+ string root = FindRepositoryRoot();
+ string helper = File.ReadAllText(Path.Combine(root, "scripts", "mint-local-publisher-credential.sh"));
+ string smoke = File.ReadAllText(Path.Combine(root, "scripts", "smoke-deployment.sh"));
+
+ Assert.Contains("if (( $# != 0 ));", helper, StringComparison.Ordinal);
+ Assert.Contains("\"gameId\": \"space-game\"", helper, StringComparison.Ordinal);
+ Assert.Contains("\"environmentId\": \"smoke\"", helper, StringComparison.Ordinal);
+ Assert.Contains("\"regions\": [\"local\"]", helper, StringComparison.Ordinal);
+ Assert.Contains("now + 600", helper, StringComparison.Ordinal);
+ Assert.Contains("stat.S_ISLNK", helper, StringComparison.Ordinal);
+ Assert.Contains("parent.st_mode & 0o077", helper, StringComparison.Ordinal);
+ Assert.Contains("metadata.st_mode & 0o022", helper, StringComparison.Ordinal);
+ Assert.Contains("metadata.st_nlink != 1", helper, StringComparison.Ordinal);
+ Assert.Contains("mint-local-publisher-credential.sh", smoke, StringComparison.Ordinal);
+ Assert.DoesNotContain("hexkey:", smoke, StringComparison.Ordinal);
+ Assert.DoesNotContain("openssl dgst", smoke, StringComparison.Ordinal);
+ }
+
+ [Fact]
+ public void EveryIncidentRunbookHasDetectContainRecoverAndVerifyGates()
+ {
+ string root = FindRepositoryRoot();
+ string runbooks = File.ReadAllText(Path.Combine(root, "docs", "operations", "incident-runbooks.md"));
+
+ for (int index = 0; index < IncidentScenarios.Length; index++)
+ {
+ string heading = $"## {IncidentScenarios[index]}";
+ int start = runbooks.IndexOf(heading, StringComparison.Ordinal);
+ Assert.True(start >= 0, $"Missing incident runbook heading: {heading}");
+ int end = index + 1 < IncidentScenarios.Length
+ ? runbooks.IndexOf($"## {IncidentScenarios[index + 1]}", start, StringComparison.Ordinal)
+ : runbooks.Length;
+ Assert.True(end > start, $"Could not find the end of runbook: {heading}");
+ string scenario = runbooks[start..end];
+
+ Assert.Contains("### Detect", scenario, StringComparison.Ordinal);
+ Assert.Contains("### Contain", scenario, StringComparison.Ordinal);
+ Assert.Contains("### Recover", scenario, StringComparison.Ordinal);
+ Assert.Contains("### Verify", scenario, StringComparison.Ordinal);
+ }
+
+ Assert.DoesNotMatch(ReusableCredential(), runbooks);
+ }
+
+ [Fact]
+ public void DocumentedOperatorOperationsAndBodiesMatchTheReleasedOpenApi()
+ {
+ string root = FindRepositoryRoot();
+ string runbooks = File.ReadAllText(Path.Combine(root, "docs", "operations", "incident-runbooks.md"));
+ using JsonDocument openApi = JsonDocument.Parse(File.ReadAllText(
+ Path.Combine(root, "docs", "api", "rendezvous-v1.json")));
+ JsonElement paths = openApi.RootElement.GetProperty("paths");
+
+ (string Method, string Path)[] documentedOperations = OperatorRoute().Matches(runbooks)
+ .Cast()
+ .Select(static match => (
+ match.Groups["method"].Value.ToLowerInvariant(),
+ match.Groups["path"].Value))
+ .Distinct()
+ .ToArray();
+
+ Assert.NotEmpty(documentedOperations);
+ Assert.All(documentedOperations, operation =>
+ Assert.True(
+ paths.TryGetProperty(operation.Path, out JsonElement path)
+ && path.TryGetProperty(operation.Method, out _),
+ $"OpenAPI does not contain {operation.Method.ToUpperInvariant()} {operation.Path}."));
+
+ Match[] actions = OperatorAction().Matches(runbooks).Cast().ToArray();
+ Assert.Equal(4, actions.Length);
+ foreach (Match action in actions)
+ {
+ string method = action.Groups["method"].Value.ToLowerInvariant();
+ string path = action.Groups["path"].Value;
+ using JsonDocument body = JsonDocument.Parse(action.Groups["body"].Value);
+ string reference = paths.GetProperty(path)
+ .GetProperty(method)
+ .GetProperty("requestBody")
+ .GetProperty("content")
+ .GetProperty("application/json")
+ .GetProperty("schema")
+ .GetProperty("$ref")
+ .GetString()!;
+ string schemaName = reference["#/components/schemas/".Length..];
+ string[] required = openApi.RootElement.GetProperty("components")
+ .GetProperty("schemas")
+ .GetProperty(schemaName)
+ .GetProperty("required")
+ .EnumerateArray()
+ .Select(static property => property.GetString()!)
+ .Order(StringComparer.Ordinal)
+ .ToArray();
+ string[] documented = body.RootElement.EnumerateObject()
+ .Select(static property => property.Name)
+ .Order(StringComparer.Ordinal)
+ .ToArray();
+
+ Assert.Equal(required, documented);
+ }
+ }
+
+ [Fact]
+ public void SdkGuideMatchesTheReleasedPackageAndTransportMatrix()
+ {
+ string root = FindRepositoryRoot();
+ string guide = File.ReadAllText(Path.Combine(root, "docs", "integration", "sdk-seams.md"));
+ using JsonDocument compatibility = JsonDocument.Parse(File.ReadAllText(
+ Path.Combine(root, "docs", "releases", "compatibility.json")));
+ JsonElement matrix = compatibility.RootElement;
+ JsonElement packages = matrix.GetProperty("packages");
+ string clientVersion = packages.GetProperty("FinalFactory.Rendezvous.Client").GetString()!;
+ string contractsVersion = packages.GetProperty("FinalFactory.Rendezvous.Contracts").GetString()!;
+ string liteNetLibVersion = matrix.GetProperty("transport").GetProperty("version").GetString()!;
+ string clientFramework = XDocument.Load(Path.Combine(
+ root,
+ "src",
+ "FinalFactory.Rendezvous.Client",
+ "FinalFactory.Rendezvous.Client.csproj"))
+ .Descendants("TargetFramework")
+ .Single()
+ .Value;
+ int httpVersion = matrix.GetProperty("contracts").GetProperty("http")[0].GetInt32();
+ int udpVersion = matrix.GetProperty("contracts").GetProperty("udp")[0].GetInt32();
+ int ticketVersion = matrix.GetProperty("contracts").GetProperty("connectionTicket")[0].GetInt32();
+
+ Assert.Contains(
+ $"FinalFactory.Rendezvous.Client\" Version=\"{clientVersion}\"",
+ guide,
+ StringComparison.Ordinal);
+ Assert.Contains(
+ $"FinalFactory.Rendezvous.Contracts\" Version=\"{contractsVersion}\"",
+ guide,
+ StringComparison.Ordinal);
+ Assert.Contains($"targets `{clientFramework}`", guide, StringComparison.Ordinal);
+ Assert.Contains($"LiteNetLib `{liteNetLibVersion}`", guide, StringComparison.Ordinal);
+ Assert.Contains(
+ "https://git.finalfactory.de/api/packages/HeiKyu/nuget/index.json",
+ guide,
+ StringComparison.Ordinal);
+ Assert.Equal(httpVersion, udpVersion);
+ Assert.Equal(httpVersion, ticketVersion);
+ Assert.Contains($"contract version `{httpVersion}`", guide, StringComparison.Ordinal);
+ }
+
+ [GeneratedRegex(@"rv1\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+", RegexOptions.CultureInvariant)]
+ private static partial Regex ReusableCredential();
+
+ [GeneratedRegex(@"(?GET|POST) `?(?/v1/operator/[a-z/-]+)`?", RegexOptions.CultureInvariant)]
+ private static partial Regex OperatorRoute();
+
+ [GeneratedRegex(@"\| `(?POST) (?/v1/operator/[a-z/-]+)` \| `(?\{[^`]+\})` \|", RegexOptions.CultureInvariant)]
+ private static partial Regex OperatorAction();
+
+ private static string FindRepositoryRoot()
+ {
+ DirectoryInfo? current = new(AppContext.BaseDirectory);
+ while (current is not null)
+ {
+ if (File.Exists(Path.Combine(current.FullName, "Rendezvous.slnx")))
+ {
+ return current.FullName;
+ }
+
+ current = current.Parent;
+ }
+
+ throw new InvalidOperationException("Could not locate the repository root.");
+ }
+}