Rendezvous/docs/operations/observability-and-operator-runbook.md

Observability and operator reference

This runbook defines the production signals and privileged controls for the Rendezvous service. The service emits System.Diagnostics.Metrics instruments from the FinalFactory.Rendezvous meter and distributed-tracing activities from FinalFactory.Rendezvous.Server. Connect those sources to the deployment's OpenTelemetry or equivalent collector. Do not add identifiers to metric labels.

Concrete detect/contain/recover/verify procedures for abuse, key compromise, targeted revocation, restart, rollback, saturation, privacy incidents, and dependency upgrades are in the incident and change runbooks.

Health and readiness

• GET /health/live proves that the HTTP process can answer. It deliberately remains independent of provisioning, the state store, drain state, and optional listeners so an orchestrator does not restart a recoverable dependency failure.
• GET /health/ready returns success only after the HTTP path is answering, the required IPv4 UDP socket is bound, any configured IPv6 UDP socket is bound, provisioning loaded successfully, the store is available, and drain has not started. A failed check returns 503 and removes the instance from new work.
• A graceful drain immediately makes readiness fail while liveness remains healthy. Existing work may complete until the bounded store drain deadline.

Metrics and traces

Instrument	Purpose	Bounded dimensions
rendezvous.http.requests / rendezvous.http.duration	HTTP volume and latency	operation, status code
rendezvous.udp.results / rendezvous.udp.duration	UDP mediation volume and processing latency	frozen/litenet operation, result
rendezvous.limiter.drops	Requests shed by admission controls	transport, fixed partition class
rendezvous.operator.authentication	Accepted, forbidden, and rejected operator authentication	result
rendezvous.audit.events	Privileged action outcomes	fixed action, result
rendezvous.connection.outcomes	Client-reported direct-connect outcomes	normalized outcome, elapsed bucket
rendezvous.pairing.latency	Time from attempt creation to successful peer introduction	none
rendezvous.queue.depth	Active join-attempt queue depth	none
rendezvous.store.active_listings / active_leases / active_attempts / replay_markers	Current ephemeral load	none
rendezvous.store.expiry_churn	Cumulative natural expiry activity	none
rendezvous.store.available	Store health (1 available, 0 unavailable)	none

HTTP responses include X-Rendezvous-Correlation-ID. It is a generated trace ID or random value, never a caller-supplied session or player identifier. UDP and HTTP activities contain operation-level data only. Logs and traces must not add tokens, capabilities, session/listing IDs, player subjects, metadata, raw IP addresses, or endpoint values.

Recommended dashboard panels are request rate and p50/p95/p99 latency by fixed operation, UDP result ratio, direct connection success ratio, pairing latency, active listings/attempts, expiry churn, limiter drops, store availability, operator authentication results, audit action results, and signing-key windows.

Alerts

Tune thresholds from the normal production baseline, then keep these conditions as distinct actionable alerts:

• Signing key expiry: page when any required signing key has less than seven days before signUntil; escalate at 24 hours. Confirm a replacement is signing and the previous key remains verify-only for the maximum credential lifetime.
• Authentication spike: warn when rejected or forbidden operator authentication exceeds five attempts in five minutes. Treat unexpected publisher-authentication growth as a possible credential or integration incident.
• Direct success regression: warn when the connected outcome ratio falls more than 20% below its seven-day same-region baseline for 15 minutes, with a minimum sample floor. Break down only by bounded outcome and time bucket.
• Saturation: warn when queue depth remains above 70% of the configured attempt limit, limiter drops are sustained, or p95 latency exceeds the service objective; page at 90% or when lease-critical traffic is shed.
• Store degradation: page immediately when rendezvous.store.available is zero or readiness fails for the store. Rising expiry churn without corresponding new work is a warning for stalled clients or clock/configuration mistakes.
• Listener/config readiness: page when no ready instances remain. Investigate UDP bind failures, a configured-but-unbound IPv6 listener, provisioning errors, and unintended drain state separately.

Operator authentication and controls

Operator credentials use a signing key configured with CredentialKinds: ["Operator"]. Operator keys cannot be scoped to a game/environment or used for publisher credentials. Mint short-lived operator credentials through the trusted provisioning process, outside the public Rendezvous HTTP service, and grant only the required permission. Never place credentials in command history, URLs, logs, or support tickets.

The application also enforces a default-deny source boundary. Configure at most 32 exact operator source IPs in Rendezvous:AbuseProtection:OperatorAllowedAddresses; an empty list disables all operator HTTP access. Development permits loopback only. Production must place /v1/operator/* behind a private management listener or reverse-proxy ACL, list only the resulting trusted management source addresses, and block that path on the public edge. If forwarded headers are enabled, keep the existing exact-proxy, single-hop trust policy and allowlist the post-forwarding operator source. Verify from both an allowed management host and a denied public host before deployment. Denied sources are charged to the bounded general HTTP partition before credential or request-body processing, then receive 404; sustained denied traffic receives the same typed 429 overload response as other public traffic.

Operator traffic has a dedicated, bounded rate/concurrency partition and critical tracker-key reserve. Public browse/join saturation therefore cannot consume the operator control budget, while compromised management sources remain rate-limited.

The OpenAPI document defines the separate OperatorBearer scheme. All endpoints are under /v1/operator:

Endpoint	Permission	Confirmation
GET /status	ReadPolicy	none; returns aggregates, tenant status, safe key status, and audit counts
POST /listings/revoke	RevokePublisher	repeat the exact listing ID in confirmListingId
POST /principals/revoke	RevokePublisher	repeat the exact subject and choose a 1-600 second revocation lifetime
POST /keys/revoke	RotateKeys	repeat the exact key ID; runtime revocation is immediate
POST /drain	ManagePolicy	send the exact value DRAIN

Publisher credentials are rejected on this surface even if their subject resembles an operator. Destructive responses do not echo identifiers. The status response does not expose player identities, raw endpoints, session metadata, capabilities, or tokens. Every authenticated operator action, rejected confirmation, and permission denial is audited with actor and target fingerprints.

Key revocation is process-local in the current single-instance store. Apply the same revocation to every instance, then replace configuration before restarting; a restart reconstructs the configured key ring. Principal revocation is bounded to ten minutes and removes that principal's active listings and attempts. A repeat action may extend an active revocation but never shortens it; wait for its original deadline rather than treating a shorter repeat as an un-revoke. Use listing revocation for one targeted session and drain before planned shutdown.

Audit retention and incident handling

The in-process audit trail defaults to 10,000 entries and 30 days. It evicts the oldest record at capacity and purges expired records on the next write. Configure Rendezvous:Audit:MaxEntries and RetentionDays within their validated bounds. Export the structured AuditTrail log events through the deployment's protected logging pipeline when durable retention is required; the in-memory trail is not a durable compliance archive. Those events include only timestamps, fixed action fields, correlation IDs, and actor/target fingerprints.

Audit records retain timestamp, fixed action/result, target kind, correlation ID, and 96-bit SHA-256 fingerprints of actor and target. Routine logs contain only the fixed action/result/target kind and correlation ID. Restrict audit access to the operator role, retain aggregates only as long as operationally necessary, and delete raw exported audit data according to the 30-day policy unless an incident hold is approved.

During an incident: confirm readiness and store health; capture aggregate graphs and correlation IDs; revoke the narrowest listing, principal, or key; drain only when isolation is required; record the action in the incident timeline; and verify that direct success, limiter drops, and authentication rates return to baseline. Do not copy player data, endpoints, or credentials into the incident record.