Rendezvous/docs/operations/diagnostic-dashboards.md

# Diagnostic dashboards

Tracking: #27

Rendezvous provides two deliberately separate, optional views. The public
session diagnostic helps a player or integration operator understand safe
session-list state using only the public browse contract. The private Grafana
dashboard exposes aggregate operational health through authenticated metrics.
Neither view grants operator privileges or exposes player identity, endpoints,
credentials, capabilities, or raw session metadata beyond the explicitly
allowlisted public browse fields.

## Public read-only session diagnostic

The static diagnostic is disabled by default. Enable it only for approved
game/environment scopes and keep the allowlist narrow:

```json
"Diagnostics": {
  "Enabled": true,
  "PollIntervalSeconds": 10,
  "MaximumRenderedSessions": 100,
  "Scopes": [
    {
      "GameId": "space-game",
      "EnvironmentId": "smoke",
      "ProtocolVersions": [1, 2],
      "Regions": ["local"]
    }
  ]
}
```

Open `/diagnostics` on the same origin as Rendezvous. The page cannot choose a
different backend, request private visibility, join a session, or call the
operator surface. It renders a bounded snapshot, then applies ordered SSE
updates. A replay reset, corrupt cursor, incomplete snapshot, transport failure,
or deliberate reconnect returns to a fresh authoritative snapshot and bounded
polling. Apply filter changes explicitly; **Reset filters** restores the
configured defaults, while **Reconnect now** tests recovery without changing
the selection.

The page uses semantic HTML, labelled controls, visible keyboard focus, status
text in addition to color, a reduced-motion mode, and a 320-pixel reflow. It
creates untrusted content with `textContent` only. The endpoint sets a restrictive
same-origin content-security policy, denies framing, disables MIME sniffing and
browser capabilities, and marks every asset/config response `no-store`.

This is a diagnostics convenience, not a game browser, management console, or
availability monitor. Disable it independently by setting `Enabled` to `false`;
all diagnostic paths then return `404` without affecting game traffic, metrics,
or health endpoints.

## Private Prometheus and Grafana view

The observability overlay pins Prometheus 3.13.1 and Grafana 13.1.0 by immutable
multi-platform image digest. Prometheus is not published to the host. Grafana is
bound to host loopback, disables anonymous access and sign-up, and reads its
administrator password from a file. The service and Prometheus share only the
metrics bearer-token file. All three secrets remain ignored by Git.

Create independent random secrets, then start the base service and overlay:

```bash
install -d -m 0700 deploy/compose/secrets deploy/observability/secrets
umask 077
openssl rand -out deploy/compose/secrets/signing-key 32
openssl rand -hex 32 >deploy/observability/secrets/rendezvous-metrics-token
openssl rand -base64 36 >deploy/observability/secrets/grafana-admin-password
export RENDEZVOUS_UID="$(id -u)"
export RENDEZVOUS_GID="$(id -g)"
test "$RENDEZVOUS_UID" -ne 0
docker compose \
  -f deploy/compose/compose.yaml \
  -f deploy/observability/compose.yaml \
  up --build --detach
```

Visit `http://127.0.0.1:3000`, sign in as `rendezvous-admin`, and open the
**Rendezvous operational overview** folder/dashboard. The provisioned panels
cover scrape/store/drain health, listing and join capacity, HTTP volume/errors
and p95, browse/SSE load, lease and join operations, UDP results/latency/bytes,
admission drops, connection outcomes, pairing latency, presence/expiry state,
signing windows, security/audit results, and process/GC/descriptor pressure.
Capacity gauges use the approved single-process envelope of 25,000 listings and
10,000 active attempts, with 70% warning and 90% critical thresholds. The UDP
response series is a conservative admitted maximum, not observed egress.

Validate merged configuration and checked-in dashboard structure before every
rollout:

```bash
RENDEZVOUS_UID="$(id -u)" RENDEZVOUS_GID="$(id -g)" \
  docker compose \
    -f deploy/compose/compose.yaml \
    -f deploy/observability/compose.yaml \
    config --quiet
./scripts/test-observability-assets.sh
```

For a real deployment, keep Grafana on a private authenticated management
network instead of host loopback, replace the local admin login with the
organization's supported identity boundary, enforce TLS at the edge, and set
retention to the approved operational period. Do not make Prometheus public.
Provisioning is read-only so local UI edits cannot silently drift from source.

## Verify, rotate, and disable

After startup, verify the dashboard shows `UP`, store `AVAILABLE`, a nonzero
signing window, and changing request/UDP panels during a smoke run. Confirm an
unauthenticated `/metrics` request returns `404`, the bearer-authenticated
collector target is healthy, Prometheus is not bound on a host port, Grafana is
not anonymously accessible, and dashboard query labels contain no identifiers.

Rotate metrics access by writing a new 32-128 character token to the secret file
with private permissions and restarting Rendezvous and Prometheus together.
Rotate the Grafana administrator password through the same protected secret
workflow. Delete both secret files after a disposable local run.

To disable aggregate observability independently, stop/remove the overlay and
set `Rendezvous:Metrics:Enabled` to `false`; `/metrics` returns `404` and the
core service continues. To disable only the public session diagnostic, leave the
overlay running and set `Rendezvous:Diagnostics:Enabled` to `false`.