# SRE Dashboard — Design and Operating Notes
This directory contains the data model for a homelab SRE dashboard that overlays the cluster's existing Prometheus instrumentation with a declarative resource graph, derived service views, and signature-driven runbook routing. The dashboard renders from these YAML files via the Next.js routes added to `oberon-ui` (`/sre`, `/services/[name]`, `/runbooks/[id]`).
This document explains what the dashboard is, why it was built this shape, what's currently modelled, and what's been learned from operating it.
## Why this exists
The cluster already had observability — kube-prometheus-stack with node-exporter, kube-state-metrics, Loki, and AlertManager, plus one Grafana dashboard for the Puck trading bot. That stack answers individual questions: "what is this metric doing right now," "did this alert fire," "are these pods running."
What it didn't answer is the structural one: **what depends on what, and how do failures propagate?** Grafana dashboards are flat panels over flat metrics. They don't carry the topology that determines blast radius. When a node fails or a network link degrades, the operator has to mentally reconstruct which services are affected, which signals will start firing, and which runbooks apply.
The SRE dashboard adds the missing layer: a declarative graph of every resource the cluster depends on, with state that propagates along dependency edges, services modelled as derived views, and signatures bound to runbooks. The result is meant to read like an x-ray of the system rather than a wall of graphs.
## Design principles
The shape of the model is opinionated and deliberate.
**1. Resources are first-class and atomic.** Every distinct thing that produces U/S/E metrics is its own resource. CPU is a resource. Memory is a resource. The IBKR connection is a resource. The MikroTik's wifi2 interface is a resource. There is no hierarchy where services "own" their resources. Resources stand on their own and exist in a single flat namespace. This mirrors the SRE Book's framing in chapter 6 (Monitoring Distributed Systems) where the primitives are the observable signals themselves, not the organisational units that produce them.
**2. Services are derived views, not authored entities.** A service is `(name, primary_resource)`. Its membership is computed as the transitive closure over dependency edges. Its SLOs are derived aggregates over member resource states. This avoids the trap of services being parallel authorities with their own metrics — a common antipattern in observability tooling where a single underlying problem gets reported five times by five different service dashboards.
**3. State propagates along edges, weighted by criticality.** Each resource has an `own_state` derived from its own metric thresholds, and an `effective_state` that combines own_state with the propagated state of upstream dependencies. Hard-criticality upstream issues propagate red; soft-criticality issues propagate as one severity tier down. The dashboard surfaces both states distinctly, so an operator can immediately tell "I'm broken because of me" from "I'm broken because something upstream is broken."
**4. Signatures and runbooks are scope-independent.** Signatures are predicates over resource state. Runbooks are bound to signatures. Neither has any concept of "which service this belongs to." A runbook applies wherever its signature fires, regardless of which service is affected. Incidents propagate by causal chain, not by service boundary, and the routing of remediation should follow the same logic.
**5. The model is declarative, version-controlled, reviewable.** The whole thing is a small set of YAML files in a git repo. Changes flow through the same review process as any other code. The dashboard rebuilds from these files on each request (cached via ISR), so there is exactly one source of truth.
## The data model in brief
Five concepts:
- **Resources** — atomic units producing U/S/E metrics. Defined in `resources.yaml`. Each has explicit thresholds per metric, so its `own_state` (green/yellow/red/unknown) is computed deterministically.
- **Edges** — dependency relationships in `edges.yaml`. Each edge has a `criticality` (hard or soft) that controls how upstream state propagates.
- **Services** — views over the resource graph in `services.yaml`. Composed of a name, primary_resource, and derived SLOs. Membership is computed by transitive closure; not authored.
- **Signatures** — predicates over resource state in `signatures.yaml`. Fire per-matched-resource when conditions hold for a configured window. Each signature is bound to a single runbook.
- **Runbooks** — markdown files in `runbooks/`. Each has front-matter metadata and a body following a standard structure (symptom, causes, diagnostic steps, mitigation, verification, prevention).
The full schema with field-level documentation is in `SCHEMA.md`.
## What's currently modelled
The cluster being represented is a four-node bare-metal Kubernetes installation with one user-facing trading bot (Puck), one public dashboard (oberon-ui), one Postgres datastore, and a MikroTik gateway routing through a consumer WiFi uplink to the public internet.
The current resource graph has roughly 45 resources covering:
- **Four hosts** (red as control plane, blue and yellow as workers, green offline due to hardware fault since 2026-06-15). Each host is decomposed into CPU, memory, disk, network, and file-descriptor sub-resources for USE-by-the-book exhaustiveness.
- **MikroTik gateway** decomposed into CPU, memory, NAT connection table, and per-interface (bridge and wifi2) sub-resources. These will populate once `mikrotik-exporter` is deployed.
- **Puck application invariants** as leaf resources: process state, IBKR connection, Postgres connection, account drawdown, PIX value staleness, tick rate. These are the business-level invariants the service must maintain; modelling them as resources lets the SLO machinery reason about them uniformly with hardware concerns.
- **Three services**: puck, oberon-ui, network. Each declares two to five derived SLOs using the `requires_green` aggregator.
- **Three externals**: internet (single uplink, no failover), IBKR (trading dependency), Cloudflare (public exposure tunnel).
The graph has explicit single-points-of-failure visible in the topology: every service routes through the MikroTik, every Puck operation depends on the IBKR external, every public dashboard access flows through Cloudflare. The dashboard doesn't try to hide these facts.
## State propagation in practice
The decision to propagate state with criticality-weighting (rather than worst-of, naive boolean, or no propagation at all) was specifically to avoid two failure modes common in observability tooling:
The first is **alert flooding**, where a single underlying problem causes hundreds of derived alerts to fire across every dependent. The criticality-weighted model means a hard-criticality upstream failure causes one downstream propagated-red state per direct dependent, not a recursive cascade across the whole graph at full severity.
The second is **state opacity**, where the dashboard shows a service as red without indicating whether the problem is in the service or upstream of it. The `own_state` and `effective_state` split makes this immediately visible: a green own_state with red effective_state surfaces "I am fine but my upstream is broken." That is a fundamentally different operational situation than "I am broken" and the dashboard treats them as visually distinct.
The signature `service-degraded-by-upstream` in `signatures.yaml` is bound specifically to this case and demonstrates the pattern.
## Lessons learned
The model has been informed and shaped by real incidents on the cluster. Several are worth documenting:
**The 2026-06-15 green node failure** revealed how stuck-Terminating pods on an unreachable node cascade into broken control plane services. When green stopped reporting, the Prometheus and Alertmanager StatefulSets that were scheduled to it became un-recoverable until force-deleted. The local-path PVCs were similarly pinned to the dead node. This shaped the modelling of nodes as having sub-resources with their own state, so that the dashboard can flag specific failed components rather than presenting the host as a single binary up/down indicator. It also informed the `node-hardware-recovery` runbook.
**The 2026-06-28 post-house-move incident** exposed how the wifi2 client uplink is a single point of failure for everything in the cluster. When the MikroTik's stored WiFi credentials no longer matched the new house's consumer router, the default route went inactive and all egress stopped — including the Cloudflare tunnel that exposes the public dashboard. Recovery required updating wifi2's SSID and password configuration directly. The `router-uplink-recovery` runbook captures the playbook from that incident, and the `network-interface/mikrotik/wifi2` resource carries an annotation explicitly noting its SPOF status. A future redundant uplink (perhaps a 4G dongle for out-of-band management) is an obvious resilience improvement but not yet implemented.
**The 2026-06-29 AP instability sessions** showed that frequent failures on the MikroTik AP are likely a combination of 2.4GHz channel congestion in the new RF environment and the newer RouterOS wifi-package having stability quirks that haven't been fully resolved. The signature taxonomy includes `router-cpu-high` and `router-conntrack-near-full` to surface resource-level pressure that would correlate with these failures, though full diagnosis depends on mikrotik-exporter being deployed.
The pattern across these incidents is consistent: the cluster has many single points of failure, and each one is now reflected in the resource graph and the signature catalogue. The dashboard exists in part to make those weaknesses operationally visible, not to hide them.
## What's not built yet
This is an honest scope statement.
**Mikrotik-exporter is not deployed.** All `network-device/mikrotik*` and `network-interface/mikrotik/*` resources currently show as `unknown` state because their backing metrics are empty. Deploying mikrotik-exporter is a known TODO; the YAML queries are written against the metric names it exposes, so deploying it will retroactively populate the dashboard without YAML changes.
**The dashboard auto-discovery layer is not implemented.** Resources, edges, and services are all hand-authored. For a homelab, this is acceptable — the cluster is small enough that the YAML is reviewable and drift is manageable. For a larger system, auto-discovery from the Kubernetes API and Prometheus targets is essential; the schema is designed to support that as a future addition.
**Runbooks are partially populated.** Three runbooks are written (ibkr-disconnected, router-uplink-recovery, node-hardware-recovery), corresponding to the incidents that have actually occurred. The remaining signatures reference runbooks that have not yet been written; the renderer is expected to degrade gracefully by showing a "runbook not yet written" placeholder. The intent is to write each new runbook in response to the incident that revealed the need for it, not speculatively.
**Multi-provider metric ingestion is not implemented.** The Prometheus client is the only backend. The design separates ingestion from the canonical resource model, so adding OpenTelemetry, CloudWatch, or other sources later would be additive rather than a rewrite.
**Historical replay is not implemented.** The dashboard renders current state. Time-travel views ("what did the graph look like at 14:08 UTC during the incident?") would be a meaningful postmortem aid but are not in scope for the MVP.
## References
The design draws on:
- *Site Reliability Engineering* (Google), particularly chapters 4 (Service Level Objectives), 6 (Monitoring Distributed Systems), 10 (Practical Alerting), and 22 (Addressing Cascading Failures).
- *The Site Reliability Workbook* (Google), chapter on capacity planning.
- *Designing Data-Intensive Applications* by Martin Kleppmann, chapter 1 (Reliable, Scalable, and Maintainable Applications) and chapter 8 (The Trouble with Distributed Systems).
- Brendan Gregg's USE method (utilization, saturation, errors per resource).
- Pat Helland's "Immutability Changes Everything" — the framing of derived views over an immutable substrate informed the services-as-derived-views decision.
- ACM Queue articles on observability and incident response (various).
Specific design choices and their sources are cited inline in `SCHEMA.md` where relevant.
## Architecture diagram
```mermaid
flowchart TD
subgraph "External"
IBKR[external/ibkr]
Internet[external/internet]
CF[external/cloudflare]
end
subgraph "Network"
MT[network-device/mikrotik]
Wifi2[network-interface/mikrotik/wifi2]
Bridge[network-interface/mikrotik/bridge]
MT --> Wifi2
MT --> Bridge
Wifi2 -.routes-through.-> Internet
Internet -.routes-through.-> IBKR
Internet -.routes-through.-> CF
end
subgraph "Hosts"
Red[host/red control-plane]
Blue[host/blue worker]
Green[host/green OFFLINE]
Yellow[host/yellow worker]
Red -.routes-through.-> MT
Blue -.routes-through.-> MT
Green -.routes-through.-> MT
Yellow -.routes-through.-> MT
end
subgraph "Services"
Puck[service/puck]
UI[service/oberon-ui]
PG[datastore/puck-postgres]
Puck --> PG
Puck -.depends-on.-> IBKR
UI -.calls.-> Puck
UI -.depends-on.-> CF
Puck -.runs-on.-> Blue
PG -.runs-on.-> Blue
UI -.runs-on.-> Yellow
end
subgraph "Puck invariants"
State[app/puck/process-state]
IBKRConn[app/puck/ibkr-connection]
PGConn[app/puck/postgres-connection]
Drawdown[app/puck/account-drawdown]
Puck --> State
Puck --> IBKRConn
Puck --> PGConn
Puck --> Drawdown
IBKRConn -.depends-on.-> IBKR
PGConn -.depends-on.-> PG
end
```
The diagram is simplified; the full graph including host sub-resources (CPU, memory, disk, network, FDs) is rendered in the dashboard itself.
## Alerting
Page-worthy alerts derive from the SLO observations in `conditions.yaml` using multi-window burn-rate rules (Google SRE Workbook model). The full philosophy — what gets paged, what stays as a dashboard signal, how conditions map to PrometheusRule manifests with worked examples — is in [`ALERTING.md`](./ALERTING.md).
The core principle: **alert on user-impact symptoms via SLO burn rate, not on per-resource thresholds.** Per-resource threshold violations drive dashboard signatures and runbook routing, not pages.
## Architecture caveat (current homelab limitation)
Prometheus, AlertManager, and this dashboard currently run inside the Kubernetes cluster they monitor. If the cluster fails, the observability stack fails with it — exactly when it's needed most. This is a known anti-pattern, accepted as a homelab constraint while there's only one cluster.
The intended next step is to move Prometheus + AlertManager + the dashboard renderer to a separate node (or a small dedicated cluster) once additional hardware lands. Until then, this stack catches failure modes that don't take the whole cluster down — which is most of them, but explicitly not all.
Documented openly on the dashboard's landing page rather than hidden, because pretending it's fine is the more interesting failure mode.