# CLAUDE.md — Phase 14: Self-Observability Polish

## Scope of This File

This file instructs you (Claude Code) to implement **Phase 14 only** of the Seedbox Command Center.

**Do not implement phase 15.** Stay in scope.

## Authoritative Specification

`PROJECT.md` §5.8 (Logging and Self-Observability) and §10.3 (Monitoring the Monitor) are authoritative. Read both. Also read:
- `PROGRESS.md` and `DECISIONS.md`.
- Phase 0's `internal/observability` — Phase 14 promotes it from "minimal health + events" to "Prometheus + Loki + OpenTelemetry".

## Phase 14 Mission

Bring self-observability to parity with the production-software standard `PROJECT.md §5.8` describes. Operators who already run a Prometheus + Grafana + Loki stack can wire the Command Center into their existing dashboards. Operators who don't get reasonable defaults out of the box (the Phase 0 endpoints).

No new product features. Phase 14 makes the system observable enough that the operator can debug it from outside when something goes wrong.

## Deliverables Checklist

Phase 14 is complete when every item below is true:

- [ ] Prometheus metrics endpoint at `GET /metrics` (no auth required if `metrics.allow_unauth` is true and the listen is Tailscale-only — which it is in production; document this exception clearly):
    - `command_center_request_total{path, method, status}` — HTTP request counter.
    - `command_center_request_duration_seconds{path, method}` — histogram.
    - `command_center_scrape_total{tracker, classification}` — scrape outcomes.
    - `command_center_scrape_duration_seconds{tracker}` — histogram.
    - `command_center_event_bus_published_total{topic}` — eventbus throughput.
    - `command_center_event_bus_subscriber_dropped_total{topic}` — backpressure drops.
    - `command_center_db_query_duration_seconds{database, query_kind}` — db latencies.
    - `command_center_integration_health{integration, status}` — gauge, 1 ok / 0 unhealthy.
    - `command_center_age_key_present{path}` — gauge.
    - `command_center_build_info{version, commit, go_version}` — gauge.
    - Plus the standard Go runtime metrics from `prometheus/client_golang`.
- [ ] Structured logs compatible with Loki / Promtail / Vector. Add explicit `level`, `component`, `correlation_id`, `trace_id`, `span_id` fields where applicable.
- [ ] OpenTelemetry traces on the integration adapter paths (autobrr, qBittorrent, cross-seed, tracker scrapes, LLM tool calls). Configurable via `otel.endpoint` (OTLP/gRPC). Defaults to disabled.
- [ ] Trace context propagation: incoming HTTP requests extract w3c traceparent; outbound HTTP requests (to autobrr, qBittorrent, etc.) inject it. Webhook deliveries (web push, Discord, ntfy) carry the trace id in their payloads where the spec allows (Discord embeds, ntfy headers).
- [ ] Heartbeat file: every 60 seconds the process writes the current UTC timestamp to a configurable path (`heartbeat.path`, default `/var/lib/command-center/heartbeat.ts`). Document the matching shell script that the operator's cron checks.
- [ ] Documentation:
    - `OBSERVABILITY.md` — describes the metrics shape, log fields, trace points, and gives ready-to-use Grafana dashboard JSON for the most useful panels.
    - `prometheus.example.yml` — scrape config for the operator to drop into their Prometheus.
    - `loki.example.yml` — same for Loki/Promtail.
- [ ] Tests: every exported metric is produced under a realistic load (synthetic via httptest); trace context propagates correctly through the autobrr client; heartbeat file is written and the cron check script returns non-zero on staleness.
- [ ] Phase 6's `automation_disconnected` and `client_unreachable` triggers now consume `command_center_integration_health` gauge transitions (so the rule engine and the metrics agree on "down").
- [ ] `PROGRESS.md` and `DECISIONS.md` updated. `DECISIONS.md` records: the metrics namespacing scheme (`command_center_*`), the trace sampling default (1.0 in dev, configurable in prod — but at single-operator throughput, full sampling is cheap), the policy on metric labels with high cardinality (don't put info_hash in labels; bucket where possible).

## Phase 14 Database Scope

No new tables.

## Repository Layout

```
internal/
  observability/
    metrics.go               # Prometheus registry + middleware
    metrics_http.go          # the /metrics handler (already minimal in Phase 0)
    trace.go                 # OTLP exporter setup
    propagation.go           # HTTP client / server propagation helpers
    heartbeat.go             # the heartbeat writer
    *_test.go
docs/                        # NEW
  OBSERVABILITY.md
  prometheus.example.yml
  loki.example.yml
  grafana/
    command-center.json
```

Files modified across the codebase:
- Every HTTP client (autobrr, qBittorrent, cross-seed, ollama, etc.) wraps its `http.Client` with the OTel + Prometheus instrumentation middleware.
- Every database query goes through a thin wrapper that emits the duration histogram metric.
- The Phase 0 access-log middleware adds trace_id / span_id to log lines.

Frontend: no changes.

## Working Rules

**`/metrics` requires no auth in production.** Operators want Prometheus to scrape this URL without managing a session cookie. Because the binary binds to Tailscale-only, "unauthenticated on the tailnet" is acceptable. `dev_mode` deployments can require auth via a config flag if the operator chooses; document the default explicitly.

**Don't put high-cardinality data in labels.** No info hashes. No release names. No URLs. Bucket aggressively: tracker by id, integration by name, scrape by classification. The `correlation_id` lives in log fields and trace baggage, never in metric labels.

**Tracing is off by default.** Operators opt in by setting `otel.endpoint`. The Phase 0 logger format remains backward-compatible.

**Heartbeat is a flat-file timestamp.** Operators may use the shipped script or write their own. The file lives in `/var/lib/command-center` so it survives in the same lifecycle as the database.

**Observability is not the product.** Phase 14 is plumbing. If a feature request creeps in ("can we add CPU temperature?"), defer it to Phase 15 polish unless it's truly self-observation.

## What "Phase 14 Complete" Looks Like

After Phase 14 ships, the operator can:

1. Add a Prometheus scrape config pointing at `https://<tailscale-hostname>:8443/metrics`.
2. Import the shipped Grafana dashboard JSON; see request rates, scrape success/failure, integration health, eventbus throughput.
3. Configure Loki/Promtail to tail journald for the command-center unit; query logs in Grafana with structured field filters.
4. Configure an OTLP collector; receive traces of every autobrr poll, every qBittorrent sync, every LLM tool call. Tap into a slow tracker scrape and see exactly where the time went.
5. Add a cron job that checks the heartbeat file every minute; alert via email if older than 90s. The "monitor the monitor" story from PROJECT.md §10.3 is now wired.
6. The Command Center's own SLO conformance is measurable, externally observable, and surfaces in the same dashboards as the rest of the operator's stack.

## Begin

Read `PROJECT.md §5.8` and `§10.3`. Implement metrics first (it's the lowest-risk and highest-value). Then log fields. Then OTel. Then heartbeat. Documentation last but mandatory. Stop at the end of Phase 14.