Use when adding metrics, logging, tracing, or alerting to services. Guides the three pillars of observability, structured logging standards, and instrumentation patterns.
Instrument services from day one — never bolt it on after an incident.
| Pillar | Question | Shape | |---------|--------------------|-------------------------------------| | Metrics | WHAT is happening? | Counters, gauges, histograms | | Logs | WHY it happened? | Contextual structured events | | Traces | WHERE it happened? | Request flow across service boundaries |
Alert on symptoms, not causes. Alert on error rate crossing a threshold, not on a specific error message. Instrument at system boundaries (HTTP handlers, queue consumers, DB calls) — not deep internals.
Always JSON:
{"timestamp":"2026-01-15T08:12:03Z","level":"error","msg":"payment failed","trace_id":"abc123","user_id":"u-789","error":"timeout"}
| Level | Meaning | Production? | |-------|---------|-------------| | ERROR | Requires human action now | Yes | | WARN | Degraded but self-healing | Yes | | INFO | Business-significant events only | Yes | | DEBUG | Development diagnostics | Never |
| Signal | Metric | Example |
|--------|--------|---------|
| Rate | Requests per second | http_requests_total |
| Errors | Error rate % | http_errors_total / http_requests_total |
| Duration | Latency histograms | http_request_duration_seconds |
traceparent header)Every service exposes:
/metrics — Prometheus scrape target/livez — Liveness probe (/healthz deprecated since K8s 1.16)/readyz — Readiness probe/metrics endpoint exposed/livez and /readyz endpoints| Component | Choice |
|-----------|--------|
| Cluster monitoring | kube-prometheus-stack (Prometheus + Grafana + Alertmanager) |
| Log aggregation | Loki + Grafana Alloy (LogQL queries; replaces Promtail EOL 2026-03-02) |
| Tracing | OTel SDK → OTLP collector → backend |
| Python | structlog / opentelemetry-instrumentation-fastapi / Prometheus client |
| Rust | tracing + tracing-subscriber (JSON) / tracing-opentelemetry / metrics-exporter-prometheus |
| Don't | Why | |-------|-----| | Log PII or secrets | Compliance and security risk | | Use unstructured log lines | Can't query, aggregate, or alert | | Alert on every individual error | Alert fatigue — alert on rates | | Skip trace context in cross-service calls | Can't follow requests across boundaries | | High-cardinality labels (e.g., user IDs) | Prometheus OOM, index explosion | | Log full request/response bodies | Storage cost + PII risk |
Search for places (restaurants, cafes, etc.) via Google Places API proxy on localhost.
Interact with GitHub using the `gh` CLI. Use `gh issue`, `gh pr`, `gh run`, and `gh api` for issues, PRs, CI runs, and advanced queries.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Start voice calls via the OpenClaw voice-call plugin.
Notion API for creating and managing pages, databases, and blocks.
Gemini CLI for one-shot Q&A, summaries, and generation.
Category:developer