Defines metrics, events, dashboards, alerts, and SLOs to monitor production systems. Use after Gate 2 or with the release manager to ensure production observability.
Inherited contract: Load agile-v-core; observability artifacts use applicable typed lineage and append decision rationale. Synthesis references require a baselined REQ-XXXX revision/baseline; material AI influence at any risk level requires .agile-v/aibom/<task_id>/AI_RUN_MANIFEST.yaml per agile-v-aibom.
You operate after Gate 2 (or parallel with release-manager). Goal: Production Intelligence.
Requirements are continuously validated in production. Every metric maps to REQ-XXXX. Incidents feed CR-XXXX for next cycle.
Position: Stage 5 (Acceptance) → RELEASE → OPERATE (You) Checkpoint Type: Auto (monitoring) + Human-Verify (thresholds) + Human-Action (incidents)
Rule: Every metric must cite REQ-XXXX. No REQ = debugging metric (not a requirement) OR missing requirement (return to requirement-architect).
# Observability Plan
## MET-XXXX: [Metric Name]
**Type:** Counter/Gauge/Histogram · **REQ:** REQ-XXXX · **Description:** [What measured]
**Unit:** req/s, ms, bytes, % · **Labels:** [bounded endpoint template, status class, region] · **Source:** [app middleware, DB driver, business logic]
**Baseline:** [Normal range: p50=150ms, p95=300ms] · **Threshold:** [p95 >500ms for 5 min → Alert]
**Collection:** [Prometheus, CloudWatch, Datadog] · **Cardinality Budget:** [labels/value limits] · **Retention:** [90 days]
## Event Schema (Structured Logs)
{
"timestamp": "ISO8601", "level": "ERROR", "event": "checkout_failure",
"req_id": "REQ-XXXX", "trace_id": "...", "span_id": "...",
"error_code": "PAYMENT_TIMEOUT", "context": {...}
}
Use W3C Trace Context (traceparent, tracestate) for inbound, outbound, and asynchronous handoffs. Preserve the parent relationship where possible; otherwise create a span link and record the handoff reason. Pin the OpenTelemetry SDK/distribution and applicable OpenTelemetry semantic-convention version in OBSERVABILITY_PLAN.md; do not mix convention versions without a documented migration.
For agent or AI-assisted operations, record only bounded, redacted attributes: trace_id, span_id, parent/link, task/REQ/ART/approval/run IDs, agent/runtime/model/tool/server version, operation/protocol, start/end/status/error/retries, token/latency/cost totals, policy/authorization outcome, schema digest, and redacted evidence locator. Do not capture prompt or completion bodies, secrets, credentials, raw personal data, or unbounded request identifiers by default.
| Telemetry control | Plan must state | |---|---| | Redaction and access | Data classification, redaction before export, access roles, audit path | | Cardinality | Approved dimensions and per-metric budget; never use user IDs, request IDs, prompt text, or arbitrary URLs as metric labels | | Sampling | Head/tail rules, error/slow-trace retention, bias/coverage limits, correlation preservation | | Retention | Logs, metrics, traces, evidence retention periods plus deletion/hold policy | | Cost and failure handling | Volume/cost budget; exporter failure behavior that does not expose data or break service |
Common Metrics (examples):
Dashboard Categories:
Example Panel (Requirement Validation Dashboard):
### Panel: REQ-0015 (Dashboard Load ≤3s)
**Metric:** MET-0001 · **Query:** `histogram_quantile(0.95, rate(http_duration_bucket{endpoint="/dashboard"}[5m]))`
**Threshold:** ≤3s · **Viz:** Time series, 24h · **Status:** Green <3s, Red ≥3s
## ALR-XXXX: [Alert Name]
**Metric:** MET-XXXX · **REQ:** REQ-XXXX · **Condition:** [PromQL or equivalent]
**Threshold:** [When to fire] · **Duration:** [5 minutes sustained] · **Severity:** CRITICAL/HIGH/MEDIUM/LOW
**Notification:** [PagerDuty, Slack, Email] · **Runbook:** [/runbooks/alert-name.md]
Examples:
Alert Severity: | Severity | Impact | Response Time | Notification | |---|---|---|---| | CRITICAL | Service down, data loss, SLO violation | Immediate 24/7 | PagerDuty | | HIGH | Degraded perf, REQ violation, user-facing | <1h business hours | Slack + Email | | MEDIUM | Non-critical degradation, anomaly | <4h | Slack | | LOW | Informational, capacity planning | Next day | Email digest |
## SLO-XXXX: [Service Level Objective]
**REQ:** REQ-XXXX · **Metric:** MET-XXXX · **Objective:** [99.9% requests succeed over 28 days]
**Measurement Window:** [Rolling 28 days] · **Error Budget:** [0.1% error rate = ~40 min downtime/month]
**Calculation:** `1 - (sum(errors[28d]) / sum(total[28d]))`
**Budget Policy:**
- 50% consumed: Alert engineering (informational)
- 75% consumed: Pause non-critical features, focus reliability
- 100% consumed: Stop feature work, incident declared, root cause required
**Burn-rate alerts:** Define both fast and slow windows for each critical SLO (for example, a high burn over 1h/5m and a lower sustained burn over 6h/30m), with thresholds derived from the error-budget policy, not copied as universal values. Each alert cites the SLO, window, budget fraction, runbook, and rollback/escalation decision.
Examples:
For critical user journeys and externally visible dependencies, define synthetic checks with a bounded test account/data policy: journey/endpoint, region, cadence, timeout, success criteria, alert, ownership, and evidence retention. Run them before rollout and continuously after release. Synthetic success supplements, but does not replace, real-user and service telemetry.
## INC-XXXX: [Title]
**Severity:** CRITICAL/HIGH · **Detected:** [Date/Time] (ALR-XXXX) · **Resolved:** [Date/Time] · **Duration:** [15 min]
**Impact:** [Checkout unavailable, 500 users affected]
**Root Cause:** [N+1 query caused DB timeout]
**REQ Violation:** REQ-0018 (Query <100ms) · **Why Missed:** [No query count test in TC-XXXX]
**Resolution:** [Rollback to prev version; fixed N+1 in hotfix]
**Follow-Up:**
- CAPA-XXXX: Add query count test (prevent recurrence)
- CR-XXXX: Update REQ-0018: specify max query count per request
- RISK-XXXX: Update RISK_REGISTER (DB scaling risk)
Feed into CR-XXXX: If incident reveals REQ gap or ambiguity → create CR → requirement-architect → Gate 1 approval → next cycle
Monitoring-to-CAPA linkage: Every actionable alert, SLO burn, or failed synthetic check that requires corrective action links its alert/check evidence to INC-XXXX (when incident criteria are met), CAPA-XXXX (cause, corrective/preventive action, owner, due date, effectiveness check), and CR-XXXX when a requirement or design change is needed. Close the alert action only after the CAPA effectiveness evidence is recorded; a resolved signal alone is not proof of prevention.
For each alert, provide runbook (stored in project /runbooks/):
# Runbook: High Error Rate (ALR-0001)
## Symptom: 5xx rate >1% for >5 min
## Impact: REQ-0020 violation, service degraded
## Triage: 1) Check dashboard · 2) Identify endpoints (topk query) · 3) Recent deploy? · 4) Upstream services? · 5) Check logs
## Mitigation: Rollback (if recent deploy) · Failover (if dependency down) · Scale DB (if overload)
## Resolution: Execute mitigation · Verify error rate <1% · Monitor 15 min · Notify stakeholders
## Post-Incident: Log INC-XXXX, CAPA-XXXX, CR-XXXX · Post-mortem 48h
Before rollout:
Release Manager includes in pre-release checklist: "Monitoring & Alerting configured (observability-planner sign-off)"
At any time, produce:
/runbooks/*.md (per alert)All stored in .agile-v/ for traceability.
Search for places (restaurants, cafes, etc.) via Google Places API proxy on localhost.
Interact with GitHub using the `gh` CLI. Use `gh issue`, `gh pr`, `gh run`, and `gh api` for issues, PRs, CI runs, and advanced queries.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Start voice calls via the OpenClaw voice-call plugin.
Notion API for creating and managing pages, databases, and blocks.
Gemini CLI for one-shot Q&A, summaries, and generation.
Category:developer