Composition / Internal cockpit

Mufflermen observability cockpit

A composed internal-facing cockpit wired from the 14 observability primitives — metric tiles up top in a dashboard grid, query bar, service map, log stream + active alert rules in the middle and a deep-dive section with the live trace, SLO cards, error budget burndown, correlation heatmap, anomaly strip, synthetic tests and the running incident timeline.

Production answer

Mufflermen observability cockpit is a reusable Oak Flats Muffler Men UI primitive with documented states, accessibility expectations, theme behavior, and implementation evidence.

Primary CTAReview Mufflermen observability cockpit states
Generative search brief

Mufflermen observability cockpit: A composed internal-facing cockpit wired from the 14 observability primitives — metric tiles up top in a dashboard grid, query bar, service map, log stream + active alert rules in the middle and a deep-dive section with the live trace, SLO cards, error budget burndown, correlation heatmap, anomaly strip, synthetic tests and the running incident timeline.

Composition · internal observability cockpit

Quotes API

RPS · 1m
RPSquotes-api
198req/s+8.2%vs 1h
RPS trend sparklineTrend over 48 samples ranging from 163.2 to 202.7.

Quotes API

p95 latency · 5m
p95 latencyquotes-api
142ms-12%vs 1h
p95 latency trend sparklineTrend over 48 samples ranging from 106.4 to 174.1.

Parts catalogue

p95 latency · 5m
p95 latencyparts-catalogue
188ms+4%vs 1h
p95 latency trend sparklineTrend over 48 samples ranging from 161.3 to 208.0.

Quote PDF

render p95 · 5m
render p95quote-pdf
484ms+24%vs 1h
render p95 trend sparklineTrend over 48 samples ranging from 317.1 to 483.0.

Workshop scheduler

p95 latency · 5m
p95 latencyworkshop-scheduler
96ms0%vs 1h
p95 latency trend sparklineTrend over 48 samples ranging from 78.2 to 109.3.

Customer SMS

delivery error rate · 5m
error ratecustomer-sms
0.61%+412%vs 1h
error rate trend sparklineTrend over 24 samples ranging from 0.0 to 0.0.
alert threshold > 2%
Metric
Filter
Group by
Queryhttp.request.duration.p95{service="quotes-api", env="prod", region="au-east-1"} by (region)
Production topology · prod · au-east-1
198 rps96 rps42 rps8 rps612 rps184 rps38 rps24 rpsedge — 412 rps p95 28msedgeext · 28msquotes-api — 198 rps p95 142msquotes-apisvc · 142msparts-catalogue — 96 rps p95 88msparts-cataloguesvc · 88msworkshop-scheduler — 42 rps p95 96msworkshop-schedulersvc · 96msquote-pdf — 8 rps p95 482msquote-pdfsvc · 482msredis — 612 rps p95 4msredisdb · 4mspostgres — 184 rps p95 22mspostgresdb · 22mssms-queue — 24 rps p95 14mssms-queuequeue · 14ms
Filter8 / 8
Live log stream · all services
Live log stream · all services
TimeSevServiceMessage
19:42:14.203customer-sms
carrier returned 5xx after 2 retries; queueing for backoff
  • traceIdf3a8…d12
  • carrierapac-2
  • retries2
19:42:14.118quotes-api
slow downstream call to parts-catalogue p99=812ms
  • traceIdf3a8…d12
  • routePOST /quotes
19:42:13.982quotes-api
quote created for vehicle WV1ZZZ7HZ8H012345
  • quoteIdQ-198342
  • customerIdC-44182
19:42:13.711workshop-scheduler
bay-3 reservation locked for 2026-05-29 09:30
  • bookingIdB-9921
  • bayIdbay-3
19:42:12.504quote-pdf
pdf render queued for quote Q-198341
  • renderIdr-7733
19:42:11.901payment-gateway
card auth declined: insufficient funds
  • paymentIdP-7711
  • code51
19:42:10.882quote-pdf
renderer worker OOM at 1.4 GB resident, restarting
  • renderIdr-7728
  • workerIdwkr-3
19:42:10.140parts-catalogue
warmed cache for sku family 'exhaust-clamps' (482 skus)
  • cachewarm
  • duration184ms

quotes-api p95 latency

quotes-api
Last 6 hours · 10m buckets3 anomalies
quotes-api p95 latency time series with 3 anomaliesTrend over 36 samples ranging from 138.0 to 412.0.
  • Brief spike on parts-catalogue p99 cascadet-23
  • PDF queue backlog caused upstream latencyt-7
  • Drift outside forecast band — investigatingt-5
Trace 7a3c…b1e2 · POST /quotes · 482ms
0122ms244ms366ms488ms
Metric correlation · last 24h
quotes.rps
quotes.latency.p95
parts.latency.p95
scheduler.bay.util
pdf.error.rate
carrier.5xx
quotes.rps
1.00
0.42
0.36
0.58
0.18
0.08
quotes.latency.p95
0.42
1.00
0.82
0.24
0.36
0.16
parts.latency.p95
0.36
0.82
1.00
0.12
0.31
0.04
scheduler.bay.util
0.58
0.24
0.12
1.00
-0.18
0.02
pdf.error.rate
0.18
0.36
0.31
-0.18
1.00
0.62
carrier.5xx
0.08
0.16
0.04
0.02
0.62
1.00
-1.00.0+1.0
Synthetic tests · last run · 6 regions
API

GET /api/quotes/health

Regionau-east-1
2m ago
124ms99.98% uptime
API

POST /api/quotes

Regionau-east-1
2m ago
484ms99.86% uptime
Browser

Browser: book service flow

Regionau-west-1
5m ago
1842ms98.92% uptime
API

POST /api/quote-pdf/render

Regionus-east-1
4m ago
3122ms96.18% uptime
API

GET /api/parts/search

Regioneu-central-1
3m ago
312ms99.92% uptime
ICMP

ICMP edge probe

Regionapac-1
6m ago
0ms96.42% uptime

INC-2026-05-28-quote-pdf-oom

6 events
  1. 19:42 AESTDetect

    PDF render error rate breaches SLO (4.4%)

    Threshold > 2% for 5m of 5m. Anomaly detection flagged a drift event 6 minutes prior.

    by alertmanager
  2. 19:43 AESTPage

    On-call paged for quote-pdf service

    by PagerDuty
  3. 19:46 AESTAcknowledge

    Sasha B acknowledged the page

    by Sasha B
  4. 19:51 AESTComms

    Internal status update posted to #incidents

    by Sasha B
  5. 19:58 AESTMitigate

    Renderer pool resized 3 → 6; OOM threshold lifted

    by ops automation
  6. 20:08 AESTResolve

    Error rate back under 0.5%, SLO restored

    by alertmanager