Observability that earns its keep
Every team I have joined had monitoring. Fewer had alerting anyone trusted, and incidents tend to happen in that gap.
Scope
What I take on
Instrumentation
Datadog, New Relic, Prometheus and Grafana, plus Splunk and ELK for logs. Getting the signals out of the system in the first place.
Custom checks
The checks that matter are usually the ones specific to your application, and writing those is most of the value.
Alert hygiene
Cutting the alerts nobody acts on. An on-call rotation that has learned to ignore its pager is worse off than one with no pager.
Dashboards people actually open
Fewer panels, chosen to answer the questions people ask at three in the morning.
Method
How it runs
-
01
Start from the incidents you have had
Your last six incidents tell you what to instrument far better than a vendor checklist does.
-
02
Alert on symptoms, not causes
Page when users are affected or about to be. Everything else goes on a dashboard or into a ticket, and nobody gets a phone call at 03:00 for it.
-
03
Every alert needs an action
If there is nothing to do about it, it should not be an alert. That one rule removes most of the noise in a typical estate.
Evidence
Where I have done this before
- LeasePlan Designed a Datadog alerting layer with custom checks that caught performance bottlenecks early, on infrastructure where availability was contractual.
- 3D Hubs Prometheus and the ELK stack across rebuilt AWS environments.
- Knab New Relic across a regulated banking platform, and Splunk in the earlier engagement.
Fit
Probably worth a conversation if
- You find out about outages from customers.
- Your on-call rotation stopped reading the alerts a while ago.
- You are paying for an observability platform and using ten percent of it.
- Something is slow, intermittently, and nobody can prove where.
Contact
Tell me what is actually broken
The contact details live on the front page. If I am not the right person for this, I will say so.