Every team adds monitoring after their first bad night. The work is identical either way - the only difference is whether you did it calmly or at 3am with users complaining.
Alert on symptoms, not causes
Page on what the user experiences: error rate, latency, failed payouts. CPU at eighty percent is not an incident if nobody notices.
The three things you will want
- A request id that ties a user's complaint to a specific trace
- Structured logs you can filter, not printed strings
- A dashboard showing the money path end to end
Every alert needs an action
If the response to a page is 'watch it for a bit', it should not have paged. Alerts that cannot be acted on train people to ignore all of them.
Practise the failure
Turn off a dependency in staging and watch what happens. The first time you see your own failure mode should not be in production.
