DK · Senior Software Engineer
Production reliability
Professional work
Finding a fault in a live system, tracing it, and leaving a fix that the next person can understand.
Engineer on the team, including production support around releases. 2021 – Present. Ongoing. This note stays free of incident detail.
Overview
A senior share of my job is what happens after the release. Redirects that do the wrong thing, API integrations that fail in a way the happy path never showed, and behaviour at the edge of the network. I have worked through production issues in those areas, including Cloudflare-related faults, without treating them as a separate discipline from the feature work.
The problem
Production faults are usually a stack of small mismatches. A redirect rule, a cached response, a client assumption, an API that changed shape. The useful skill is to reproduce the failure and narrow it, not to guess in public.
Architecture
I use the signals we already have: logs, the monitoring stack (including New Relic), the release that just went out, and a local reproduction where that is possible. Observability is only useful if someone is willing to read it during an incident and then tidy the code so the same trap is louder next time.
Decisions
- Reproduce first. A theory that I cannot trigger is not a fix.
- Prefer a small, reviewed change over a clever one when the site is already misbehaving.
- Write down the cause in the pull request so the next similar fault is shorter.
Trade-offs
- A fast mitigation and a proper fix are different jobs. Shipping only the mitigation leaves the next release exposed. Shipping only the proper fix can leave users broken tonight. The trade-off is explicit, and it is a team decision.
Where it stands
I have supported production releases and the hypercare that follows them. I am not publishing incident timelines, customer impact, or internal hostnames.
No incident identifiers, configuration, or customer data.
Stack
- Production support
- Observability
- New Relic
- Cloudflare
- Web security
Back to lekker.dev