2 posts tagged with "observability"

A backend-for-frontend layer is one of the easier architectural decisions to get right. Each client gets a service that speaks its language, the domain services stay clean, and every team owns a box with a clear border drawn around it. An architect reviews the diagram and approves it — and the diagram is correct.

This is a field story about what the diagram doesn't show. Four correctly-written service calls, each behaving exactly as designed, turned one screen render into roughly a thousand queries against a sharded collection. The screen stopped loading, one collection's query time dwarfed every other collection in the cluster, and the ticket said "MongoDB is slow."

MongoDB was not slow. Nobody had ever counted what one user action actually costs.

In 2022, I added a trace id to the two shared libraries every service my team owned already used – the logger and the HTTP client – so that one query in Grafana would return every log line of one request, across every service it touched. The platform is an enterprise system built as microservices from day one; it runs more than 200 of them today across several teams. My team owns 80+ of those. The trace id now covers more than 100 – more than we own – because other teams adopted it.

Before that change, a cross-service bug on a test environment was a meeting: a QA engineer, a business analyst, the team lead, usually a developer or two, sometimes DevOps – several hours of reading logs by timestamp and reconstructing the call chain from memory of the code. After that, the same investigation became a text field on a Grafana board, and the person typing into it was most often the analyst, not an engineer. Minutes rather than hours, and in most cases no engineer at all.

The cost is stated up front because it matters. There are no span IDs. Which service called which is inferred from a forwarded User-Agent and timestamps rather than being known. That was the trade.

This is a field report: what debugging looked like, what changed, and two cases as they happened. The mechanism is documented as a reference architecture, RA-004, and reconstructed as runnable code in team-workspace; this article explains it only as far as the story needs. If you read one section, read the two cases.