One Vulnerability, a Hundred Pipelines, One Board
Twice a month, give or take, the same task lands on my team with a fresh CVE number attached: a high-severity vulnerability in a dependency; patch it everywhere, now. "Everywhere" is close to a hundred microservices, each one a separate GitLab repository with its own pipeline. And "now" does not pause anything else – feature work carries on, because the roadmap doesn't know what a CVE is.
The fixing itself stopped being the hard part a while ago. The hard part is what comes after: watching a hundred pipelines deliver the fix through unstable infrastructure and manual gates, and knowing – not hoping, knowing – that it landed everywhere. Until recently, that knowledge cost a wall of browser tabs and most of a working day.
This post is about the two weekend days that removed the tabs.