Prevention Has a Ceiling: Designing CI Pipelines That Survive Being Breached
A follow-up to The Silent Exfiltration.
That article described a pipeline in which a single malicious dependency could read every secret in the runner environment and post it to an arbitrary host, before any scan, on any branch, without a merge. It ended with a remediation ladder.
This one starts with an uncomfortable observation: organisations that climbed that ladder are still being hit. The largest software companies in the world, with dedicated supply chain security teams and eight-figure security budgets, had credentials compromised in this attack class in 2026. The conclusion is not that they were careless. The conclusion is that prevention has a ceiling, and above that ceiling a different approach is required – one that assumes the attacker is already executing within the build, and asks how much they obtain, how far they reach, and how long before anyone notices.
1. The Attack Did Not Stop. It Industrialised
The first article used the classic examples, including event-stream, ua-parser-js, colors, and faker. The incidents were memorable precisely because they were unusual – a maintainer handing over a package to the wrong person, a hijacked publishing account, and a developer sabotaging his own work.
Those were events. What followed was infrastructure.
Two things changed, and both matter more than any individual compromise does.
The payload learned to reproduce
The Shai-Hulud lineage introduced self-propagation to the npm ecosystem. The mechanic is simple, and it's why the numbers changed by orders of magnitude. A compromised package executes on install. It harvests credentials from the environment, the filesystem, and local credential stores. Among those credentials is a registry publishing token. The payload then uses that token to publish itself to every other package the victim maintainer can write to.
Each infection is a new distribution point. The growth is not linear.
Recent waves reached hundreds of packages across thousands of versions within hours of the first publication. No human operator was driving that. The operator wrote the seed and went to sleep.
The propagation detail that undermines most defensive assumptions is version numbering. Worm-published releases are patch increments. A package at 13.0.19 gets poisoned as 13.0.20. Every dependent declaring ^13.0.0 (which is to say, almost every dependent) resolves to it on the next unpinned install. No major version bump to review, no changelog to read, and no pull request to approve.
A committed lockfile blocks that path, and it is worth being precise about how much it blocks. npm ci, --frozen-lockfile, and their equivalents resolve to recorded versions and will not pick up the poisoned patch. What they do not cover is every install that runs outside them: pipeline utilities invoked with npx or installed on the fly, CI actions referenced by tag rather than digest, container images built without a lockfile, dependency-bump automation configured to auto-merge patch releases, and the developer laptop where somebody adds a package by hand and re-resolves the whole tree. Lockfile discipline in the application repository is not lockfile discipline across the estate, and the gap between the two is where this attack class lives.
In those cases, nothing enters your repository. Nothing appears in a diff. The malicious code arrives at build time from a registry you trust, under a version constraint someone wrote in good faith.
The blast radius became enterprise-scale
In March 2026, attackers compromised two releases of LiteLLM, an open-source gateway that normalises API calls across large language model providers, and published them to the Python Package Index. The compromised versions were available for roughly forty minutes.
But forty minutes was sufficient.
CloudSEK and Hudson Rock, examining the exfiltrated material months later, estimated the reach at more than 2,500 organisations and roughly 434,000 CI/CD pipelines [2]. The harvested material included cloud access keys, repository tokens, SSH private keys, Kubernetes secrets, package publishing credentials, environment variable dumps, and AI provider keys.
The list of affected organisations is the part worth sitting with. Microsoft, Amazon, Cisco, Samsung, Salesforce, Airbus U.S. Space & Defence, Thales, Deutsche Bahn, Munich Re, and The London Stock Exchange Group [1].
These organisations have security engineers. Several of them even sell security engineering services.
Two structural details of that campaign deserve more attention than they received.
The entry point was a security tool. LiteLLM itself was not directly compromised. The attackers targeted Trivy – an open-source vulnerability scanner – within LiteLLM's build pipeline and used that position to poison the published artefacts. The scanner was not the thing being scanned. It was the thing with write access.
Every organisation running a pipeline uses a set of tools to inspect the build. Scanners, linters, coverage reporters, SBOM generators, deployment helpers. These are installed with less scrutiny than application dependencies, precisely because they are security tooling, and they frequently run with higher privileges because they need to read everything. They are also the category most likely to be fetched at floating versions, because nobody pins their scanner the way they pin their framework. That is an attractive combination.
The root cause was an incomplete rotation. Reporting on the campaign indicates that the attackers held a leaked automation token that had been rotated but never fully revoked – leaving a window of roughly three weeks during which they could force-push to published version tags with code that still appeared legitimate to everyone downstream.
Hold that thought. It returns in section 6, and it is the single most important point in this article.
The cleanup failed too
After the leak became public, independent researcher Kevin Beaumont tested a sample of the exposed credentials. Some of them still worked, even though the affected organisation said it had rotated the keys [1].
In July 2026, the FBI issued a FLASH advisory warning that actors linked to the campaign were likely to weaponise the harvested credentials long after the original intrusion [2]. Credentials do not expire simply because an incident report was closed.
So: an attack that required no merge, no code review failure, and no human error at the victim organisation. Executed against companies with world-class security functions. Detected months later. Remediated incorrectly.
This is not a story about negligence. It is a story about a control model that is misaligned with the threat.
2. The Ceiling on Prevention
Preventive controls in CI/CD share an assumption that does not hold under this attack class: that there is a moment, before anything dangerous happens, when a decision can be made.
There usually is not.
The kill chain completes before the controls fire
Consider the ordering of a conventional pipeline.
Secrets are injected into the job environment at container start, because the job might need them. Dependencies are installed early, because everything downstream depends on them. Security scanning runs after installation, because it requires a resolved dependency tree. Enforcement decisions – blocking a merge, failing a deploy – arrive later still, often several stages and tens of minutes after installation completes.
A malicious lifecycle script executing during installation therefore encounters an environment in which every secret is already present and nothing that could have stopped it has run yet.
By the time the scanner reports, the credential is gone. The scanner's verdict determines whether a bad artefact ships. It has no bearing on whether a secret left the building, because that happened at minute three and the verdict arrives at minute twenty-eight.
This produces the most common category error in pipeline security: artefact-integrity controls are widely deployed as though they were credential-protection controls. Blocking a build limits what reaches production. It does not un-exfiltrate anything.
Gate placement matters more than gate quality. A mediocre control that runs before installation protects more than an excellent one that runs after installation.
Provenance proves the wrong thing
Supply chain security has spent several years converging on attestation. SLSA, in-toto, Sigstore, and trusted publishing. The promise is verifiable build provenance: cryptographic evidence of what was built, from what source, and by what pipeline.
The evidence is real. What it attests to is narrower than most readers assume.
Provenance establishes that an artefact was produced by a specific build process from a specific source revision, and that neither was tampered with in transit. SLSA's threat model is explicit about what remains out of scope at every level: malicious source code committed by an authorised identity, compromise of an upstream dependency, and account takeover of a legitimate maintainer [3].
Every recent major campaign falls into those categories.
When a worm publishes using a stolen maintainer token, the release pipeline is genuine. The source revision is from the genuine repository. The signature is valid. The attestation is correct. Everything verifies, yet the package is malicious.
Attestation answers "was this built by the pipeline it claims?" It does not answer "was what entered the pipeline safe?" Those are different questions, and the second is the one that decides whether you are compromised.
This is worth stating plainly because attestation is currently being sold as the answer to supply chain compromise, and organisations are adopting it as such. It is a genuine and valuable control against real threats – build system tampering and artefact substitution. It is not a control against a compromised maintainer, which is the dominant delivery mechanism in practice.
Scanners cannot see what is not yet known
Software composition analysis matches a resolved dependency tree against vulnerability databases. This approach works well for its intended purpose: identifying known-vulnerable versions of legitimate packages.
A newly published malicious package is not in any database. It has no CVE because nobody has reported it. It will acquire one hours or days later, after somebody notices. The scan that runs during the window returns clean, and it is correct, because it answers a different question than the one you needed answered.
A second, less obvious but more consequential gap exists. Many enforcement policies distinguish between direct and transitive dependencies – blocking builds on findings in packages the project declares, while merely reporting findings deeper in the tree.
The reasoning is operational rather than negligent. Direct dependencies are actionable; a team can upgrade them. Transitive findings often cannot be resolved by the team that owns the pipeline, and blocking on them produces builds that cannot be unblocked.
But transitive dependencies are not a majority of the graph. They are almost all of it. GitHub's measurement across public repositories put the median JavaScript project at ten direct dependencies and, where a lockfile was present, 683 transitive ones [5]. Registry-level academic measurement is less extreme but points the same way: half of all dependent npm packages carry at least twenty-one transitive dependencies, against a median of two direct [6]. Both figures predate the current wave, and the trend since has been upward.
And supply chain compromise is, structurally, a transitive phenomenon. The attacker does not compromise the package you chose. They compromise something three levels below it.
The result is a gate that is blind in exactly the region where the threat lives, yet reports green.
A passing first-stage scan means "nothing blocking was found." It does not mean "nothing is there."
Pinning helps less than it appears to
The standard advice is to pin dependencies to immutable versions or digests. It is sound advice, though it is frequently oversold.
Recent empirical work examining counterfactual scenarios across large dependency graphs found that pinning direct dependencies alone does not meaningfully protect deep graphs and introduces a measurable cost: pinned projects fall behind on security updates, creating a second exposure while addressing the first [4]. Pinning without complete lockfile resolution leaves transitive dependencies floating – which is where the attack occurs.
The qualified position: lockfile-based full-graph resolution with verified integrity hashes, combined with an update process that does not sit still for months. Not digest pinning at the top level as a standalone control.
What follows from this
Every one of these controls is worth deploying. That is not the argument.
The argument is that each of them moves probability and cost, and none of them closes the category. An organisation can adopt all of them, correctly, and still be breached by a sufficiently well-timed compromise of a sufficiently deep dependency.
The 2026 evidence supports this directly. The organisations on that leaked credential list have these controls. They were breached anyway.
So the design question changes. Not "how do we prevent this?" – that has an answer, and the answer has a ceiling. The question becomes: when this happens, how much does the attacker gain, how far can they reach, and how long until someone knows?
Those three questions have much better answers than the first one.
3. Designing for Compromise
The framing that follows treats each control by where it cuts the chain, and – because this is the part that determines whether anything actually gets built – by who has to own it. A large organisation's supply chain posture is usually not limited by knowledge. It is limited by the fact that the team who understands the problem does not own the infrastructure where the fix resides.
Blocking execution
The highest-leverage single change remains disabling package manager lifecycle scripts during installation. For npm and yarn, use --ignore-scripts; in the project configuration, set ignore-scripts=true; and in other ecosystems, use --no-build or equivalent constraints. Scripts that are genuinely required – mostly native module compilation – are then invoked explicitly, by name, for the specific packages that need them.
This removes the primary execution path used by every worm in this family. It takes a day of work to identify which packages legitimately require build steps, and it occasionally breaks something in a way that requires a real fix rather than a workaround.
An application team can do this alone. It requires no platform change, no procurement, and no other team's roadmap. Of all the controls in this article, it has the best ratio of protection to friction, and it should be the default in every CI configuration unless there is a specific documented reason not to.
Its ceiling is important and is covered in section 5: lifecycle scripts are one execution path among several, and the others are expanding.
Bounding what executing code can read
Assume execution succeeded anyway. The next question is what the environment contains at that moment.
In most pipelines, the answer is everything. Variables are defined at the project or group level, injected into every job, and available from the first instruction of the first script. A job that clones a repository and installs dependencies has, at that moment, access to deployment credentials it will never use, registry publishing tokens for artefacts it will never produce, and cloud keys for environments it will never touch.
Three changes narrow this, in order of increasing difficulty.
Job-level variable scoping. Most CI platforms support restricting variables to specific environments, protected branches, or named jobs. Applied correctly, the install job has no deployment credentials, as it does not deploy. This is largely a configuration exercise that an application team can drive, though it requires cooperation from whoever administers group-level variables.
Short-lived, federated credentials. OIDC-based workload identity replaces long-lived static secrets with tokens minted at job start, scoped to a specific repository, branch, and environment, and valid for an hour. The compromise of such a token is time- and reach-bounded.
This is a substantial improvement, but it is not a solution. The token exists in the job environment while the job runs, which is exactly when the malicious code is executing. The change is that the attacker gets an hour of scoped access rather than a permanent key to production. That is a large difference in blast radius and no difference at all in whether the compromise succeeded.
One common misconfiguration completely undermines it: trust policies with wildcard subject claims permit any repository in an organisation, including untrusted pull requests from forks, to mint production credentials. A federated identity configuration is only as narrow as its subject claim.
Separating the untrusted stage. The strongest approach fetches credentials at the point of use from a secrets manager, authenticated by the workload identity, rather than pre-injecting them as environment variables. On its own, this does less than it appears to. Removing the secret value does not remove the ability to fetch it: the service account token is at a fixed path, the OIDC request token and endpoint are environment variables, and the instance metadata service responds at a known address. Malicious code executing in that job authenticates to the vault exactly as the build step would and retrieves the same secret.
What makes it work is separation. Dependency installation and the build run in a job with no trust relationship to anything – no federated identity, no mounted service account token, no metadata access – and pass their output to a credentialed stage that publishes or deploys it. The containment holds because that stage moves the artefact rather than executing it, so a poisoned build product travels no further than storage.
This is architecturally correct but expensive to retrofit across an established estate, which is why almost nobody has done so.
Bounding what executing code can do
Runner privilege posture is the control most likely to be set incorrectly by accident, because nobody chose it.
A CI job container typically runs as root, retains a default capability set, has no seccomp profile, is not confined by mandatory access control, has a writable root filesystem, and has a Kubernetes service account token mounted and readable. No one made any of these decisions. It is what happens when a base image does not specify a user and a pod specification does not specify a security context.
The consequence is evident in a way that makes the point better than any argument: two jobs in the same pipeline, on the same runner, with the same pod specification, will have completely different privilege postures if their images differ – because posture follows whatever the image author configured. A tool image whose upstream maintainers added a USER directive runs unprivileged. The general-purpose language image that executes untrusted dependency code runs as root.
The hardened job is the one that did not need hardening. The exposed job is the one that runs the attacker's code.
Fixing this means implementing a pod security baseline: non-root by default, capabilities dropped, seccomp profile applied, read-only root filesystem where feasible, and service account token projection disabled unless a job requires cluster access.
This is platform-team territory. An application team can request it but cannot implement it. This is why it remains undone in many organisations, even though somebody has already written the ticket.
Preventing the exfiltration
Egress control is the only measure in this article that breaks the chain rather than narrowing it. Harvested credentials are worthless to an attacker who cannot transmit them.
Runners typically have unrestricted outbound access because restricting it is operationally painful and nobody wants to be the person who broke the build at eleven at night. The correct posture is an allowlist: the package registry, the internal artefact store, the container registry, the observability endpoints, and nothing else. This is implemented as network policy at the cluster layer, or as an egress proxy through which all outbound traffic must pass.
Two limits deserve honesty.
TLS interception is not egress control. A decrypting proxy sees and logs traffic to arbitrary destinations; it does not block it. Any host with a valid public certificate remains reachable, and the payload never needs to defeat the proxy to use it. Nor does disabling client-side certificate validation help the attacker – NODE_TLS_REJECT_UNAUTHORIZED=0 and its equivalents make interception silent rather than bypassing it, because the client simply stops objecting to the proxy's certificate. Interception without a destination allowlist is visibility, not enforcement. It is worth noting that TLS interception does not break package integrity verification – lockfile hashes are computed over the payload after decryption – so the common objection to deploying it does not hold. But neither does the common belief that deploying it establishes a control.
DNS is an exfiltration channel. If a payload can resolve arbitrary domains, it can encode data into subdomain queries at low bandwidth and high reliability. Egress allowlisting that permits unrestricted DNS resolution has left a door open.
An allowlist also decays. Every urgent build failure creates pressure to add an entry, and entries are added under pressure and never reviewed. A quarterly review of the allowlist is not bureaucratic overhead; it is the only thing preventing the control from becoming a list of every host anyone has ever needed.
What ephemerality does and does not do
Ephemeral runners – a fresh container per job, destroyed on completion – are widely treated as a containment control. The reasoning is that malware cannot persist where there is no persistent host.
That reasoning is correct, and its scope is narrow.
Harvesting, exfiltration, and onward propagation using stolen tokens all complete within the job's lifetime. The container's subsequent destruction changes nothing about any of it. What ephemerality removes is post-job persistence: the background daemon, the modified shell profile, the scheduled task. Real, but not the part of the attack that hurt you.
Ephemerality also carries a cost that is rarely counted. There is no host to image, no filesystem to examine, no process list to capture. When an alert fires days later, the environment that produced it no longer exists. Forensic reconstruction depends entirely on what was streamed from the runner while it ran – which, in most pipelines, is job logs and nothing else.
"Our runners are ephemeral" belongs in the same category of comfortable statements as "our scans are green." Both are true, yet neither means what it is intended to mean.
What outlives the pod
State that survives job destruction is where persistence actually lives in a CI estate.
Registry proxy caches. This is the one most organisations have overlooked. Enterprise mirrors (Artifactory remote repositories, Nexus proxy repositories, CodeArtifact, Azure Artifacts, Cloudsmith, Verdaccio) cache upstream artefacts on first fetch and serve them from local storage thereafter.
When the upstream registry unpublishes a malicious version, the mirror is unaware. The bytes remain in local storage, and the mirror continues serving them to every build that resolves that version. The public ecosystem has remediated; yours has not.
Each platform requires an explicit administrative purge operation. This includes deleting the artefact, clearing the cache on the remote repository path, and calling the delete-package-version API. None of this happens automatically, and none of it is part of a standard incident response runbook.
The check is trivial and worth running today:
# Compare what upstream offers against what your mirror serves
curl -s https://registry.npmjs.org/<package> | jq -r '.versions | keys[]' | sort > /tmp/upstream.txt
curl -s https://<your-mirror>/<package> | jq -r '.versions | keys[]' | sort > /tmp/mirror.txt
comm -13 /tmp/upstream.txt /tmp/mirror.txt # versions your mirror has that upstream does not
Anything in that output is a version the public ecosystem has removed, yet your infrastructure still distributes it. Most of it will be harmless. Some of it may not be.
Build caches. Shared cache keys evaluated across branches allow a job on an untrusted branch to write content that a job on a protected branch later restores and executes. Cache keys should be scoped so that untrusted contexts cannot write to keys that trusted contexts read.
Shared runner volumes. Persistent build directories, shared dependency caches, and shared container layer caches on a node. Where these are shared across jobs and teams, they create a lateral path between tenants that no network policy addresses.
That last point generalises to something worth stating directly. In a shared runner pool, the trust boundary is the node, not the repository. A pipeline compromise on infrastructure shared between teams is not scoped to the team that triggered it, and an incident response that asks only "was our project affected?" answers an organisational question in technical terms. The blast radius follows the infrastructure. It is very often investigated along the org chart.
4. Winning on Time
Everything above narrows the damage. None of it guarantees zero damage. So the remaining variable – the one that separates a contained incident from a catastrophic one – is the elapsed time between compromise and knowledge.
The LiteLLM credentials were harvested in March and became public knowledge in August. Over the five months that followed, affected organisations operated normally, with the stolen credentials in circulation and no reason to believe anything was wrong.
Time-to-know is a design parameter. It can be engineered down.
Signals available inside a running job
A build job has an unusually well-defined behavioural profile. It clones, resolves, downloads, compiles, tests, packages, and uploads. Almost everything outside that profile warrants an alert, and the false-positive rate is far lower than in production environments precisely because builds are repetitive.
Signals worth instrumenting:
- A shell or interpreter process spawned by a package manager lifecycle script
- Bulk reads of the process environment by anything that is not the build tool
- Outbound connections to hosts outside the allowlist, including resolution attempts
- Reads of credential paths that a build has no reason to touch:
~/.ssh,~/.npmrc,~/.aws,~/.docker/config.json,~/.git-credentials - Writes to those same paths, indicating persistence attempts rather than theft
- Process execution from within a dependency directory
Runtime sensors capable of collecting these exist at the kernel layer and can be deployed to a runner node pool. That is platform work. Where it is not available, a meaningful subset can be approximated within the job itself – checksum credential files before and after installation, and fail loudly on any change.
Canary credentials
This is the highest-value, lowest-cost detection control available in CI, yet it is dramatically under-deployed relative to its capabilities.
The technique: inject a credential into the pipeline environment that no legitimate process will ever use, named so that automated harvesting will collect it – AWS_SECRET_ACCESS_KEY_BACKUP, NPM_PUBLISH_TOKEN_LEGACY, or something plausible. The credential has zero permissions. Its only property is that any attempt to authenticate with it raises an alert, including the source address and timestamp.
The signal quality is exceptional. Legitimate builds never touch it, so there is no baseline to tune against and no false alerts to contend with. A single alert is not a probability – it is a confirmed harvest, with a timestamp that bounds the compromise window and a source that identifies the collector.
Both hosted canary token services and zero-permission cloud credentials work. The cost is measured in minutes.
The limits are real and should be stated whenever you recommend the technique. Sophisticated payloads enumerate environment variables to identify decoy patterns before exfiltrating. Payloads that extract federated tokens from process memory, rather than reading environment variables, will bypass an environment-variable canary entirely. And a canary detects harvesting, not execution – a payload that establishes persistence without touching credentials produces no signal.
It catches automated credential stealers, which make up the overwhelming majority of this attack class. It does not catch a targeted operator who is looking for it.
Stream the evidence out
Because ephemeral runners destroy their own evidence, the only forensic record is what left the runner while it was still running. Ship job logs, process telemetry, and network flow records to durable storage as they are produced, rather than collecting them afterwards – by then there is nothing to collect.
The test is simple. If an alert fires about a build that ran two weeks ago, what can be reconstructed? In many organisations, the honest answer is a text log with secrets masked out.
5. The Node You Do Not Control
Every control described so far applies to infrastructure. There is a machine in the loop that none of them reach.
The developer workstation is a richer target than any CI runner and is defended less well. It holds long-lived credentials that CI never sees – personal access tokens, cloud profiles, SSH keys with passphrases cached in an agent, registry credentials, and session cookies for every internal system. It persists across reboots, making it a viable host for persistence mechanisms that ephemeral runners defeat. It is rarely subject to the network egress policy that governs the build estate. And it belongs to somebody who has to be able to work.
A perfectly hardened pipeline sits downstream.
Opening a repository can execute code
The single most under-appreciated fact in this space: on a modern developer machine, code executes on clone and open, without any package manager involvement.
--ignore-scripts addresses package lifecycle hooks. It does not address anything below.
Editor task automation. The VS Code task schema supports a runOn property with the value folderOpen, which executes a defined command in a background shell when the workspace is opened [9]. No user action beyond opening the folder is required.
Editor settings point to repository binaries. Workspace settings can override tool paths – the interpreter, the formatter, the language server, and the alternate toolchain path. When set to a relative path inside the repository, the editor executes attacker-supplied binaries when it initialises background services. This one is particularly quiet, because nothing in the configuration looks like a command.
Development container lifecycle hooks. postCreateCommand, updateContentCommand, postStartCommand and their siblings execute arbitrary shell commands when an environment is built or started. This is the feature's documented purpose [10].
Committed Git hooks. A repository configured with core.hooksPath pointing to a tracked directory causes local scripts to run on routine operations – checkout, commit, merge.
JetBrains run configurations. XML files in the project directory that define startup tasks and external tools.
The AI agent surface
This is the newest execution surface and the fastest-growing, and it deserves to be treated as a first-class risk rather than a footnote.
Repository-resident agent configuration – including hook definitions, session initialisation files, and custom sub-agent and skill declarations – specifies terminal commands that run when an agent begins a session or ingests repository context. Model Context Protocol manifests declare local server processes, invoked as node, python, or npx commands, that the agent's tool runner starts when the workspace opens.
Cloning a repository and pointing an agent at it can therefore execute attacker-controlled commands before installing any dependencies.
The same applies within the pipeline. Automated review agents, remediation bots, and agentic CI steps parse repository-resident files, and indirect prompt injection embedded in those files can redirect the agent's tool-calling using the privileges of the runner's token.
Workspace trust mechanisms exist and restrict some of this. Their practical effectiveness is limited by two factors: developers grant trust reflexively because they encounter the prompt several times a day, and automated contexts (CI runners, headless agents) are non-interactive and bypass the prompt entirely.
Meanwhile, mainstream dependency scanning and static analysis tools read manifests and source code. They do not, as a rule, read editor configuration, container hooks, or agent manifests. This surface is not merely unprotected; in most organisations, it is unmonitored.
It is cheap to look. Section 7 includes the sweep.
The trap in the standard response
Recent payloads in this family include a mechanism that inverts conventional incident response and targets persistent hosts.
The pattern: alongside credential theft, the payload installs a lightweight watcher – a user-level service or launch agent, depending on the platform. The watcher holds a copy of the stolen token and polls an upstream API with it at short intervals. When the API begins rejecting the token, the watcher concludes that the theft has been discovered and executes a locally stored handler command. In observed cases, the handler has been destructive: recursively deleting the user's home directory, taking source code, local state, and forensic evidence with it.
Credential revocation is the trigger.
This is elegant from the attacker's perspective. Revocation is the first step in every incident response runbook. It requires no command-and-control infrastructure, survives domain takedowns and sinkholing, and fires reliably because everybody does it.
The response ordering on affected hosts has to change:
- Isolate the host at the network layer first. A watcher that cannot reach the validation endpoint cannot observe the revocation.
- Remove the persistence mechanism. Enumerate user-level services, launch agents, scheduled tasks and shell profile modifications.
- Clean the artefacts – dependency trees, package caches, cloned repositories.
- Then revoke and rotate, from a different machine.
- Rebuild the host. Do not repair it.
One qualification for step 1. The trigger observed in this family is an authentication rejection – a completed request returning 401 or 403 – not a failure to connect. A watcher that fired on timeouts would destroy its host every time the machine lost connectivity, which is why implementations distinguish the two. That is observed behaviour rather than a guarantee, so isolation should be followed immediately by persistence enumeration rather than treated as a stable state.
A specific hazard follows: automated secret scanning that revokes leaked credentials on detection, without any host containment step, will fire the trigger before a responder is involved. Automated revocation is a good control, but this class of finding now needs a manual gate.
This is a workstation pattern, not a pipeline one. On genuinely ephemeral CI runners, no watcher survives job completion – it never receives a second poll. The exceptions are worth noting: an attacker who deliberately extends job execution, a payload that writes to a shared volume, or a runner with a mounted host socket. Otherwise, ephemerality defeats this mechanism, which is one of the few places where it delivers exactly what it promises.
The last mile is a notification path
Detection on a developer machine produces value only if the developer learns about it quickly and in a way that does not cause them to reflexively do the destructive thing.
Three things must be in place before an alert is useful:
- A path to the individual. Not a ticket in a security queue reviewed on Tuesday. Contact the person whose machine it is directly within minutes.
- A first safe instruction. Disconnect from the network. Do not delete anything, do not revoke anything, do not reinstall. Wait.
- A response that does not punish reporting. Developers who expect blame report late or not at all, and late reporting is the difference between a compromised host and an entire estate.
The human factor is not removable, and treating it as a failure mode to be engineered away produces worse outcomes than accepting it. Developers will open repositories. They will grant workspace trust. They will run agents on unfamiliar code, because that is increasingly the nature of the work.
The engineering problem is not preventing that. It is building the channel that captures it and tells someone, fast.
6. Rotation Is Not Remediation
Return to the two facts from section 1.
The LiteLLM compromise was made possible by an automation token that had been rotated but not fully revoked. After the compromise was disclosed, tested credentials still worked at an organisation that had announced it had rotated them.
The same failure at both ends of the same campaign. It is the most instructive detail available, yet it is almost entirely absent from the coverage.
"Rotation" describes three separate operations that are routinely performed together.
Value replacement. A new secret string is generated and distributed. The old string no longer authenticates. This is what almost everyone means by rotation, and it is the only one most organisations actually perform.
Session and token invalidation. Credentials derived from the old secret – active sessions, refresh tokens, issued bearer tokens, and temporary cloud credentials minted before the change – remain valid until they expire on their own schedule, unless explicitly revoked via a separate mechanism. Replacing a value does not terminate sessions established with it. This is how a rotated key continues to work, as Beaumont's testing demonstrated.
Exposure surface. Whether the same credentials remain readable by the same code, in the same place, under the same conditions as before. A rotation produces a new value and injects it into the identical pipeline, at the identical stage, available to the identical untrusted dependency installation, with the identical outbound network access.
The attacker's execution path remains unchanged. Only the string differs.
This is what makes mass rotation such a satisfying yet misleading incident response. It is visible, measurable, and produces a completion report, yet it addresses the least important of the three dimensions. An organisation can rotate every credential in its estate and be in precisely the same position the following morning, with fresh secrets sitting in the same exposed environment.
There is a gap in how this is governed, though not the one you might expect. Guidance on CI/CD credential hygiene is well developed [8]. OWASP's CI/CD Top 10 treats insufficient credential hygiene as a top-ten risk in its own right, prefers temporary credentials to static ones, calls for a rotation procedure and stale-credential detection where static credentials must persist, and recommends re-scoping pipeline secrets to the minimum set of pipelines and steps that require them [7]. That last item is an exposure-surface control, and a good one.
What is missing is the join. Rotation is specified as a schedule; scope reduction is specified as a posture; and nothing asks whether the surface was still the same on the far side of a rotation carried out in response to an incident. Compliance is demonstrated by showing that values changed. Whether the new values sit in the same place, are readable by the same code, and are under the same conditions is a separate question, and it is not on anyone's checklist.
The verification that should follow every rotation:
- Confirm the old value is rejected – authenticate with it and expect a failure.
- Confirm that derived sessions are terminated – query the session and token management APIs for credentials issued before the rotation timestamp.
- Confirm the exposure surface has changed – re-run the measurement in section 7 and compare the credential count and privilege posture with the pre-incident baseline.
If the third number is identical, the incident is not closed and is repeatable.
7. Measure Your Own Exposure
Four tests. All read-only, all reversible, and all removed afterwards. They will not tell you that you are safe. They will tell you where you are standing, which is more useful to know and the only one available.
The first two measure how much would leak. The second two measure whether you would find out.
Test 1 – Credential inventory
The question: what does code executed during dependency installation have access to before any gate has run?
Add a job that performs no project setup. No authentication step, no checkout requirements, no installation. Override the pipeline's default preamble so nothing runs before it.
audit-secret-scope:
stage: .pre
image: <the exact image your install job uses>
before_script: [] # override inherited setup
script:
- printenv | cut -d= -f1 | sort
- echo "total variables: $(printenv | wc -l)"
The equivalent on a workflow-based platform is a job with no checkout step, running env | cut -d= -f1 | sort.
Names only. No values, not even truncated ones. Printing the first characters of a token may seem like a safe compromise, but it is not – a prefix narrows a search across a leaked corpus considerably, and the output of this job lands in a log that is more widely readable than the variables themselves. The count and the names are the entire finding.
Read the output as a list of what a single malicious postinstall can obtain for free. Most teams running this for the first time are surprised twice: by the total and by how many entries are unrelated to the job's purpose. Treat the count as a floor, not a total. Variables scoped to protected branches, specific environments, or later stages will not appear in a job running early on an ordinary branch, and those are often the most valuable. Run the same probe from a protected-branch pipeline to see the upper bound.
Test 2 – Runner privilege posture
The question: if the code executes, what can it do?
echo "uid=$(id -u) gid=$(id -g) groups=$(id -G)"
grep -E '^(CapEff|CapBnd|NoNewPrivs|Seccomp):' /proc/self/status
capsh --decode=$(grep '^CapEff:' /proc/self/status | awk '{print $2}') 2>/dev/null
cat /proc/self/attr/current 2>/dev/null || echo "MAC: unconfined"
touch /.probe 2>/dev/null && echo "root fs: writable" && rm -f /.probe || echo "root fs: read-only"
test -r /var/run/secrets/kubernetes.io/serviceaccount/token \
&& echo "cluster service account token: mounted and readable"
Nothing here attempts escalation or escape. It reads its own process status.
Pin this job to the image where the dependency installation actually occurs, not to the pipeline template's default image. The privilege posture follows the image, and running the probe under the default yields a reassuring result for a container that never executes untrusted code. This is the single most common reason this test returns an incorrect answer.
Run it against two or three different job images within the same pipeline and compare. The divergence is the finding: it shows that the posture was never configured, only inherited.
Test 3 – Plant a canary
The first two tests are diagnostic. This one is a control, and it stays.
In the pipeline. Create a credential with no permissions – an IAM user with an empty policy or a hosted canary token. Configure alerting for any authentication attempt. Inject it as a pipeline variable with a name a harvester would collect:
AWS_SECRET_ACCESS_KEY_BACKUP
NPM_TOKEN_LEGACY
DOCKER_REGISTRY_PASSWORD_OLD
No legitimate process references it. Any alert is a confirmed harvest, with a timestamp and a source address.
On the workstation. The same idea as where the richer credentials live. A decoy credentials file in the expected location – an extra profile in the cloud credentials file, a plausible entry in a registry configuration – with alerting on use. This is the detection path for section 5, where a hardened pipeline provides no coverage at all.
State the limits alongside the deployment so nobody mistakes them for a boundary: a payload that enumerates decoy patterns before exfiltrating, or that extracts federated tokens from memory rather than reading environment variables, will pass this. It catches automated credential stealers. It is not a defence against a targeted operator.
Cost: under an hour, and it is the best hour available in this article.
Test 4 – Sweep the non-install execution surface
The question: what in these repositories runs when someone opens them?
# Editor task automation and container lifecycle hooks
git grep -lE '"runOn"[[:space:]]*:[[:space:]]*"folderOpen"|postCreateCommand|updateContentCommand|postStartCommand|onCreateCommand'
# Committed git hooks
git grep -lE 'core\.hooksPath'
find . -path ./node_modules -prune -o -type d -name '.githooks' -print
# Tool paths redirected into the repository
git grep -nE '(defaultInterpreterPath|alternateTools|\.formatterPath|languageServerPath)'
# Agent and MCP configuration
find . -path ./node_modules -prune -o \
\( -name 'mcp.json' -o -path '*/.claude/*' -o -path '*/.cursor/*' \
-o -path '*/.idea/runConfigurations/*' \) -print
Run it across the estate rather than in a single repository. The finding is rarely a malicious file. The finding is that these paths exist, are committed, execute on open, receive no review attention because they are not application code, and are not read by any of the scanners currently in the pipeline.
Add them to the paths that require review on change. That takes one line in a code owners file and closes a gap that --ignore-scripts does not address.
Reading the results
A clean result from any of these is not an all-clear. Test 1 returning a small number indicates the blast radius is small today. Test 4 returning nothing indicates no attacker has used that path in these repositories yet.
They establish a baseline. The value arrives later – when something happens, the question is whether the exposure surface has changed, and there is a prior measurement to compare against.
Section 6 argued that a rotation that does not change the numbers in tests 1 and 2 has not remediated anything. That argument requires the numbers to exist. Take them now.
8. What Good Looks Like
The instinct in supply chain security is to build a wall. The evidence from the past two years shows that every wall has a height above which the attacker is already inside it, and that building higher does not move that line.
The better model is a shock absorber. It does not prevent the impact. It determines how much of the impact reaches what matters.
Three properties, in order of value:
Raise the cost of the easy attack. Disable lifecycle scripts, enforce lockfiles, restrict egress, and run runners unprivileged. None of this stops a determined adversary. All of it removes the automated commodity attack, which makes up the overwhelming majority of this traffic. Worms take the cheapest path available, and the point is not to be that path.
Bound what a successful attack can reach. Scoped, short-lived, per-job credentials. A node boundary that is actually a boundary. Caches that untrusted contexts cannot poison. The difference between an incident and a catastrophe is almost never whether the attacker got in. It is what was adjacent to them when they did.
Shorten the gap between compromise and detection. Canaries, runtime signals, evidence streamed from ephemeral infrastructure, and a notification path to a human that works in minutes. Five months of silence is not a detection failure. It is a design decision that nobody made deliberately.
And beneath it all, the discipline that makes the difference between a security programme and a compliance artefact: measure the exposure rather than assuming it, and measure it again after every change that was supposed to have reduced it.
The organisations on that leaked credential list did not fail to prevent an unpreventable attack.
They failed to notice it.
References
- Ars Technica - Terabytes of credentials leaked in massive supply-chain attack (August 2026). https://arstechnica.com/security/2026/08/terabytes-of-credentials-leaked-in-massive-supply-chain-attack/
- TechRadar Pro - Massive supply-chain attack sees terabytes of data belonging to some of the world's biggest and most sensitive organizations leaked online (August 2026). https://www.techradar.com/pro/security/massive-supply-chain-attack-sees-terabytes-of-data-belonging-to-some-of-the-worlds-biggest-and-most-sensitive-organizations-leaked-online
- SLSA - Threats & mitigations (specification v1.2). https://slsa.dev/spec/v1.2/threats
- He, H., Vasilescu, B., & Kästner, C. - Pinning Is Futile: You Need More Than Local Dependency Versioning to Defend against Supply Chain Attacks (FSE 2025, arXiv:2502.06662). https://arxiv.org/abs/2502.06662
- GitHub - The 2020 State of the Octoverse: Securing the World's Software. https://arxiv.org/pdf/2110.10246
- Decan, A., Mens, T., & Grosjean, P. - An Empirical Comparison of Dependency Network Evolution in Seven Software Packaging Ecosystems (arXiv:1710.04936). https://arxiv.org/pdf/1710.04936
- OWASP - CICD-SEC-6: Insufficient Credential Hygiene. https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-06-Insufficient-Credential-Hygiene
- NIST SP 800-204D - Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines. https://csrc.nist.gov/pubs/sp/800/204/d/final
- Visual Studio Code - Tasks schema reference. https://code.visualstudio.com/docs/reference/tasks-appendix
- Development Containers Specification - Development container specification. https://containers.dev/implementors/spec/
Citation set is provisional – the SLSA threat-model characterisation, the campaign figures, and the registry cache behaviours still require primary-source verification before publication.