Skip to main content

Log-Based Distributed Tracing (Trace-Id Propagation Through Shared Libraries)

Metadata

1. Executive Summary​

This architecture correlates a request across every microservice it touches using one HTTP header, one field in a structured log line, and the log aggregation pipeline that container platforms already run.

A request carries a single ID in x-trace-id. Every service writes that ID as traceId on every JSON log line it emits to stdout. The container runtime captures stdout, a node-level shipper forwards it, and the log store indexes the pod labels — so the full chain of one request across a dozen services is a single query:

{namespace=~"platform"} |= "01M0J6EYRY4TFEPR9PHJZ1QHPF"

The ID is delivered through the shared logger and HTTP-client libraries the services already depend on, and made ambient through the framework's request-scoped dependency injection. Adoption per service is two module imports (three for a service that makes outbound calls); after that, no application code mentions a trace ID. An automatically emitted request/response record pair turns the correlated lines into a call graph with status codes and in-process durations.

The pattern gives up exact call topology. There are no span IDs, so caller → callee edges are inferred from the forwarded User-Agent and timestamps rather than known; concurrent identical calls are ambiguous, and sub-request timing and datastore internals are out of scope. In exchange, the application-side mechanism is a few hundred lines, nothing runs beside the process, every request is in the logs with no sampling, and a log-store outage costs search rather than the request.

Provenance. The author designed and implemented this pattern in 2022 as the owner of the shared libraries on an enterprise platform built as microservices from its first day. It has been in production since then, on a platform of 200+ microservices (with 80+ NestJS services under the author's team's responsibility), where trace ID propagation currently covers more than 100 services across multiple teams. At least two further teams adopted it — one by taking the libraries, one by reimplementing the contract in a Python service — and one inbound integration adopts a partner organisation's event id as the trace id, so an issue found on one side can be searched on the other with no shared tooling. The linked repository is a sanitised reconstruction of that design, not the production code; its design document states what it does and does not exercise.

2. Applicability Criteria​

Use this pattern when​

  • Shared libraries already reach every service. A common logger and HTTP client — or the willingness to introduce them — turn the rollout into a library release rather than a per-service project. This is the decisive precondition: the pattern is cheap because the delivery vehicle already exists.
  • A log pipeline already exists. Container stdout → node-level shipper → a store that can filter lines by content (Loki, Elasticsearch/OpenSearch, CloudWatch Logs, and similar). The pattern adds one field and one query. It adds no collection infrastructure.
  • A tracing backend is not available, or not yet — budget, procurement, or an approval cycle longer than the problem can wait — and request correlation is the part that cannot wait.
  • Investigations are mostly "what happened to this request across services", and the people running them are not only engineers. One ID and one query are enough for an analyst or support tier to operate without an engineer in the room.
  • Every request must be findable. The investigations in question cannot tolerate sampling.

Do NOT use this pattern when​

  • Per-operation spans are required — sub-request timing, database, cache or message-broker internals, exact parent/child edges under concurrency. Span-based tracing (OpenTelemetry and comparable stacks) provides that, and this pattern does not. The two are independent and can coexist; this pattern never touches the other.
  • Log retention is shorter than the investigation window. Correlation lives in the logs and dies with them.
  • Log volume is already the binding constraint. The pattern adds two lines per request per service and two per outbound call.
  • The chain is dominated by hops you do not control and that will not forward a header. The trace ends where the header stops.

3. Architecture Overview​

3.1 The contract​

Headerx-trace-id — lowercase on the wire, read case-insensitively
Log fieldtraceId — omitted when absent, never null (a null field would match every untraced line in a substring search)
Valueany non-empty string. A ULID at HTTP entry points by convention; derived ids for fan-out work; a partner's identifier where one arrives. Never validated — only truncated at 128 characters
Responseechoed back to the caller on every response, including errors
Ownershipone package exports the header name, the field name and the separator; the logger and the HTTP client depend on it and not on each other

The header name is an exported constant, not a string literal repeated at call sites. A service forwarding x-traceid, or one sub-request sending X-Trace-Id while everything else sends lowercase, breaks correlation for exactly one hop — and every individual log line still looks correct, which is what makes that gap hard to find. Owning the name in one place is what makes the failure impossible rather than unlikely.

3.2 Four decisions, one of them framework-shaped​

The whole pattern is four decisions. Three are wire formats and a header rule; only the third depends on the framework.

  1. One header name, owned by one module. Any language.
  2. One structured log line per record, with the id as a field, written to stdout. Any logger that emits JSON.
  3. A way to make the id ambient for the duration of a request, so application code never passes it. In NestJS this is request-scoped dependency injection (§4.2). Elsewhere it is AsyncLocalStorage, contextvars, MDC, or context.Context.
  4. An outbound client that re-attaches the header. One line wherever services already share an HTTP client.

A service in another language, owned by another team, joins the same traces as soon as it does 1, 2, and 4. That is the practical reason the contract is kept this small.

3.3 The flow​

The property that matters in the second diagram: everything left of the log store is not new. Container stdout capture and a node-level shipper exist whether or not anything is correlated. The pattern contributes one field in the JSON and the query on the right.

3.4 One string, four places​

Attribution by label and attribution by line body have to agree, so one value is used everywhere a service names itself:

container name  =  DEPLOYMENT_NAME  =  serviceName (log field)  =  User-Agent (outbound)

In the reference implementation the deployment manifest sets DEPLOYMENT_NAME equal to the container name, the log field and the User-Agent are both read from that one variable, and the HTTP client refuses to boot without a userAgent. The reason is in §4.4: User-Agent is the only edge information a trace has.

4. Implementation Details​

The reference implementation is three packages. The split is the design: the logger and the HTTP client both depend on the contract package and not on each other.

PackageRoleCode lines
@tw/tracingThe contract: header name, id format, the seed88
@tw/loggerThe id on every line; the request/response pair587
@tw/http-connectorCarries the id to the next service386

Adoption per service, in app.module.ts:

@Module({
  imports: [
    TracingModule.forRoot(),   // seeds x-trace-id in middleware — before guards and interceptors
    LoggerModule.forRoot(),    // id on every log line; registers the request/response interceptor
    HttpConnectionModule.forRoot({ userAgent: SERVICE_NAME, logger: { /* … */ } }), // carries it onward
  ],
})
export class AppModule {}

A leaf service that calls nobody omits the third import. It still seeds an id, because requests reach it directly — a scheduler, a health prober, an engineer with curl — and those are the calls nobody thinks to trace until they are the ones failing.

4.1 Seed in middleware, before guards and interceptors​

The seed runs as framework middleware, applied to every route. In NestJS, the request lifecycle is middleware → guards → interceptors → pipes → handler, and two things depend on the ID existing before interceptors run:

  • The request-logging interceptor is registered through APP_INTERCEPTOR, which the framework pushes onto the global list during NestFactory.create() — before anything added later by app.useGlobalInterceptors(). A seed registered as a global interceptor therefore runs second, and the request/response pair for a request that arrived without an ID is logged without one. That request is the first hop of every trace.
  • Guards run before interceptors at all. A request rejected by an auth guard never reaches an interceptor, so an interceptor-based seed leaves every 401 and 403 untraceable.

Middleware is the only stage that runs before both. The symptom of getting this wrong is subtle: requests that arrive with an id work fine, so the only visible effect is traces that begin one service too late.

The seed itself, ensureTraceId:

export function ensureTraceId(req: RequestLike, maxLength: number = MAX_TRACE_ID_LENGTH): string {
  const existing = getTraceId(req);
  if (existing !== undefined && existing.length <= maxLength) return existing;

  const traceId = existing === undefined ? newTraceId() : existing.slice(0, maxLength);
  if (!req.headers) req.headers = {};

  for (const key of Object.keys(req.headers)) {
    if (key.toLowerCase() === TRACE_ID_HEADER) delete req.headers[key];
  }

  req.headers[TRACE_ID_HEADER] = traceId;
  return traceId;
}

Three properties of this function carry the design:

  • It mutates req.headers in place. Every request-scoped provider downstream reads the same object, so seeding it once here is what makes the ID available everywhere without being passed as an argument.
  • It inherits rather than overwrites. The first service to see a request creates the ID; every service after it adopts one. That is what makes the trace distributed.
  • It truncates and does nothing else. The 128-character cap is not validation — a derived ID is legitimate and unpredictable. It caps one thing: an ID the platform didn't mint is copied onto every log line of every service in the chain, so its length multiplies across the whole fan-out. Truncation is idempotent, and the cap is the same everywhere, so a long ID is shortened once at the first hop and every later service inherits the identical value.

A rejected request still ties to its caller. The seeding middleware has run, so the exception filter logs the rejection under the id — a warn line with no request/response pair is the complete, correct trace of an unauthenticated request:

{"timestamp":"2026-09-09T10:24:10.486Z","level":"warn","serviceName":"products-service","context":"GET /v1/products/1","traceId":"01M22V4KQN0Y54EMSD18FQ2DCP","message":"401 UNAUTHENTICATED: No token provided"}

4.2 Make the id ambient through request-scoped DI​

This is the centrepiece, and the mechanism is often misdescribed. There is no AsyncLocalStorage, no continuation-local storage, and no explicit Scope.REQUEST declaration anywhere. NestJS promotes any provider that injects the REQUEST token to request scope automatically, and that scope bubbles up to every provider that depends on it. Injecting REQUEST is the entire opt-in.

request-scoped-logger.service.ts:

@Injectable()
export class RequestScopedLoggerService implements ITraceLogger {
  constructor(
    @Inject(BASE_LOGGER) private readonly logger: LoggerService,
    @Optional() @Inject(REQUEST) readonly req?: Record<string, unknown>,
  ) {}

  info(message: string, context?: string): void {
    this.logger.info(message, context, this.traceId);
  }

  get traceId(): string | undefined {
    return getTraceId(this.req);
  }
}

What a developer writes, and what comes out:

this.logger.warn(`User ${id} not found`, 'UsersService.findOne');
{"timestamp":"2026-09-05T15:21:43.738Z","level":"warn","serviceName":"users-service","podId":"b4799cf77-8t452","context":"UsersService.findOne","traceId":"01M0J6EYRY4TFEPR9PHJZ1QHPF","message":"User 9 not found"}

Two arguments, no trace ID, no context object, no scope to open or close. When the context argument is omitted, the logger derives Class.method from the call stack — service name and context are what turn a found line into a place in the code, and the id alone is not enough for that.

Request scope is affordable because of the singleton/wrapper split. In logger.module.ts:

providers: [
  LoggerService,
  { provide: BASE_LOGGER, useExisting: LoggerService },   // one instance per process
  RequestScopedLoggerService,                              // one stateless wrapper per request
  { provide: APP_INTERCEPTOR, useFactory: /* request logging */ },
]

useExisting, not useClass: the configured, level-filtering logger is instantiated once per process, and only a wrapper holding a reference to it is built per request. The same idiom appears exactly twice in the whole system — once for the logger, once for the HTTP client — and a third outbound concern (a message broker, a cache, an audit trail) is the same forty lines again.

Three limits of the automatic path are designed, not accidental:

  1. It is opt-in per injection site. Code that injects the singleton LoggerService directly still passes the id by hand. Automatic behaviour belongs only to the request-scoped wrappers.
  2. @Optional() is load-bearing. Outside an HTTP request — a scheduler tick, onModuleInit, bootstrap — the REQUEST token cannot resolve. @Optional() turns that into req === undefined rather than a DI failure, and the id is silently absent. There is no fallback id. Silent is correct here: a logger that throws at startup is worse than a log line without an id.
  3. It does not survive an async boundary. The context is DI-scoped rather than ambient, so work deferred past the response — a floating promise, a message handed to a broker — loses it. Such work carries the id explicitly, which is why an id belongs in a message payload (§4.6).

4.3 Carry it outbound​

The request-scoped HTTP client inherits the same way. The entire inbound-to-outbound propagation is one spread in request-scoped-http-connection.service.ts:

return this.baseConnector.connect<RT, DT, B>({
  ...params,
  ...(!params.traceId && { traceId: getTraceId(this.req) }),
  headers: { ...forwarded, ...objectKeysToLowerCase(params.headers) },
});

An explicit traceId wins; inheritance is the fallback, never an override — which is what lets a handler fan out under derived IDs. Two further rules keep the propagation from being switched off by accident:

  • The trace id is not part of forwardHeaders. It has its own path, so no change to header forwarding can disable correlation.
  • The two contract headers are written last, and every casing variant of them is deleted first. In http-connection.service.ts:
private setContractHeader(headers, name, value, wireCase = name): void {
  for (const key of Object.keys(headers)) {
    if (key.toLowerCase() === name) delete headers[key];
  }
  headers[wireCase] = value;
}

Delete-then-write rather than overwrite, because { 'User-Agent': a, 'user-agent': b } is two distinct keys in a plain object that only become one header at the fetch boundary — where they are combined into a, b rather than resolved. That is a value no receiver reads as an identity, and nothing upstream of the wire would have shown it.

The singleton client, used outside requests, deliberately never reads an ambient id. A process-wide object that reaches for request state is how an id ends up attached to the wrong request under concurrency.

4.4 The request/response pair is what makes a graph​

A trace id alone only groups log lines. What turns them into a call graph is a pair of records emitted automatically by an interceptor that LoggerModule.forRoot() registers — a service cannot end up logging without the records a trace needs:

{"direction":"request.in","method":"GET","path":"/v1/users/1","caller":"products-service"}
{"direction":"response.out","method":"GET","path":"/v1/users/1","statusCode":200,"duration":1}

The HTTP client writes the matching pair on its side of every outbound call:

directionWritten byLevelMeans
request.inrequest-logging interceptorinfo (compact) · verbose (full)a request arrived here
response.outrequest-logging interceptorinfo (compact) · verbose (full)this service answered it
request.outHTTP clientinfo, alwaysthis service called someone
response.inHTTP clientverbosethat call came back

Four consequences worth knowing:

  • caller is the forwarded User-Agent, and it is the only edge information that exists. There is no parent span ID. A service whose User-Agent disagrees with the name it logs under produces edges that match no node, and its calls appear as orphaned roots — while every individual log line still looks correct. This is why §3.4 is a rule and why the client refuses to boot without an identity.
  • request.out is always logged at info, because the callee's record exists only if the callee logs at all. The caller's own statement that the call happened is half a trace and can't come from anywhere else — a service that hasn't adopted the pattern still appears in the graph as the target of someone's call. Two lines per outbound call is the price.
  • duration is measured in-process, between the interceptor seeing the request and seeing the response. It is immune to clock skew between pods, and it is not network time.
  • Failed requests still get a response.out, with the status read off the exception, because the exception filter has not written the response yet when the interceptor observes the error.

Health, readiness, version and metrics paths are excluded by default. Left in, they would be most of the log volume, and every probe would be a root span nobody asked for.

4.5 The log line​

json.formatter.ts is the on-the-wire contract:

{"timestamp":"…","level":"info","serviceName":"orders-service","podId":"b4799cf77-8t452","context":"OrdersService.getOrder","traceId":"01M0J6…","message":"…","trace":"…"}
  • Field order is contractual and asserted in a test, because humans scan raw lines in a log viewer and a stable shape is what makes them scannable.
  • Transport is process.stdout.write (stderr for errors). No logging framework, no transport, no buffer, no queue: nothing inside the process can drop the line that was needed, and there is nothing to flush before a crash — including the crash under investigation.
  • The id lives in the line body, not in a log label. Loki-class stores index labels, not content; a per-request ID promoted to a label is unbounded cardinality, which is pathological for the index. A narrow label selector plus a line filter is the shape the store is built for, and that single query is the entire distributed-trace lookup.
  • serviceName comes from DEPLOYMENT_NAME, which must equal the container name (§3.4).

4.6 Work with no inbound request​

A scheduler tick, a queue consumer, a bootstrap task: no request, so no id to inherit. The ID is built and threaded by hand, using the singleton logger and client:

const sessionTraceId = newTraceId();

for (const page of pages) {
  const traceId = deriveTraceId(sessionTraceId, 'page', page);
  this.logger.info(`Processing page ${page}`, 'SyncService.run', traceId);
  await this.connector.connect({ url, method: 'GET', traceId });
}

deriveTraceId appends a suffix rather than replacing the id (01M0J6…-page-3, 01M0J6…-page-3-chunk-2), so a substring filter on the session id returns the session and every derived leg. That property is the only reason the shape matters, and it is why the format is never validated: a format check would refuse to trace real traffic.

For an asynchronous broker, the two halves of a hop behave differently, and this is the clarifying detail:

LegContext available?Mechanism
Publish — service → brokeryes, still inside the originating requestautomatic, via the request-scoped client
Delivery — broker → subscriberno, the request returned long agomanual: the id travels in the message payload and is read back out

Only the delivery side needs manual threading.

4.7 IDs from outside the platform​

Two production practices follow from the value being any non-empty string, and neither is exercised by the reference repository:

  • Frontend-minted IDs. The browser mints the id and sends it to its backend-for-frontend. A request that didn't come through the frontend receives an ID minted by the BFF in a distinguishable format, so the shape of a trace's ID shows whether the UI was bypassed. Because the ID is echoed on every response, a tester reads the failing request's ID from the browser's network tab, and a user-facing error can display it for a ticket. The reference repository carries the client-side interceptor as source, with the caveat that its frontends make no HTTP calls.
  • A partner's identifier as the trace id. Where the platform receives events from another organisation's application, the receiving handler uses that organisation's event id as the trace id for every chain it fires — through the explicit-id path in §4.3. The two sides share no logging stack, yet an issue on one side yields an ID the other side can search for in its logs. This is what "never validated" buys.

4.8 Payload logging and masking​

Request logging has three modes: compact (default — direction, method, path, caller, status, duration), full (the same plus masked headers and bodies, at verbose level), and off. The default is deliberate: tracing has to work in every environment with no configuration, because a service that must be reconfigured before it can be traced will not be traced on the day it breaks.

full is an opt-in for a flow under scrutiny. In practice, it earns its cost on lower environments during third-party integration work — to confirm exactly what was sent and read the partner's complete response — and is almost never needed in production, because by then the integration has been verified. The consequence is a logging discipline rather than a tool: ordinary log lines have to make a process legible on their own.

Masking covers credentials only: authorization headers (scheme kept readable, because "Basic where Bearer was expected" is a real bug), cookies, API keys, and a fixed list of body and query keys, at the top level of an object. It does not cover personal data. Payload logs are production data and are treated as such.

5. Reconstruction Tooling (Optional Layer)​

Nothing above depends on this section. A trace ID plus one query already returns every correlated line in the log viewer, in order, and for most investigations that is the whole job.

What follows makes a separate point: once correlation lives in the platform's own log lines, purpose-built tooling on top of them is cheap. The reference implementation's reconstruction engine is dependency-free JavaScript that isn't deployed, published, or in any service's dependency tree; deleting it wouldn't change a single line of application code.

It pairs request.in with response.out per (service, method, path) in timestamp order, collapses spans into caller → callee edges, nests them into a call tree by choosing the nearest span belonging to the reported caller in either direction (pod clocks are not synchronised), and reports coverage, wall time, repeated calls, and every error and warning logged under the id.

A flat id carries no span identity, so the reconstruction is a heuristic — and the tool says so rather than presenting a clean graph:

FlagMeansDoes not mean
ambiguousanother request was open on the same endpoint; the pairing is a guessthat the call failed. Do not trust the duration; the call happened
paired: falseone half of the pair is missing — usually the window clipped itthat the request was incomplete
unterminatedno response was ever logged — crash, timeout, or still in flightnoise. It is frequently the answer
gapsa service took part but emitted no request logsthat it was skipped. Usually it has not adopted the logger, runs below info, or is not ours
repeatedEdgesthe same call was made more than once in one requestwhich of cache misses or duplicated work it is. Reported, never resolved
found: false with an unreachable probean environment could not be searchedproof of absence

Unparseable lines surface as raw participants rather than being dropped: a service that logs in an unfamiliar format must still appear, or the trace grows a hole that reads as "this service was never called". Environments are searched cheapest first, because finding a trace is cheap and proving its absence is not.

The same engine is exposed as an MCP tool, grafana_trace_id, so an AI agent gets the assembled chain in one call. Because the chain names services whose source is checked out in the same workspace, the step from "which service failed" to "which line failed" needs no context-gathering — that workspace is RA-003, and the boundary between the two is clean: this architecture covers correlation and reconstruction; RA-003 covers the context an agent needs to act on the result.

The reconstruction has surfaced real defects without anyone looking for them: in the reference repository, an N+1 against the users service (two products share an owner and each is resolved independently); in production, broken caching of an auth token on the author's team's own service, and events still being sent to an outdated third-party tool.

6. Scalability & Evolution Path​

6.1 Other stacks​

The contract is two wire formats — the header and the request/response pair — so the mechanism ports by re-deciding only decision 3 of §3.2:

RuntimeAmbient-id mechanism
Express / Fastify / Koathe seeding middleware ships framework-free; attach a per-request logger to req
Node without a DI containerAsyncLocalStorage — which, unlike DI scope, does survive async boundaries
Gocontext.Context
Pythoncontextvars plus a logging filter that injects the id
Java / SpringMDC with a servlet filter
Browsermint an id per call in the API client and set the header

In production, the Python reimplementation and the partner-id adoption (§4.7) show this is true in practice, not only in principle.

6.2 A third outbound concern​

Adding a message broker, a cache, or an audit trail to the automatic path repeats the §4.2 idiom: one singleton doing the work, aliased under a token with useExisting, and a request-scoped wrapper that holds nothing but a reference to it and reads the id off REQUEST.

6.3 Who operates it​

Because the whole lookup is one id and one query, the pattern's evolution in production was organisational rather than technical: first-line investigation moved from engineers to the business analyst on test environments and to the support tiers in production, and later to AI agents driving the reconstruction tool. The field report covers that with cases.

7. Trade-off Analysis​

FeatureBenefitDrawback
Application codeA header, a log field, two module imports per service; nothing per callOpt-in per injection site; scheduled and queue work threads the id by hand
InfrastructureRides on stdout capture and the existing log store; nothing beside the process; no sampling; zero cost when idleLog volume: two lines per request per service, plus two per outbound call
Call topologycaller → callee edges, status codes and in-process durations recoverable from the pairEdges are inferred from User-Agent and timestamps, not known; concurrent identical calls are ambiguous; no per-operation spans
DepthEvery error and warning of the request under one ID, in every serviceNo database, cache or broker internals; no sub-request timing
Failure modeLog store down: lose search, not the request. No exporter in the request pathCorrelation lives and dies with log retention
InteroperabilityAny stack that forwards a header and logs a field joins the trace; foreign IDs adoptable as-isDepends on the identity rule (§3.4); a drifted User-Agent orphans a service's calls silently unless caught at boot
OperabilityOne ID, one query — usable by non-engineersThe reconstruction layer is heuristic and must be read with its flags

8. Conclusion​

This architecture treats request correlation as a property of the log line rather than of a tracing stack. One header, one field, and the request/response pair are wire formats; delivering them through shared libraries and making the ID ambient through request-scoped DI is what makes adoption two imports and coverage close to total. The cost is stated plainly — call topology is inferred, not known — and it was accepted in exchange for a mechanism with no runtime beside the application, no sampling, and a failure mode that loses search rather than requests.

It has run in production since 2022, covering more than 100 microservices across a 200+ service platform, crossed a language boundary and an organisational one, and moved first-line investigation away from engineers. The reference implementation is small enough to read in an afternoon.

References​