Explain why clock skew across services produces 'negative duration' spans and how to handle it

Flashcard·Hard·4.0 · 0·~30s·Asked atDatadog Decagon Stripe

Attempt it

Span timestamps are recorded with the local wall clock of whichever process emitted them. If two services are on machines whose clocks differ by even a few hundred milliseconds, a child span exported from service B can look like it started before its parent in service A. Most backends clamp negative durations to zero on render and rely on parent-child links (not timestamps) to draw the tree. The practical handling is (a) keep NTP/chrony tight, (b) trust span ids over timestamps for hierarchy, and (c) treat tiny negative durations as a known artifact rather than a real anomaly.

Your answer

Trace timestamps come from the local clock of whichever process recorded them. If two services run on different machines, their clocks are not perfectly synchronized; NTP keeps them close (typically within 1 to 50 milliseconds) but never identical. **Why negative durations happen.** Service A starts a parent span at time T_A. Service A calls service B over the network. Service B is on a different machine whose clock is currently 20ms behind. Service B receives the call and starts its child span using its own clock, which reads (T_A + 5ms network) but expressed at B's offset reads (T_A + 5ms - 20ms) = (T_A - 15ms). The exported child span has a start_time that looks earlier than the parent's start_time. Same thing on the end side: a child span exported from B can end at a wall-clock time that maps to before its recorded start. **How backends handle it.** Most observability backends clamp tiny negative durations to zero or to a small positive value at render time. They do not rely on timestamps for parent-child ordering; they rely on the explicit parent_span_id field in the OTel context. The tree drawing is correct even when the timestamps disagree. **The practical handling.** Keep NTP or chrony tight on every host. Treat tiny negative durations (a few hundred ms or less) as a known artifact. Investigate only when the skew is large enough to suggest NTP failure.

**Root cause.** Span timestamps are emitted in the local wall-clock time of whichever process produced them. Two services on different machines have slightly different clocks. NTP or chrony keeps them aligned, but never perfectly. Skews of 1 to 50ms are normal in healthy clusters; 100ms or more indicates an NTP problem. **Why this produces 'negative duration' or 'child before parent' artifacts.** Service A starts a parent span at its local clock T_A and makes a network call to service B. By the time service B's process records the child span's start time, its local clock reads T_B = T_A + network_latency + clock_offset, where clock_offset is the skew between the two machines. If clock_offset is negative (B's clock is behind A), B's recorded start can look earlier than A's start. The child appears to begin before its parent. The same applies to the end side. A child span ends on service B at T_end_B. If B's clock is ahead of where it was during the start, but the start was already imported with a different clock state (NTP made a step adjustment mid-flight), end can look earlier than start, producing a negative duration. **How backends cope.** Two design choices that make the system robust. - **Hierarchy via parent_span_id.** OTel carries parent_span_id in the trace context; backends build the tree from ids, not timestamps. A child with an earlier-looking timestamp still attaches correctly. - **Clamping negative durations.** At render time, negative durations clamp to zero or a tiny epsilon. The Gantt chart looks sensible. **Practical handling.** Run NTP or chrony everywhere and alert when offset exceeds 50ms. Treat tiny negative durations as artifacts, not bugs. Investigate >100ms skews because they indicate NTP failure with wider blast radius (security tokens, log ordering, distributed locks). Trust span ids over timestamps when they disagree.

TL;DR

Each service stamps spans with its own wall clock. Even tens of milliseconds of NTP drift can flip child versus parent ordering. Backends clamp negative durations and rely on parent-span ids for the tree.

Memory aid

Easy to grasp

Imagine two friends watching a relay race from different stadiums, each timing the runners with their own watch. If their watches disagree by half a second, the friend at the second stadium might record the second runner starting before the first finished, even though they did not. The race tree is still right (everyone agrees who handed off to whom), but the times look wrong. Fix the watches with NTP, and trust the handoff record over the watches when they disagree.

Key concepts

Observability and tracing LLM system design Model context protocol

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Clock skew is one of those topics that looks like a bug but is actually a property of distributed systems. Every machine in your fleet has its own clock, those clocks drift relative to each other, and NTP or chrony keeps them close but never identical. Tracing across services means stitching together timestamps from different clocks, and when those clocks disagree, the trace shows artifacts: a child span appears to start before its parent, or a span has a negative duration.

This deep dive walks through why the artifacts happen, what OpenTelemetry's design does to make the trace usable anyway (parent_span_id is the load-bearing piece), how backends cope at render time, the operational discipline that keeps the artifacts small and rare, and when you would actually need a hardware clock instead of NTP.

Mental model: timestamps lie a little; ids do not. Order the tree by parent_span_id, clamp negative durations at render, alert on host-clock offset, and accept tiny negative durations as a known property of distributed time.

Why timestamps disagree across services

The root cause is simple and physical.

Every process uses its local wall clock

When a span starts, the SDK records the start time as 'whatever the operating system's wall clock says right now'. The OS gets this from the hardware clock, periodically corrected by NTP or chrony. The corrected value is close to UTC but not exact.

Cross-machine clocks differ

Machines A and B do not share a clock. Each has its own hardware oscillator, its own NTP daemon, and its own correction history. At any given moment, A's wall clock and B's wall clock might differ by a few milliseconds.

The OS clock is not monotonic across NTP adjustments

NTP can step the clock backward when the local clock is too far ahead of true time. The step is instantaneous. If a span on machine B starts before the step and ends after the step, the recorded end time can be earlier than the recorded start time, producing a negative duration without any physical violation of causality.

What 'normal' looks like

A healthy cluster with chrony or systemd-timesyncd typically holds skew under 10 to 50ms. Cloud VMs across the same region usually hit 1 to 5ms. Cross-region or cross-cloud traces can see 50 to 200ms skew. None of these are bugs; they are operating points.

What 'broken' looks like

Skew over 200ms across hosts in the same datacenter indicates NTP failure: the daemon is dead, the upstream server is unreachable, or the host's hardware clock has a fault. The downstream effects are wider than just traces; security tokens, distributed locks, and log ordering can all break.

How OpenTelemetry stays robust despite this

Operational discipline: monitor, alert, accept

When you need a hardware clock instead

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

AWS, GCP, and Azure all document host-clock drift as a known issue across VM live migrations and cold starts in their tracing best-practices guides.
Datadog, Honeycomb, and Tempo all clamp negative-duration spans at render time and emit a warning event rather than failing the trace.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you monitor clock skew across the fleet?

Run chrony on every host, scrape its 'offset to server' metric to Prometheus, alert at 50ms warning and 200ms critical. Cross-check with periodic ping-based clock-skew probes to a known good reference.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating tiny negative durations as a real anomaly to debug. They are nearly always clock skew artifacts; the real anomaly is anything more than a few hundred milliseconds, which means NTP is broken.

60 second bullets to scan on the way to the call.

Why each service stamps spans with its own wall clock
How network latency and clock offset interact to produce inverted timestamps

Primary sources. Browse if you want the original framing.