Lately I’ve been trying to find (and understand) the limits of time
syncing between Linux systems. How accurate can you get? What does it
take to get that? And what things can easily add measurable amounts of
time error?

After most of a month (!), I’m starting to understand things. This is
kind of a follow-on to a
previous post, where I walked through my setup and goals, plus another
post where I discussed time syncing in general. I’m trying to get
the clocks on a bunch of Linux systems on my network synced as closely
as possible so I can trust the timestamps on distributed
tracing records that occur on different systems. My local network
round-trip times are in the 20–30
microsecond (μS) range and I’d like clocks to be less than 1 RTT
apart from each other. Ideally, they’d be within 1 μS, but 10 μS is
fine.

It’s easy to fire up Chrony
against a local GPSTechnically, GNSS,
which covers multiple satellite-backed navigation systems, not just the
US GPS system, but I’m going to keep saying “GPS” for short.-backed time source and see it claim to be within X
nanoseconds of GPS, but it’s tricky to figure out if Chrony is right or
not. Especially once it’s claiming to be more accurate than the
network’s round-trip time20 μS or so., the amount of time needed for a single CPU cache
miss50-ish nanoseconds., or even the amount of time that light would take to span
the gap between the server and the time source.About 5 ns per meter.

I’ve spent way too much time over the past month digging
into time, and specifically the limits of what you can accomplish with
Linux, Chrony, and GPS. I’ll walk through all of that here eventually,
but let me spoil the conclusion and give some limits:

• GPSes don’t return perfect time. I routinely see up to 200 ns
  differences between the 3 GPSes on my desk when viewing their output on
  an oscilloscope. The time gap between the 3 sources varies every second,
  and it’s rare to see all three within 20 ns of each other. Even the best
  GPS timing modules that I’ve seen list ~5 ns of jitter on their
  datasheets. I’d be surprised if you could get 3-5 GPS receivers to agree
  within 50 ns or so without careful management of consistent antenna
  cable length, etc.
• Even small amounts of network complexity can easily add 200-300 ns
  of systemic error to your measurements.
• Different NICs and their drivers vary widely on how good they are
  for sub-microsecond timing. From what I’ve seen, Intel
  E810 NICs are great, Intel X710s are very good, Mellanox
  ConnectX-5 are okay, Mellanox ConnectX-3 and ConnectX-4
  are borderline, and everything from Realtek is questionable.
• A lot of Linux systems are terrible at low-latency work.
  There are a lot of causes for this, but one of the biggest is random
  “stalls” due to the system’s SMBIOS
  running to handle power management or other activities, and “pausing”
  the observable computer for hundreds of microseconds or longer. In
  general, there’s no good way to know if a given system
  (especially cheap systems) will be good or bad for timing
  without testing them. I have two cheap mini PC systems that have
  inexplicably bad time syncing behavior,1300-2000 ns. and two others with inexplicably good time
  syncing20-50 ns. Dedicated server hardware is generally more
  consistent.

All in all, I’m able to sync clocks to within 500 ns or so on the
bulk of the systems on my network. That’s good enough for my purposes,
but it’s not as good as I’d expected to see.

Now, it’s certainly possible to do better than this in
specific cases. For examples, see Chrony’s examples
page, where they get <100 ns of error over the network for a specific test case. In general, though it’s going to be hard to
do much better than 200 ns consistently across a real network without a
lot of careful engineering.

I’ll explain my conclusions in a bit, but first some background and
context.

My Setup

For the sake of testing time, I’m using 8 different (but identical)
servers as time clients and 5 different GPS-backed time sources, all
local.

Time Sources

1. ntp1: an older LeoNTP GPS-backed NTP server. In the
   garage, connected to its own outdoor GPS antenna. Only has a 10/100 Mbps
   Ethernet connection, but this hasn’t mattered in practice.
2. ntp2: identical hardware to ntp1. Sitting
   on my desk and connected to a different Ethernet switch. Connected to a
   GPS antenna splitter and an outdoor antenna.
3. My desktop. A 32-core AMD Threadripper 5975WX with a ConnectX-6 NIC
   (2×40 Gbps) for network traffic and an Intel E810-XXVDA4T (using 2×10
   Gbps, one to each switch) with a GPS receiver and hardware timing
   support. Shares the antenna with ntp2, ntp4,
   and ntp5.
4. ntp4:Where is NTP3 you ask? I ran out of antenna ports, and
   anyway the system that I dubbed ntp3 only supports PTP, not
   NTP. a Raspberry Pi CM5 with a Timebeat GPS module including
   PPS timing straight to the NIC. Connected via 1 GbE.
5. ntp5: a Raspberry Pi CM5 with a Waveshare GPS module
   with GPIO PPS but no working Ethernet PPS. Connected via 1 GbE.

Test Devices

1. Eight identical servers (d1 through d8)
   running Ubuntu 24.04 with identical Chrony configs. The servers are HPE M510 blades with
   16 Xeon-D cores in a pair of HPE EL4000
   enclosures. Each enclosure is connected to both of the core
   switches, giving each of the 8 servers 2 dedicated 10 GbE links via a
   built-in Mellanox ConnectX-3 NIC. Chrony metrics are
   collected every 10 seconds and stored in Prometheus for analysis.
2. A Siglent
   SDS1204X-E Oscilloscope connected to the PPS outputs from
   ntp2, ntp4, and my desktop. It can show
   relative differences in PPS times within about a nanosecond.The oscilloscope only has 200 MHz bandwidth but
   captures 1 billion samples per second, so I’d expect it to be able to
   show differences between PPS sources to somewhere between 1 and 5
   nanoseconds. In any case, the observed differences are much larger than
   this, see below.

Network

The core of the network is a pair of Arista
7050QX-32S switches. These are 32-port 40 GbE switches with hardware
support for PTP. They’re older, but very solid.

Linux systems with multiple network connections (the 8 test servers
and my desktop) are connected to each with a /30 link per interface and
then run OSPF
with ECMP
to provide redundancy. Devices with a single network interface
(Raspberry Pis and LeoNTP devices in this example) are connected to
layer 2 switches which are then connected to the core switches via MLAG
links. This means that there are multiple possible paths between any two
devices through the network, as both ECMP and LAG use a mix of source
and destination addresses to decide which link to use. So the path
between d1 and ntp1 may be almost completely
different from the path between d2 and ntp1,
even though d1 and d2 are sitting less than an
inch from each other and share all of the same physical network links.
Even more entertaining, the path back from ntp1 to
d1 and d2 may or may not be the same as the
forward path. This only matters when nanosecond-level timings are
involved, as we’ll see in a bit.

Sources of Error

So — finally — I have multiple NTP servers, presumably synced to GPS
satellites as accurately as possible, and multiple servers, all synced
to the NTP servers over a relatively low-latency network. How accurately
are my servers syncing to GPS time? And where is that going wrong?

Chrony’s claims

So, if you’re trying to see how accurate Chrony’s time syncing is,
the easiest place to start is with Chrony’s own metrics. In this case,
Chrony claims that it’s had a median offset of 25–110 ns over the past
day:

Now, this isn’t the best metric for a number of reasons, but it’s a
start. It says that Chrony thinks that it’s synced to within 110 ns of
something, but it doesn’t really tell us anything about what
it’s synced to or how accurate it actually is. So, let’s dig in a bit
deeper.

GPS error and drift

First, the GPS receivers in my NTP time servers aren’t perfectly
accurate. Even top-tier GPS receivers will still have ~5 ns of timing
noise, and lower-tier ones will be 20–60 ns (or possibly higher).Datasheet links: the ublox
ZED-F9T in my desktop claims 5 ns of accuracy and 4 ns of jitter.
The ublox
NEO-M8T in NTP5 (not graphed here) claims 20-500ns of accuracy,
depending on the antenna. And LeoNTP
claims 30ns of RMS accuracy and 60ns of 99-th percentile accuracy.

Fortunately, this is relatively easy to measure, at least when the
devices are within a few feet of each other. You can connect an
oscilloscope to their PPS outputs and directly view the differences
between them. Here’s the result for ntp2,
ntp4, and my desktop:

Notice that (a) they don’t all agree and (b) they move around
relative to each other. In this sample, there’s about a 200ns difference
between NTP4 (top, yellow) and my desktop (bottom, blue). Some of this
is due to cable length differences (my antenna and PPS leads aren’t all
identical-length, so there’s probably ~20ns in difference there alone),
but that doesn’t explain all of it.

Even ignoring the NTP4, there’s ~25ns in variance between
ntp2 (purple, middle) and my desktop (blue, bottom). Notice
that they move relative to each other over time in a bit of a
pattern.

In general, offsets can mostly be compensated for, either in
Chrony or directly on the GPS device, but jitter is trickier.

Depending on how you look at things, I’m seeing a minimum of
25 ns of error at this level, and potentially up to 200ns.

When you give Chrony multiple time sources that are all equivalently
good, then it’ll generally average its time across the whole set of
sources. So adding one time source with 200 ns of offset to 2 other
mostly-identical time sources should only add ~67 ns of error at most,
and possibly no error at all, if Chrony decides that the 200 ns source
is too far off to be used.

Network error

Chrony tries to compensate for network delays when it syncs to NTP
sources over the network, but it has to make some assumptions that
aren’t always true. It assumes that network delays are symmetrical (that
is, if it takes 30 μS for network traffic to get from the client to the
server and back, then it takes 15 μS each way). This isn’t generally
true, but for a lot of networks it’s close enough.

Apparently it’s not particularly true for my network.

One of the things that I’m monitoring with Chrony and Prometheus is
the current offset for each time source on each Chrony client. I have
data for my 8 test servers (d1 through d8)
tracking the relative offsets for ntp1 and
ntp2. I was expecting to see that either ntp1
or ntp2 was consistently ahead of the other one, given
cable lengths, network delays, antenna differences, and so forth.

Instead, half of the servers see ntp1 as running faster,
while half show ntp2 as running faster:

quantile_over_time(
  0.5,
  chrony_sources_last_sample_offset_seconds{instance=~"${client}",source_address="10.1.0.238"}[1h]
)
- on (instance)
quantile_over_time(
  0.5,
  chrony_sources_last_sample_offset_seconds{instance=~"${client}",source_address="10.1.0.239"}[1h]
)

The servers can’t agree on whether ntp1 runs faster than
ntp2 or not — 4 of the 8 see ntp1 as faster,
while 4 see ntp2 as faster, with the servers in two bands
around +100ns and -300ns. This has been consistent for weeks.To be clear, since half of the d* servers
are in one enclosure and half are in another: the timing differences are
basically random, and don’t follow which chassis they’re in or which
network cables they use. Of the 4 physical servers in each enclosure, 2
think ntp1 is faster and 2 think ntp2 is
faster, but which two aren’t even consistent between
enclosures.

Presumably this is caused by asymmetric traffic paths in my network.
If you look back to the network diagram above, you’ll see that the test
servers each have a link to each core switch, and that the L2 switches
that the NTP servers use are each connected to both core switches. Any
time you have redundant links like this, something has to
decide which path any given packet is going to take over the network. In
general network people really dislike just picking paths at
random, largely because that’d mean that packets could
arrive out of order, and a lot of TCP stacks hate out-of-order
traffic. So, generally, traffic is assigned to a path using a hash of
source and destination addresses.The exact implementation varies widely and is
frequently configurable on higher-end devices. For L2 links most devices
just hash the source and destination MACs, while for L3 links the hash
usually includes the source and destination IPs and may include
the TCP/UDP port numbers or other easy-to-locate data.

Presumably one of the possible paths between servers and time sources
on the network is faster than the others, and paths that hash onto the
faster path consistently skew the results in one direction or the other.
A less complex (and less redundant!) network would have less of this
sort of error, but asymmetric round trip times show up
everywhere in networking when you’re counting nanoseconds. At
some level, this isn’t avoidable.

So, on my network, this seems to cause a minimum of 200ns of
potential error, as various paths take different amounts of time, and
Chrony isn’t able to compensate automatically.Chrony has a per-source setting for adjusting latency
asymmetry, so I could probably hand-adjust all 16 (d* ->
ntp*) config lines to minimize the error if I
really cared about ~200 ns of error, but it’s unlikely that
it’d buy me much useful accuracy.

Cross-server synchronization

As an experiment, I told all 8 of my test servers to use each other
as time sources. I added them using Chrony’s noselect flag,
so they wouldn’t try to use each other as authoritative; they’d just
monitor the relative offsets between servers and record them over
time.I’m actually measuring time really
aggressively between d* servers. I’m polling every 250 ms
and averaging across 5 samples to try to minimize noise. Flags from
chrony.conf:
server xxx noselect xleave presend 9 minpoll -2 maxpoll -2 filter 5 extfield F323

Here’s the medianNote that the median isn’t really the best way to look
for offsets in general, but since Chrony maintains its own view
of time and slowly adjusts it time relative to its sources, so a few
wildly inaccurate responses won’t really change Chrony’s time much, if
at all. offset between servers, in nanoseconds, over 4 hours:

quantile_over_time(
  0.5,
  chrony_sources_last_sample_offset_seconds{instance=~"d[1-8].*",source_address=~"10.0.0.10[0-9]"}[5m]
)

Notice that they’re all within 207ns of each other, but the timings
aren’t particularly consistent. For instance, looking at the timings
between d2 and d3 show that they’re 29 ns
apart when you query one direction and 23 ns apart when you query in the
other direction, but they’re both off in the same direction. If
network error wasn’t a factor, then I’d expect to see one number be
positive and the other be negative; that’s not always the case here.

In general, this aligns nicely with the 200-300ns of error seen in
the previous section, but it shows that there’s a serious limit to how
accurately Chrony can measure nanoseconds on this hardware.

Observed offsets across all
sources

Earlier, I discussed the difference between ntp1 and
ntp2, and how each server had a different view of the
difference between them. On average, ntp1 seems to run
50–150 ns ahead of ntp2.

Remember that my big goal here is less accurate time and
more consistent time. This 50–150 ns of inconsistency isn’t a
big deal, but when I started adding additional time sources, I
discovered that some of them were even further away from
ntp1 and ntp2, and I wanted to minimize the
total time spread. I’d really like it if adding additional NTP sources
to the mix didn’t make things even less consistent.

There are a lot of things to like about the LeoNTP time servers, but
configurability isn’t one of them. There’s no way that I can
see to add an offset between GPS time and the NTP time that they export.
On the other hand, the 3 Chrony-based time servers (my desktop,
ntp4, and ntp5) can be adjusted to
control the offset between GPS time and NTP time. And, in fact, you
can’t really run with a 0-second offset because GPS time is
based on TAI
and NTP time is usually UTC,Strictly speaking, it’s usually a weird mutant that
mostly ignores leap seconds, look up “leap smear” for the
ugly mess. and the two are currently 37 seconds apart.Leap seconds make life hard.

Originally, I discovered that time from my desktop was around 1 μS
when compared with ntp1 and ntp2 by
d* servers, and time from the Raspberry Pi-based
ntp4 was almost 38 μS off! To mitigate this, I graphed the
average offset between each time source across all 8 servers and then
adjusted offsets on my desktop and ntp4 to be as close as
possible to the median of ntp1 and ntp2. To do
this, I changed my desktop’s TAI offset from -37 to
-36.999999160 and the offset of ntp4 to
-37.000033910.

Now, all 4 sources are basically in unison:

avg by (source_address) (
  quantile_over_time(
    0.5,
    chrony_sources_last_sample_offset_seconds{
      instance=~"d[1-8].*",
      source_address=~"10[.].*",
      source_address=~"${ntpsource}"
    }[1h]
  )
)

Why were times so far off? For my desktop, it’s probably a mix of
multipath weirdness and delay in the network stack. 840 ns isn’t a huge
amount of time, although it’s bigger than what I’ve seen elsewhere.

I’m less sure what’s going on with ntp4. It was
originally seeing over 50 μS of error, but reducing the Ethernet
coalescing limits on eth0ethtool -C eth0 tx-usecs 0 rx-usecs 0 helped quite a bit. I’m going to have to keep poking at
this for a while.

Observed NTP jitter
across all sources

I can compare the jitter of 4 of my GPS time sources across all 8
d* servers. To calculate jitter in this case, I’m looking
at the difference between the 1st and 99th percentile of each source’s
offset from Chrony’s best estimate of the current time. I’m calculating
the percentiles over 15 minute windows, subtracting the 1st percentile
from the 99th, and then averaging those results across all 8
servers.It’s not the best way to do this statistically, but
there’s a limit to what you can do with Prometheus easily.

(
  avg by (source_address) (
    quantile_over_time(
      0.99,
      chrony_sources_last_sample_offset_seconds{
        instance=~"d[1-8].*",
        source_address=~"10[.].*",
        source_address=~"${ntpsource}"
      }[15m]
    )
  )
) - (
  avg by (source_address) (
    quantile_over_time(
      0.01,
      chrony_sources_last_sample_offset_seconds{
        instance=~"d[1-8].*",
        source_address=~"10[.].*"
      }[15m]
    )
  )
)

Over the past hour, that works out to:

So, my desktop (with a fast NIC and a very good GNSS module) has the
least jitter. The two LeoNTP boxes are next, with a bit more, and the
Raspberry Pi has 2x the jitter of my desktop. Since Chrony averages out
offsets across sources and over time, jitter isn’t necessarily
a big deal as long as it’s under control.

Which brings up ntp5, which I’d excluded from the
previous graph. Here’s why:

I still haven’t figured out why this loses accuracy every 2 hours,
but there are other weird things about ntp5, so I’m not all
that worried about it overall.

Things that hurt syncing

Along the way, I’ve found a bunch of things that hurt time syncing. A
short list:

• Network cards without hardware timestamps. Realtek, for
  instance.
• Tunnels. I had 3 servers who were sending traffic to the network
  with ntp1 and ntp2 over VxLAN
  originally, and their time accuracy was terrible. I suspect that the
  NICs’ hardware timestamp wasn’t propagated correctly through the tunnel
  decapsulation. Plus, it made network times even less symmetrical.
• NIC packet coelescing. On Raspberry Pi CM5s especially, I had to
  disable NIC coelescing via ethtool -c or I had terrible
  accuracy.
• Software in general. I get the best results on NTP servers where the
  GPS’s PPS signal goes directly into the NIC’s hardware, completely
  bypassing as much software as possible.
• Running ptp4l and Chrony on the same
  ConnectX-4 NIC, or potentially ConnectX-3 or -5 NICs. Intel seems
  perfectly happy under the same situation.

Summary

So, in all, I’m seeing time syncing somewhere in the 200–500 ns range
across my network. The GPS time sources themselves are sometimes as far
as 150 ns apart, even after compensating for systemic differences, and
the network itself adds another 200–300 ns of noise.

In an ideal world, it’d be cool to see ~10 ns accuracy, but it’s not
really possible at any level with this hardware. My time sources aren’t
that good, my network adds more systemic error than that, and when I try
to measure the difference between test servers I see a couple hundred
nanoseconds of noise. So 10 ns isn’t going to happen.

On the other hand, though, I’m almost certainly accurate to within 1
μS across the set of 8 test servers most of the time, and I’m
absolutely more accurate than my original goal of 10 μS.