Time synchronization (Managing NFS and NIS, 2nd Edition)

14.7. Time synchronization

Distributing files across several servers introduces a dependency on synchronized time of day clocks on these machines and their clients. Consider the following sequence of events:

caramba % date 
Mon Sep 25 18:11:24 PDT 2000 
caramba % pwd 
/home/labiaga 
caramba % touch foo 
caramba % ls -l foo 
-rw-r--r--   1 labiaga  staff          0 Sep 25 18:18 foo 
 
aqua % date 
Mon Sep 25 17:00:01 PDT 2000 
aqua % pwd 
/home/labiaga
aqua % ls -l foo 
-rw-r--r--   1 labiaga  staff          0 Sep 25  2000 foo 
aqua % su 
aqua # rdate caramba 
Mon Sep 25 18:16:51 2000 
aqua % ls -l foo 
-rw-r--r--   1 labiaga  staff          0 Sep 25 18:18 foo

On host caramba, a file is created that is stamped with the current time. Over on host aqua, the time of day clock is over an hour behind, and file foo is listed with the month-day-year date format normally reserved for files that are more than six months old. The problem stems from the time skew between caramba and aqua: when the ls process on aqua tries to determine the age of file foo, it subtracts the file modification time from the current time. Under normal circumstances, this produces a positive integer, but with caramba 's clock an hour ahead of the local clock, the difference between modification time and current time is a negative number. This makes file foo a veritable Unix artifact, created before the dawn of Unix time. As such, its modification time is shown with the "old file" format.[39]

[39]Some Unix utilities have been modified to handle small time skews in a graceful manner. For example, ls tolerates clock drifts of a few minutes and correctly displays file modification times that are slightly in the future.

Time of day clock drift can be caused by repeated bursts of high priority interrupts that interfere with the system's hardware clock or by powering off (and subsequently booting) a system that does not have a battery-operated time of day clock.[40]

[40]The hardware clock, or "hardclock" is a regular, crystal-driven timer that provides the system heartbeat. In kernel parlance, the hardclock timer interval is a "tick," a basic unit of time-slicing that governs CPU scheduling, process priority calculation, and software timers. The software time of day clock is driven by the hardclock. If the hardclock interrupts at 100 Hz, then every 100 hardclock interrupts bump the current time of day clock by one second. When a hardclock interrupt is missed, the software clock begins to lose time. If there is a hardware time of day clock available, the kernel can compensate for missed hardclock interrupts by checking the system time against the hardware time of day clock and adjusting for any drift. If there is no time of day clock, missed hardware clock interrupts translate into a tardy system clock.

In addition to confusing users, time skew wreaks havoc with the timestamps used by make, jobs run out of cron that depend on cron-started processes on other hosts, and the transfer of NIS maps to slave servers, which fail if the slave server's time is far enough ahead of the master server. It is essential to keep all hosts sharing filesystems or NIS maps synchronized to within a few seconds.

rdate synchronizes the time of day clocks on two hosts to within a one-second granularity. Because it changes the local time and date, rdate can only be used by the superuser, just as the date utility can only be used by root to explicitly set the local time. rdate takes the name of the remote time source as an argument:

% rdate mahimahi 
couldn't set time of day: Not owner 
% su 
# rdate mahimahi 
Mon Sep 25 18:16:51 2000

One host is usually selected as the master timekeeper, and all other hosts synchronize to its time at regular intervals. The ideal choice for a timekeeping host is one that has the minimum amount of time drift, or that is connected to a network providing time services. If the time host's clock loses a few seconds each day, the entire network will fall behind the real wall clock time. All hosts agree on the current time, but this time slowly drifts further and further behind the real time.

While the remote host may be explicitly specified, it is more convenient to create the hostname alias timehost in the NIS hosts file and to use the alias in all invocations of rdate:

131.40.52.28    mahimahi timehost 
131.40.52.26    wahoo 
131.40.52.150   kfir

Some systems check for the existence of the hostname timehost during the boot sequence, and perform an rdate timehost if timehost is found.

This convention is particularly useful if you are establishing a new timekeeping host and you need to change its definition if your initial choice proves to be a poor time standard. It is far simpler to change the definition of timehost in the NIS hosts map than it is to modify the invocations of rdate on all hosts.

Time synchronization may be performed during the boot sequence, and at regular intervals using cron. The interval chosen for time synchronization depends on how badly each system's clock drifts: once-a-day updates may be sufficient if the drift is only a few seconds a day, but hourly synchronization is required if a system loses time each hour. To run rdate from cron, add a line like the following to each host's crontab file:

Hourly update: 
52 * * * * rdate timehost > /dev/null 2>&1 
 
Daily update: 
52 1 * * * rdate timehost > /dev/null 2>&1

The redirection of the standard output and standard error forces rdate 's output to /dev/null, suppressing the normal echo of the updated time. If a cron-driven command writes to standard output or standard error, cron will mail the output to root.

To avoid swamping the timehost with dozens of simultaneous rdate requests, the previous example performs its rdate at a random offset into the hour. A common convention is to use the last octet of the machine's IP address (mod 60) as the offset into the hour, effectively scattering the rdate requests throughout each hour.

The use of rdate ensures a gross synchronization accurate to within a second or two on the network. The resolution of this approach is limited by the rdate and cron utilities, both of which are accurate to one second. This is sufficient for many activities, but finer synchronization with a higher resolution may be needed. The Network Time Protocol (NTP) provides fine-grain time synchronization and also keeps wide-area networks in lock step. NTP is outside the scope of this book.