"Detected incorrectly running system clock" log message

[Peter]

This is for all people that find messages like this in their Log files:

2010-04-09 21:58:22.019663|WARNING |PktHandler | | detected incorrectly running system clock

Often these are also accompanied by clients dropping from the server with the message "Connection Lost".

What does this message mean?

"detected incorrectly running system clock" means we queried for the current time
once, got some value A, then later queried again, got B, but:
i) B signified an earlier point in time than A, so in effect the clock warped into the past, or
ii) B is too far past A to make sense (when we know the two queries are not far apart), in this case the time warped into the future.

On unix systems we use "gettimeofday" to query the system clock, on windows we use timeGetTime.

Why does it occur?

Unix
On Unix Systems any change of the system time can be the cause, scripts running ntpdate or ntpd obviously alter the system time. Make sure you use ntpd since it adjusts the time in small delecate steps that do not cause this problem. Some users have reported problems even with ntpd:

The message seems pretty clear: something (or somebody) changed system-time in such a step which TS3 considers as problem. As a result TS3-server kicks everybody out, and needs some time to recover. It could happen i.e. when system time is updated by ntp-daemon, as it was in my case: I noticed I have this problem only when I have ntpd running. So I checked what ntpd was doing when this happened, comparing logs for TS3 and ntpd. And I found messages like this in /var/log/ntpd.log (about one per hour):

2010-11-04 T 17:14:45+00:00 obelix ntpd[15932]: time reset -0.182409 s

After a little digging into ntp documentation I found:

"...Normally, the time is slewed if the offset is less than the step threshold, which is 128 ms by default, and stepped if above the threshold..."

It means, if difference between local time and ntp-server time is >0.128s, system time is adjusted using stepping, because ntpd considers with slewing it would take too much time (in my case, kernel allows about +/- 0.5ms per second, so correcting ~0.18s deviation would take ~6min). And that is exactly what was causing problems for TS3-server in my case (and some other software, e.g. dovecot, which died a few times too). Finally, I have found solution using ntpd docu again:

"...tinker [ step step ]
Spedifies the step threshold in seconds. The default without this command is 0.128 s. If set to zero, step adjustments will never occur..."

So I included "tinker step 0" in my /etc/ntpd.conf, restarted ntpd, and I have no more "incorrectly running system clock" messages in TS3-log, and clients have never been kicked off my TS3-server since then

Other users reported problems when using screen and going into "Copy mode". Make sure to leave copy mode and don't leave the TS3 server running with copy mode activated.

All Systems
Of course there is also a Hardware component here. The operating system uses the system time source to provide the time to us, if the system clock is behaving badly, there is nothing the operating system or TS3 can do.

Also we have done some changes to the ways we handle time problems to be more robust to all but the nastiest time inconsistencies. The current 3.0.0-rc1 server already contains some of these fixes, the soon to be released 3.0.0 server will contain some more. We advise you to upgrade to it.

[ANR Daemon]

For Linux, especially Debian-based systems, the problem could be not the NTPD running (It does not cause time leaps at all), but a stupid script in /etc/network/if-up.d/ called ntpdate.
THAT is what WILL cause time leaps each time any of your network interfaces going up.
Lower the executable bit on that file, and make sure your NTPD is correctly configured. ( "ntpq -p" should list your peers, not empty list and not some strange abuse )