"Detected incorrectly running system clock" log message

[Peter]

This is for all people that find messages like this in their Log files:

2010-04-09 21:58:22.019663|WARNING |PktHandler | | detected incorrectly running system clock

Often these are also accompanied by clients dropping from the server with the message "Connection Lost".

What does this message mean?

"detected incorrectly running system clock" means we queried for the current time
once, got some value A, then later queried again, got B, but:
i) B signified an earlier point in time than A, so in effect the clock warped into the past, or
ii) B is too far past A to make sense (when we know the two queries are not far apart), in this case the time warped into the future.

On unix systems we use "gettimeofday" to query the system clock, on windows we use timeGetTime.

Why does it occur?

Unix
On Unix Systems any change of the system time can be the cause, scripts running ntpdate or ntpd obviously alter the system time. Make sure you use ntpd since it adjusts the time in small delecate steps that do not cause this problem. Some users have reported problems even with ntpd:

The message seems pretty clear: something (or somebody) changed system-time in such a step which TS3 considers as problem. As a result TS3-server kicks everybody out, and needs some time to recover. It could happen i.e. when system time is updated by ntp-daemon, as it was in my case: I noticed I have this problem only when I have ntpd running. So I checked what ntpd was doing when this happened, comparing logs for TS3 and ntpd. And I found messages like this in /var/log/ntpd.log (about one per hour):

2010-11-04 T 17:14:45+00:00 obelix ntpd[15932]: time reset -0.182409 s

After a little digging into ntp documentation I found:

"...Normally, the time is slewed if the offset is less than the step threshold, which is 128 ms by default, and stepped if above the threshold..."

It means, if difference between local time and ntp-server time is >0.128s, system time is adjusted using stepping, because ntpd considers with slewing it would take too much time (in my case, kernel allows about +/- 0.5ms per second, so correcting ~0.18s deviation would take ~6min). And that is exactly what was causing problems for TS3-server in my case (and some other software, e.g. dovecot, which died a few times too). Finally, I have found solution using ntpd docu again:

"...tinker [ step step ]
Spedifies the step threshold in seconds. The default without this command is 0.128 s. If set to zero, step adjustments will never occur..."

So I included "tinker step 0" in my /etc/ntpd.conf, restarted ntpd, and I have no more "incorrectly running system clock" messages in TS3-log, and clients have never been kicked off my TS3-server since then

Other users reported problems when using screen and going into "Copy mode". Make sure to leave copy mode and don't leave the TS3 server running with copy mode activated.

All Systems
Of course there is also a Hardware component here. The operating system uses the system time source to provide the time to us, if the system clock is behaving badly, there is nothing the operating system or TS3 can do.

Also we have done some changes to the ways we handle time problems to be more robust to all but the nastiest time inconsistencies. The current 3.0.0-rc1 server already contains some of these fixes, the soon to be released 3.0.0 server will contain some more. We advise you to upgrade to it.

[ANR Daemon]

For Linux, especially Debian-based systems, the problem could be not the NTPD running (It does not cause time leaps at all), but a stupid script in /etc/network/if-up.d/ called ntpdate.
THAT is what WILL cause time leaps each time any of your network interfaces going up.
Lower the executable bit on that file, and make sure your NTPD is correctly configured. ( "ntpq -p" should list your peers, not empty list and not some strange abuse )

[Ar2rJke]

2012-01-01 00:19:01.959597|INFO    |ServerLibPriv |   | TeamSpeak 3 Server 3.0.1 (2011-11-17 07:34:30)
2012-01-01 00:19:01.972678|INFO    |DatabaseQuery |   | dbPlugin name:    SQLite3 plugin, Version 2, (c)TeamSpeak Systems GmbH
2012-01-01 00:19:01.972972|INFO    |DatabaseQuery |   | dbPlugin version: 3.7.3
2012-01-01 00:19:01.974302|INFO    |DatabaseQuery |   | checking database integrity (may take a while)
2012-01-01 00:19:02.504925|INFO    |Accounting    |   | Licensing Information
2012-01-01 00:19:02.504997|INFO    |Accounting    |   | type              : Non-profit
2012-01-01 00:19:02.505068|INFO    |Accounting    |   | starting date     : Thu Sep 22 00:00:00 2011
2012-01-01 00:19:02.505100|INFO    |Accounting    |   | ending date       : Sat Sep 22 00:00:00 2012
2012-01-01 00:19:02.505124|INFO    |Accounting    |   | max virtualservers: 10
2012-01-01 00:19:02.505146|INFO    |Accounting    |   | max slots         : 512
2012-01-01 00:19:02.529442|INFO    |FileManager   |   | listening on 0.0.0.0:30033
2012-01-01 00:19:02.669042|INFO    |CIDRManager   |   | updated query_ip_whitelist ips: 127.0.0.1
2012-01-01 00:19:02.673225|INFO    |Query         |   | listening on 0.0.0.0:10011
2012-01-13 14:05:41.645397|WARNING |PktHandler    |   | detected incorrectly running system clock (>)
2012-01-13 14:05:42.347604|WARNING |PktHandler    |   | detected incorrectly running system clock (>)

themselves already see the server worked 13 days without any errors. everything was fine. Today, decided to check the log and saw an error ... while users are sitting without a problem, question: it's scary or not (a critical error or not)?

*** Thread Merged ***

[robmarryatt]

Can someone please help me.We were all connected to the server and everyone timed out. I thought it was a Dos attack but when looking at the logs I see this.

2012-11-02 01:12:08.093858|WARNING |PktHandler | | detected incorrectly running system clock (<)
2012-11-02 01:12:17.318984|WARNING |PktHandler | | detected incorrectly running system clock (<)
2012-11-02 01:12:18.066010|WARNING |PktHandler | | detected incorrectly running system clock (<)
2012-11-02 01:12:18.070445|WARNING |PktHandler | | detected incorrectly running system clock (<)
2012-11-02 01:12:18.070542|WARNING |PktHandler | | detected incorrectly running system clock (<)
2012-11-02 01:12:18.070602|WARNING |PktHandler | | detected incorrectly running system clock (<)
2012-11-02 01:12:18.070818|WARNING |PktHandler | | detected incorrectly running system clock (<)
2012-11-02 01:12:18.070878|WARNING |PktHandler | | detected incorrectly running system clock (<)

This thread got merged

[Koshei]

Hello, i am using the new version of TeamSpeak3 server 3.0.7.2 on debian amd64 and i have sometimes mass connection lost of my server. Here is errors from log file and i dont know what is this and how i can repair. Anyone have some solution step by step? Please, this is very important for me. Someone can help?

2013-07-03 19:40:06.321492|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.321520|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.321542|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.321564|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.321611|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345411|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345488|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345523|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345577|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345621|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345668|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345694|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345716|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345756|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345778|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345800|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345911|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.345959|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:40:06.694437|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:42:37.227363|ERROR |DatabaseQuery | | db_open() select * from clients where not exists (select id1 from group_server_t error: database disk image is malformed
2013-07-03 19:45:41.688888|ERROR |DatabaseQuery | | db_open() select * from clients where not exists (select id1 from group_server_t error: database disk image is malformed
2013-07-03 19:48:37.229181|ERROR |DatabaseQuery | | db_open() select * from clients where not exists (select id1 from group_server_t error: database disk image is malformed
2013-07-03 19:50:52.424939|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:50:52.425100|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:50:52.425127|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:50:52.425149|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:50:52.425169|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:50:52.425189|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:50:52.425279|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 19:51:39.312651|ERROR |DatabaseQuery | | db_open() select * from clients where not exists (select id1 from group_server_t error: database disk image is malformed
2013-07-03 19:54:37.734359|ERROR |DatabaseQuery | | db_open() select * from clients where not exists (select id1 from group_server_t error: database disk image is malformed
2013-07-03 19:57:37.281292|ERROR |DatabaseQuery | | db_open() select * from clients where not exists (select id1 from group_server_t error: database disk image is malformed
2013-07-03 19:57:37.483465|WARNING |PktHandler | | detected incorrectly running system clock (>)
2013-07-03 20:00:37.227009|ERROR |DatabaseQuery | | db_open() select * from clients where not exists (select id1 from group_server_t error: database disk image is malformed
2013-07-03 20:03:37.229336|ERROR |DatabaseQuery | | db_open() select * from clients where not exists (select id1 from group_server_t error: database disk image is malformed

[ARCHI]

error: database disk image is malformed

Stop your TS3 Server, delete your ts3server.sqlitedb file and whole files/ folder. This will delete corrupted database and allow you to start from scratch.

[Koshei]

Server is working good but its a problem with the detection system clock and i dont know how can i fix this problem. I read about ntpd but i cant synchronise it with my server...

[skinhead]

Hi i have about 1000 lines of this error in my logs this only showd up after i changed to my new VPS also now my tsviewer doesnt work and the music bot both use queries and were working fine before

2014-06-10 17:18:04.190929|WARNING |PktHandler | | detected incorrectly running system clock (>)

[skinhead]

I sent this to my host they are waiting for the technician to check it. I really don't understand this topic...

[ToKKaN]

Hello the last 3-4 days my server loses connection to all clients 4-5x per day. It's pretty annoying and I really dont know what causes the problem. If I go to the log folder all log dates are incorrect.

2015-03-24 22:53:45.939249|INFO    |ServerLibPriv |   | TeamSpeak 3 Server 3.0.11.2 (2014-12-15 14:43:52)
2015-03-24 22:53:45.966231|INFO    |ServerLibPriv |   | SystemInformation: Linux 2.6.32-042stab103.6 #1 SMP Wed Jan 21 13:07:39 MSK 2015 x86_64 Binary: 64bit
2015-03-24 22:53:45.966309|INFO    |ServerLibPriv |   | Using hardware aes
2015-03-24 22:53:46.002560|INFO    |DatabaseQuery |   | dbPlugin name:    SQLite3 plugin, Version 2, (c)TeamSpeak Systems GmbH
2015-03-24 22:53:46.004622|INFO    |DatabaseQuery |   | dbPlugin version: 3.6.21
2015-03-24 22:53:46.009233|INFO    |DatabaseQuery |   | checking database integrity (may take a while)
2015-03-24 22:53:46.923415|INFO    |Accounting    |   | Licensing Information
2015-03-24 22:53:46.923514|INFO    |Accounting    |   | type              : Non-profit
2015-03-24 22:53:46.923602|INFO    |Accounting    |   | starting date     : Fri Oct  3 00:00:00 2014
2015-03-24 22:53:46.923646|INFO    |Accounting    |   | ending date       : Sun Apr  5 00:00:00 2015
2015-03-24 22:53:46.923685|INFO    |Accounting    |   | max virtualservers: 10
2015-03-24 22:53:46.923772|INFO    |Accounting    |   | max slots         : 512
2015-03-24 22:53:48.317586|INFO    |              |   | Puzzle precompute time: 1353
2015-03-24 22:53:48.319246|INFO    |FileManager   |   | listening on 0.0.0.0:30033
2015-03-24 22:53:48.671921|INFO    |CIDRManager   |   | updated query_ip_whitelist ips: censored 
2015-03-24 22:53:48.674600|INFO    |Query         |   | listening on 0.0.0.0:10011
2015-03-25 17:25:02.781912|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.798019|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.798180|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.863012|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.863190|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.863299|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.863567|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.905567|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.916988|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.917080|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.993902|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:02.994046|WARNING |PktHandler    |   | detected incorrectly running system clock (<)
2015-03-25 17:25:03.008224|WARNING |PktHandler    |   | detected incorrectly running system clock (<)

the log says "detected incorrectly running system clock" but if I use the date command in my vps it has the correct date and time.



I have a vps from OVH. I hope someone can help me!

[ANR Daemon]

ntpq -p
?

[Razilla]

I also have this problem. In a fresh Debian 8 VM, nothing else apart from TS3 is running. And every time it happens it brings the whole VM down, has to be restartet, not even SSH works. I even updated to 3.0.11.3 Beta, no change.

[topkek88]

Hello member,
I have a really bad problem with my TeamSpeak 3 Server (3.4.0) running on my OpenVZ Server with Ubuntu 16.04. Sometimes my TeamSpeak Server crashes for some seconds and all clients disconnect and then they reconnect. It happens out of nowhere and you can never foresee if it crashes. This happens maybe 1-2 times a day. I really dont know what I could do.

OpenVZ server specs: 4 vCores, 4gb RAM

In the server log it says:

|INFO |PktHandler |1 |Terminating connection because of > 255 resends
|INFO |PktHandler |1 |Dropping client 7 because of resend timeout
|INFO |VirtualServerBase|1 |client disconnected 'USERNAME'(id:*) reason 'reasonmsg=connection lost'

I would be very happy if you someone could help me

[dante696]

The log and your description do not describe a crash.
More like network problems.

Please read and show log 0 if there is a crash.
And check the network on machine where server is running.

[topkek88]

The log and your description do not describe a crash.
More like network problems.

Please read and show log 0 if there is a crash.
And check the network on machine where server is running.

This is an excerpt of the log 1 and this entry appears for every single user on the TeamSpeak.

In log 0 this here gets spammed like 300 times:

WARNING |PktHandler | |detected incorrectly running system clock (<)