Unusual high system load when running ts3-server

[syscore]

Hello,

i am running a ts3-beta30 x86 server on a Ubuntu 10.04-LTS virtual machine (KVM/libvirt). the virtual machine uses two cores of a intel core-i7 920 processor and has 1gb ram. the only services running at this virtual machine are teamspeak3 and a left4dead2-server.

now i have some issues with teamspeak. ts3 generates a very high and unstable system load. this happens with and without clients. the load is definitively caused by ts3, as i tried it without the source server and the load stays high. if ts3 is offline and only the source server is running, the load averages around 0.4 ...

graphic from system monitoring attached. load < 0.5 = ts3 disabled

any ideas ? the spikes above > 2.0 are way to high for a simple service.
the ts3 is a NPL, with a maximum of 32 users, voice encryption off.

[florian_fr40]

Hello

Do you get something (error or information messages) in the server log ?

[syscore]

Hello,

nothing unusual in the logs :/

2011-01-03 11:04:30.971563|INFO    |ServerLibPriv |   | Server Version: 3.0.0-beta30 [Build: 12998], Linux
2011-01-03 11:04:31.001677|INFO    |DatabaseQuery |   | dbPlugin name:    SQLite3 plugin, Version 2, (c)TeamSpeak Systems GmbH
2011-01-03 11:04:31.003342|INFO    |DatabaseQuery |   | dbPlugin version: 3.7.3
2011-01-03 11:04:31.006015|INFO    |DatabaseQuery |   | checking database integrity (may take a while)
2011-01-03 11:04:31.035825|INFO    |SQL           |   | pruning old database log entries where timestamp is older than 90 days
2011-01-03 11:04:31.224878|INFO    |Accounting    |   | Licensing Information
2011-01-03 11:04:31.225166|INFO    |Accounting    |   | type              : Non-profit
2011-01-03 11:04:31.225288|INFO    |Accounting    |   | starting date     : Sun Jan  2 00:00:00 2011
2011-01-03 11:04:31.225317|INFO    |Accounting    |   | ending date       : Mon Jan  2 00:00:00 2012
2011-01-03 11:04:31.225345|INFO    |Accounting    |   | max virtualservers: 10
2011-01-03 11:04:31.225367|INFO    |Accounting    |   | max slots         : 32
2011-01-03 11:04:31.276239|INFO    |FileManager   |   | listening on 0.0.0.0:30033
2011-01-03 11:04:31.482715|INFO    |VirtualServer |  1| listening on 0.0.0.0:6666
2011-01-03 11:04:31.489129|INFO    |CIDRManager   |   | updated query_ip_whitelist ips: 127.0.0.1,
2011-01-03 11:04:31.490194|INFO    |Query         |   | listening on 0.0.0.0:10011
2011-01-06 14:02:39.699891|INFO    |ServerMain    |   | Received signal SIGTERM, shutting down.
2011-01-06 14:02:39.774250|INFO    |VirtualServer |  1| stopped

server logs are fine too, no signs for an attack or ddos

[Baraan]

I noticed quite a similar problem, see my thread from July last year: http://forum.teamspeak.com/showthread.php?t=56445

I find the high interrupt count especially peculiar, do you notice something similar on your system?
I would still be concerned running the full 500 users, but for my small user base this is no problem. But if somebody would knew something, I would be curious for a solution, because the servers the VM is running on do a lot more without any load issues.

[syscore]

Hmm, i don't know if these interrupt counts are normal or not.

the host's eth0 has around 10k interrupts/sec when bandwidth is maxed out (100 mbit, nightly backup to ftp).

the VM looks quite normal i think, a high count of local timer is normal. (i reconfigure the VM to 1 GB RAM)

[Screech]

I've had a similar High CPU usage reported by my VM status page since I updated to beta30. Not found the cause. Even more confusing is that running top in the VM terminal it looks like the normal low CPU usage. Not found the reason that only while running beta30 my VM server's status page is showing this high usage.

[Baraan]

Just for the record: We both don't have high CPU usage even far from it, we have nearly no CPU usage. But we both experience high load.

[syscore]

I think the problem is associated with a high count of interrupt/context switches.

If i start the ts3-server, the number of context switches increases about 2000 per second. i don't know if that is a very high value, but another vm with a webserver, mysql, ftp, etc. pp. has only a number of 0-60 context switches per second.

maybe that is an problem of the ts3 server, but it could also be a circumstance due to the virtualization.

maybe i'll try that with my desktop, it has a core i7-860, the server has a i7-920.

[Baraan]

maybe i'll try that with my desktop, it has a core i7-860, the server has a i7-920.

And did you? i would love to know how it affected the load. And if someone runs a TS3 server on a vmware esx server with some more users (200+?) I would really like to know how the load looks like...

[syscore]

No, i had no time yet. But i will do this this week.

I found a possibility to lower the load by disabling the tickless kernel (boot ubuntu via grub flag: nohz=off, both vm and host).

now, the average load is smaller than before, as you can see on the attached monitoring stats. at the weekly graphic you can clearly see the point at which i enabled nohz=off

[Baraan]

Interesting, maybe I'll find time tomorrow to restart the server... Thanks.

[foobar_hh]

Same here, nobody seems to have a solution. Any hints? This is still happening with 3.0.5 on amd64 Kernel 3.3.4 (rolled my own..)

[dafire]

I have installed ts3 on a fresh installed server... nothing is on the server besides that idle ts3 server

top - 10:24:45 up 1 day, 17 min,  1 user,  load average: 1.23, 1.39, 1.35
Tasks: 106 total,   1 running, 105 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16320632k total,   607316k used, 15713316k free,    32736k buffers
Swap: 33553332k total,        0k used, 33553332k free,   213312k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
  818 teamspea  20   0  256m 7876 4692 S    1  0.0  18:45.19 ts3server_linux    
 4759 user1     20   0 17332 1304  960 R    0  0.0   0:00.21 top

the system is a ubuntu 10.4 (minimal install image)

Linux omikron 3.2.0-26-generic #41-Ubuntu SMP Thu Jun 14 17:49:24 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

I checked the log files and it looks like that the load was that high and there wasn't even a virtualserver running because I did not move the license key from the old server

==> ts3server_2012-07-04__08_07_29.780087_0.log <==
2012-07-04 08:07:29.841909|INFO    |ServerLibPriv |   | TeamSpeak 3 Server 3.0.6.1 (2012-06-29 07:59:05)
2012-07-04 08:07:29.842478|INFO    |DatabaseQuery |   | dbPlugin name:    SQLite3 plugin, Version 2, (c)TeamSpeak Systems GmbH
2012-07-04 08:07:29.861133|INFO    |DatabaseQuery |   | dbPlugin version: 3.7.3
2012-07-04 08:07:29.911153|INFO    |DatabaseQuery |   | checking database integrity (may take a while)
2012-07-04 08:07:30.534827|WARNING |Accounting    |   | Unable to find valid license key, falling back to limited functionality
2012-07-04 08:07:30.701191|INFO    |FileManager   |   | listening on 0.0.0.0:30033
2012-07-04 08:07:35.762749|ERROR   |VirtualSvrMgr |   | loading VirtualServer(1) failed to start, error: max slot limit reached
2012-07-04 08:07:35.779057|ERROR   |VirtualSvrMgr |   | loading VirtualServer(2) failed to start, error: max slot limit reached
2012-07-04 08:07:35.779644|INFO    |CIDRManager   |   | updated query_ip_whitelist ips: 127.0.0.1 
2012-07-04 08:07:35.779954|INFO    |Query         |   | listening on 0.0.0.0:10011


==> ts3server_2012-07-04__08_07_29.780087_1.log <==
2012-07-04 08:07:35.769357|INFO    |VirtualServer |  1| stopped

==> ts3server_2012-07-04__08_07_29.780087_2.log <==
2012-07-04 08:07:35.779083|INFO    |VirtualServer |  2| stopped

[ARCHI]

2012-07-04 08:07:35.762749|ERROR |VirtualSvrMgr | | loading VirtualServer(1) failed to start, error: max slot limit reached
2012-07-04 08:07:35.779057|ERROR |VirtualSvrMgr | | loading VirtualServer(2) failed to start, error: max slot limit reached

Put valid license file in your teamspeak folder or change such records in your database.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
818 teamspea 20 0 256m 7876 4692 S 1 0.0 18:45.19 ts3server_linux

Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id

Can't see a problem with teamspeak using 1% of mem and 0.1% of cpu.

[dafire]

The license file is still active on the old server... I will move it to the new one when we switch the dns records.

load average: 1.23, 1.39, 1.35

is what concerned me a little ... while the cpu usage of server is at 0.1% it still has a system load > 1.