TS3-server: too many interrupt-requests & context-switchings (bug or feature?)

[Rhinox]

I discovered this by chance: While trying to reduce power consumption of my server I found it difficult to keep cpu in low-power mode. Quick checking of vmstat-output showed I'm having too high values for in/cs (interrupt-requests, context-switchings). And to my big surprise, Teamspeak3-server (using the last version, 3.0.5) is single cause for this.

Without TS3-server running my in/cs numbers are ~25/25. Even these numbers are quite high for tickless-kernel, but there are a lot of other services running (i.e. ssh, cron, ntp, syslog, apache, mysql, vsftp, sendmail, bind) so it is understandable. But as soon as I start TS3-server, those numbers explode to ~400/1300 and even higher! And that all is with "idle" TS3 (no voice/query users connected).

obelix ~ # vmstat 5 -S M
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0  13628    274   1199    0    0     0     6   25   21  0  0 100  0
 0  0      0  13628    274   1199    0    0     0     0   20   17  0  0 100  0
 0  0      0  13628    274   1199    0    0     0     0   25   24  0  0 100  0
 0  0      0  13625    274   1199    0    0     0     0  291 1153  0  0 100  0 <= TS3 started
 0  0      0  13625    274   1199    0    0     0    36  318 1338  0  0 100  0
 0  0      0  13625    274   1199    0    0     0     2  277 1303  0  0 100  0
 0  0      0  13625    274   1199    0    0     0     0  516 1502  0  0 100  0
 0  0      0  13625    274   1199    0    0     0     0  357 1363  0  0 100  0
 0  1      0  13625    274   1199    0    0     0    28  484 1458  0  0 100  0
 0  0      0  13625    274   1199    0    0     0    10  504 1493  0  0 99   1
 0  0      0  13627    274   1199    0    0     0    11  652 1311  0  0 100  0
 0  0      0  13628    274   1199    0    0     0    26   26   25  0  0 100  0 <= TS3 stopped
 0  0      0  13628    274   1199    0    0     0     0   20   17  0  0 100  0
 0  0      0  13628    274   1199    0    0     0     0   21   17  0  0 100  0

So I'd like to ask: why is TS3 server causing ~400 interrupts and ~1400 context switchings per second?

These are crazy high values for single process, I have never seen something like this before. I turned every possible service off suspecting there is some inter-process collision between TS3 and any other service, but to no avail. Those numbers were still the same: in/cs extremely high. And before someone might ask: I have quad-core and 16GB RAM on my server, it almost never touches swap.

Quick check of /proc/interrupts revealed it is system-timer (IRQ 0) which is called so frequently:

obelix ~ # cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:     107189     107397     106950     108063   IO-APIC-edge      timer
...
(after ~5 seconds, with TS3-server running, ~2000 IRQ more!)
...
  0:     107780     107981     107535     108640   IO-APIC-edge      timer
...
(after further 30 seconds, without TS3-server running, just ~600 IRQ more)
...
  0:     107921     108115     107672     108780   IO-APIC-edge      timer

So I suppose somewhere in TS3-code you are using very short timer-value (less than 0.003s), this expires frequently and consequently calls IRQ-0 (system-timer) ~300 times per second. But I can not imagine why TS3-server needs to call timer >300 times per second. I do not know how is it in windows-world, but for linux common timer-values nowadays are ~1 (system-timer called once per second, or even less frequently)...

[Peter]

Hi,

it is true that we use rather a lot of context switches, but one thing you must understand is that being a "real-time" communication system the TeamSpeak 3 server has quite a few low timeout timers, keeping track of various connection parameters. This is something we can improve, but I don't think we will ever be anywhere close to values of apache/ssh/cron, where timing is not as critical as with real time voice communication.

[Rhinox]

There is some point in what you are saying, but those timeout-timers you use are extremely low. If server is waiting for connections, there is hardly any need to check some parameters so frequently. I could force my apache to make >1000 context switches per second, but *under serious load*. It seems to me those timers in TS3-server are hard-coded, they remain the same no matter if there are some clients connected or not.

Moreover, "lag" (partially caused by network delay itself, and partially by TS-clients/server) is much higher than a few miliseconds. In this case "real-time" communication means 0.1-1s delay. Compared to this, context-switching somewhere in level of 1kHz does not seem to be adequate. 10 Hz would be more than enough (at least in idle-mode).

I'm not sure if it is allowed here; just for comparison I tested two similar voice-communication servers: ventrilo and mumble/murmur (I know, different quality, features, licenses, etc, but all of them are "real-time" comm software too). None of them caused so high values for IR and CS. Actually, I did not observed *any* increasing of IRQ/CS when running only vent/mumble servers. Interesting info can be found using "powertop", with all three voice-comm servers running together (all "idle", no clients connected, it might be different under load):

Top causes for wakeups:
  96.7% (1079.6)   ts3server-bin
   2.3% ( 25.2)   [kernel core] hrtimer_start_expires.clone.6 (tick_sched_timer)
   0.2% (  2.2)   mysqld
   0.1% (  1.3)   [ahci] <interrupt>
   0.1% (  1.1)   apache2
   0.1% (  1.0)   ntpd
   0.1% (  1.0)   monit
   0.1% (  1.0)   ventrilo_srv
   0.1% (  0.9)   [eth0] <interrupt>
   0.1% (  0.8)   [Rescheduling interrupts] <kernel IPI>
   0.0% (  0.5)   events/0
   0.0% (  0.4)   cc1plus
   0.0% (  0.3)   [Function call interrupts] <kernel IPI>
   0.0% (  0.3)   flush-9:4
   0.0% (  0.2)   flush-9:8
   0.0% (  0.2)   sendmail
   0.0% (  0.2)   init
   0.0% (  0.1)   [kernel core] __mod_timer (neigh_timer_handler)
   0.0% (  0.1)   murmurd
   0.0% (  0.1)   flush-9:5
   0.0% (  0.1)   bdi-default
   0.0% (  0.1)   flush-9:7
   0.0% (  0.1)   [kernel core] __mod_timer (md_safemode_timeout)

TS is excelent software I have been using it for many years and I will stick with it. It is just something strange with all those system interrupts and context switches. There is too much system overhead and too many wasted cpu-cycles. I hope your devs will look at it one day...

[Teddy]

I nearly forgot this thread, but today I checked if something changed since then and it did indeed!

# vmstat 5 -S M
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0   2509    157    134    0    0     0     7    5    7  0  0 99  1
 0  0      0   2509    157    134    0    0     0     2    4    6  0  0 100  0
 0  0      0   2509    157    134    0    0     0     0    6   12  0  0 100  0
 0  0      0   2505    157    134    0    0     0     0   55  234  1  0 99  0    <- TS3-server started
 0  0      0   2505    157    134    0    0     0     7   53  218  0  0 100  0
 0  0      0   2505    157    134    0    0     0     9   50  216  0  0 100  0
 0  0      0   2505    157    134    0    0     0     0   49  216  0  0 100  0
 0  0      0   2505    157    134    0    0     0     2   52  220  0  0 100  0   <- TS3-server stopped
 0  0      0   2508    157    134    0    0     0     8    5    7  0  0 100  0
 0  0      0   2508    157    134    0    0     0     0    3    6  0  0 100  0
 0  0      0   2508    157    134    0    0     0     0    4    8  0  0 100  0

Still quite high numbers of in/cs, but reduced by factor ~6 since the top post! Thank you TS-devs, good job!
Problem solved, thread can be closed.