Individual servers dying off

[Randall_James]

Not sure what to do here, I have individual servers that start acting as though they are not running. They show up on the list but no way to access them. Try to shut one of them down via the admin "server stop" gui and TS does a core dump and I have to do a "killall" via shell. Is there a log I might find with some data on what is going on?

RH9
v2.0.20.1
2.4.26-grsec
PHP Version 4.3.8

[CdnBacon]

Randall, care to download and then zip your server.log for us to see?

[mjrwoody]

I'm having a very similar issue. It cropped up about a week ago. Random TS servers just die off. If i access the web admin site as Superadmin, the rest go too.

I thought perhaps Database corruption, so i deleted my DB and started over, problem was back two days later.

The problem seems to crop up in "prime time" between 8pm-midnight. Is there a max # of servers supported, or max # of users that i might be hitting?

Let me know where to send my logs! I'm pretty desperate.

Server version: 2.0.20.1 Linux
RedHat Enterprise v3.0

[Randall_James]

Randall, care to download and then zip your server.log for us to see?

Firstly the Email notification on here sucks. Never works..........

I wont make the logs public but more than happy to let one of the admins here access to them. I do log more than basic data and so do not want that information made public.

[Randall_James]

I'm having a very similar issue. It cropped up about a week ago. Random TS servers just die off. If i access the web admin site as Superadmin, the rest go too.

I thought perhaps Database corruption, so i deleted my DB and started over, problem was back two days later.

The problem seems to crop up in "prime time" between 8pm-midnight. Is there a max # of servers supported, or max # of users that i might be hitting?

Let me know where to send my logs! I'm pretty desperate.

Server version: 2.0.20.1 Linux
RedHat Enterprise v3.0

Yes and then the restart can be a real problem as server_linux process still stay running and must be killed off manually this is truly a pain

These are the only odd ball lines I see in my logs over the last week or 2. However my logs are a couple megs, I will suspened all extra logging and see what I can get.

01-08-04 15:58:47,ERROR,All,TTSUDPSender.Execute, SID: 4 Exception1 EIdSocketError.Host not found.
01-08-04 15:58:47,ERROR,All,TTSUDPSender.Execute, SID: 4 Exception2 EIdSocketError.Host not found.

04-08-04 16:41:02,ERROR,All,TTSUDPSender.Execute, SID: 39 Exception2 EIdSocketError.Host not found.

[bystander]

Sounds like the same problem I've been having ever since we started using the current version weeks ago.

[Randall_James]

Ok here is everything from start to servers dying off, Looked tonight and at least 3 of them were dead

17-08-04 00:33:36,ALL,Info,server,	Starting VirtualServer id:24 with port:8789
17-08-04 00:33:38,ALL,Info,server,	Starting VirtualServer id:37 with port:8792
17-08-04 00:33:39,ALL,Info,server,	Starting VirtualServer id:36 with port:8777
17-08-04 00:33:40,ALL,Info,server,	Starting VirtualServer id:31 with port:8888
17-08-04 00:33:42,ALL,Info,server,	Starting VirtualServer id:38 with port:8773
17-08-04 00:33:43,ALL,Info,server,	Starting VirtualServer id:40 with port:8787
17-08-04 00:33:44,ALL,Info,server,	Server init finished
17-08-04 00:33:44,WARNING,Info,server,	TeamSpeak Server daemon activated
17-08-04 14:38:50,ERROR,All,TTSUDPSender.Execute,	SID: 3 Exception1 EIdSocketError.Host not found.
17-08-04 19:23:07,ERROR,All,TTSUDPSender.Execute,	SID: 37 Exception1 EIdSocketError.Host not found.
17-08-04 20:50:07,ERROR,All,CLIENT,	SID: 36 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 20:52:18,ERROR,All,CLIENT,	SID: 36 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 20:52:50,ERROR,All,CLIENT,	SID: 36 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:00:45,ERROR,All,CLIENT,	DBAddUser Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:01:23,ERROR,All,CLIENT,	SID: 36 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:01:48,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:06:45,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:06:58,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:07:15,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:07:26,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:07:49,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:10:14,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:10:25,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:11:34,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:11:52,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:13:38,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:22:40,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:23:00,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:45:35,ERROR,All,CLIENT,	DBAddUser Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:45:58,ERROR,All,CLIENT,	DBAddUser Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:46:40,ERROR,All,CLIENT,	DBAddUser Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:48:38,ERROR,All,CLIENT,	DBAddUser Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:48:40,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:49:14,ERROR,All,CLIENT,	DBAddUser Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:55:13,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:55:23,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:55:28,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:55:30,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:55:38,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:55:46,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:55:53,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:56:00,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:56:51,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:59:02,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 21:59:15,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:04:46,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:04:53,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:05:12,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:05:24,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:08:11,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:08:11,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:08:11,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:08:11,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:08:16,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:09:42,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:09:52,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:10:04,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:10:24,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:10:54,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:14:51,ERROR,All,CHANNEL,	SID: 37 GetChannelAttribs Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:16:02,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:16:11,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:17:44,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:23:05,ERROR,All,CLIENT,	SID: 4 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:23:19,ERROR,All,CLIENT,	SID: 4 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:23:35,ERROR,All,CLIENT,	SID: 4 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:23:54,ERROR,All,CLIENT,	DBAddUser Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:24:10,ERROR,All,CLIENT,	DBAddUser Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:25:13,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:26:57,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:30:47,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:30:55,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:32:28,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:33:12,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Operation Not Supported
17-08-04 22:33:17,ERROR,All,CLIENT,	SID: 37 DBValidLogin Exception: EDatabaseError.dbExpress Error: Invalid Username/Password
18-08-04 19:32:40,ERROR,All,TTSUDPSender.Execute,	SID: 4 Exception1 EIdSocketError.Host not found.
19-08-04 07:16:25,ERROR,All,TTSUDPSender.Execute,	SID: 39 Exception1 EIdSocketError.Host not found.
19-08-04 21:36:37,ERROR,All,TTSUDPSender.Execute,	SID: 4 Exception2 EIdSocketError.Host not found.
21-08-04 09:25:47,ERROR,All,TTSUDPSender.Execute,	SID: 35 Exception1 EIdSocketError.Host not found.
21-08-04 13:00:01,ERROR,All,TTSUDPSender.Execute,	SID: 3 Exception2 EIdSocketError.Host not found.
22-08-04 00:54:55,ALL,Info,server,	Server shutdown initialized
22-08-04 00:55:21,ERROR,All,Server,	RemoveAllClients Exception: EAccessViolation.Access violation at address 4140A9D0, accessing address 08017A64
---------------------------------------------------------------
-------------- log started at 22-08-04 00:59 -------------

[Randall_James]

So no help here? How many others are having this random die off anyway?

[mjrwoody]

I don't know what the "magic number" is, but i think we're hitting so active user limit. My servers die off at peak times (8pm - midnight) when the most people are on.

While i haven't found a fix to make this stop happening, i've found that just starting another TS on a secondary IP works like a charm. So i've dropped the usage of the first server down and moved servers to the secondary. A third is ready to roll when needed.

[Randall_James]

I don't know what the "magic number" is, but i think we're hitting so active user limit. My servers die off at peak times (8pm - midnight) when the most people are on.

While i haven't found a fix to make this stop happening, i've found that just starting another TS on a secondary IP works like a charm. So i've dropped the usage of the first server down and moved servers to the secondary. A third is ready to roll when needed.

Maybe it is just mine but I have also noticed that the servers that are dying off are also the "Public" servers. Our user servers are not having these problems. Is it possible that users have found a way to crash the individual servers?? Just a thought........

[mjrwoody]

My error is similar, but not identical:

01-09-04 21:14:05,ERROR,All,TTSUDPSender.Execute, SID: 1 Exception2 EIdSocketError.Socket Error # 101 Network is unreachable.

That all appears on a single line (one log entry). If i get into the web interface to restart that server (SID 1) it lets me login, when i click on "stop" my web page goes away (page not found) and the whole entire server crashes (all my TSes).

Only way to get it back it to stop the TS service and restart it.

[Randall_James]

My error is similar, but not identical:

01-09-04 21:14:05,ERROR,All,TTSUDPSender.Execute, SID: 1 Exception2 EIdSocketError.Socket Error # 101 Network is unreachable.

That all appears on a single line (one log entry). If i get into the web interface to restart that server (SID 1) it lets me login, when i click on "stop" my web page goes away (page not found) and the whole entire server crashes (all my TSes).

Only way to get it back it to stop the TS service and restart it.

Well the log entry is not identical but the problem most certainly is. I suppose that there is a problem here that is at least not my imagination...

[friedrooster]

I get the same error:

30-09-04 08:18:07,ERROR,All,TTSUDPSender.Execute, SID: 1 Exception1 EIdSocketError.Socket Error # 101^M
Network is unreachable.

it happens at the time my isp does its daily reconnect, before dyndns is updated...

hope this helps for a fix... kinda annoyin if my buddies get booted atm.

The Fryed Rooster

[Rooter]

I get the same error:

30-09-04 08:18:07,ERROR,All,TTSUDPSender.Execute, SID: 1 Exception1 EIdSocketError.Socket Error # 101^M
Network is unreachable.

it happens at the time my isp does its daily reconnect, before dyndns is updated...

hope this helps for a fix... kinda annoyin if my buddies get booted atm.

The Fryed Rooster

What operating system do you receive that on...RedHat Enterprise?

[naquaddah]

I get the same error:

30-09-04 08:18:07,ERROR,All,TTSUDPSender.Execute, SID: 1 Exception1 EIdSocketError.Socket Error # 101^M
Network is unreachable.

it's likely to get a "Network unreachable" message while the interface is shut down.
So this is not really related to the other problem.

but i'm not sure what causes James' problem