Forum

Results 1 to 11 of 11
  1. #1
    Join Date
    June 2018
    Posts
    16

    [Server >= 3.3.0] The New Query System

    We all know that our query system can be best described as an functional exercise in a spartanic user experience. But for a long time we could not improve on the existing query for fear of breaking compatibility with existing tools that connect to the server using the query.

    When we decided that it is finally time to encrypt the query traffic, we took the opportunity to improve the user experience without sacrificing the comparability to existing tools.

    Implementing SSH gave us the opportunity to completely re-implemented the query from the ground up and making sure the new query code is more flexible, faster, and allowing for future expansion. We wanted that users should feel right at home when connecting to the server query using a interactive ssh query. We did this by adding proper handling of terminal escape sequences(for example: arrow keys), tab completion, a search-able history and Emacs style shortcuts.

    While all these features are great for users working directly with the ssh query, the very same features make it hard to implement for tools for the new ssh query. This is why we implemented a second mode into ssh exclusively for tools. This mode can be reached by allocating a shell without pty and allows access to good old query but encrypted with ssh. We hope this second mode allows for an easy migration of tools to use ssh encryption when possible, without changing much of there existing code base. To make even easier we will publish examples on how to connect to the ssh query using python and php upon release of version 3.3.0.

    When we implemented the new server query, we took a long look at the flooding and banning systems of the current server query, and found it rather lacking. They are too complex to implement on server side and on the client side, additionally it punishes flooding too readily and harshly. So we changed it.

    The first big change: no more disconnects, when flooding the server. Instead every line sent, that triggers the flooding system, will instead result in a flooding error, and not actually be processed by the server. Additionally, the server gives connected clients a little bit more leeway when triggering the flooding system. Only when a client is abusing the server, and is not slowing down when receiving the flood errors, will the server step in and soft ban the client. Unlike a regular ban, a query ban will not result in a disconnect, instead the client will not be able to send commands for a given period of time.

    Here is how the new query flooding system works:
    When a client connects he is given a quota of 10 allowed commands he can send. Every command being send decreases that quota by one, but at same time the quota regenerates at a rate of 10/3s. Meaning a client that connects can send 10 commands immediately and an additional command every 0.3 seconds.

    Should the client exceed that quota he will get a 'error id=524 msg=client\sis\sflooding extra_msg=please\swait\s1\sseconds' message. If he choose to ignore this errors and continue to send commands, then the server will escalate and increase the timeout from around 1 second to 5 minutes.

    Behind all of this is a new infraction system. Just like the command quota it has a maximum(3) and regeneration rate(-1/1s). Every time the server rejects a command due to flooding he increases this number by one. Once the rounded up infraction count has reached three, will the server reset the flood wait time to 5 minutes.

    Meaning that once a client has triggered the soft ban due to flooding, he must send 1 command per second or less, for the duration of the ban. Of course any command send while being soft banned will be reject, but the client will see how long the ban is still in effect. Warning, all commands that were rejected due to flooding will still be deducted from the allowed quota, thus lengthening the ban time by 0.3s for every command send.

    Many of the values described above can be configured using the server query itself.
    • The quota can be changed by editing SERVERINSTANCE_SERVERQUERY_FLOOD_COMMANDS. Default: 10 commands
    • The speed at which the quota regenerates can be changed by editing SERVERINSTANCE_SERVERQUERY_FLOOD_TIME.
      Default: 3 seconds
    • The length of the soft ban can be changed by editing SERVERINSTANCE_SERVERQUERY_BAN_TIME. Default: 300 seconds


    Please let us what you think about the new server query, and what we can still improve to make it even better.

  2. #2
    Join Date
    December 2004
    Location
    RF
    Posts
    2,996
    I think if you want to improve something in this area, the format of the server replies could be improved. I.e. JSON.

  3. #3
    Join Date
    June 2018
    Posts
    16
    Sorry that is not possible. While we understand the need to make the output of the query more readable for the user, i don't think json is the right answer.
    For one it would be a breaking change for every tool and guide that exists, because the commands and there responses would have be changed syntactically.
    On the other side, i do not see how json would substantially improve the user experience when people are using the query directly.

  4. #4
    Join Date
    November 2017
    Location
    Cologne, Germany
    Posts
    155
    Quote Originally Posted by mmuenchow View Post
    Sorry that is not possible. While we understand the need to make the output of the query more readable for the user, i don't think json is the right answer.
    For one it would be a breaking change for every tool and guide that exists, because the commands and there responses would have be changed syntactically.
    On the other side, i do not see how json would substantially improve the user experience when people are using the query directly.
    TeamSpeak has always been good at finding excuses to keep the status quo even if that means living in the past and I'd be surprised if we'd ever get to see something from you guys that's actually truly innovative or even "well designed".

    JSON is the de-facto standard data-interchange format of the web and instead of actually thinking about the suggestion and where it may come from and how it could be done the first thing we get is "Sorry that is not possible" and the feeling that you didn't even understand the context this suggestion was made in.

    You probably don't get what people are actually using your "query interface" for. It was designed as just that: a query interface like Telnet. You could get a bit of information from it and it was never designed to be used for automation. But what are people supposed to use to get information from and send information to their TeamSpeak server then when a "telnet look-alike" is all we got?

    Imagine what could be done if the TeamSpeak server actually had a true API with a standardized response format that most tools could parse out of the box rather than implementing their own parsers for your home-grown response format.

    That's what people are talking about. It's 2018 now and TeamSpeak still doesn't have a proper interface to talk with web applications. Really?

  5. #5
    Join Date
    June 2011
    Location
    Germany
    Posts
    4,319
    If you are developing a web application, the data format is the smallest step you have to take – by spending 5 minutes on it. And maybe another 5 minutes when you find out that serversnapshotdeploy -new has an output format different from all other functions. If your browser cannot connect to raw TCP, why would you bother about the output format being not a format that is supported by the browser?

    I doubt the query interface was ever made to be used by hand, e.g. being unable to correct spelling mistakes (until 3.3.0 fixed it only in SSH) makes it unfeasable for any use other than maybe creating a virtual server.

    Maybe one could be able to change the data format to JSON by some command.

  6. #6
    Join Date
    December 2004
    Location
    RF
    Posts
    2,996
    Quote Originally Posted by mmuenchow View Post
    Sorry that is not possible. …
    Translation: "No, we don't know what that is and do not want to think if and how it could be implmented, we just stick(stuck) with what we have and you have to deal with it."

    You know, by this point in time (read: after ten years of observing TS3 development), I already have a very, very low expectation of potential features and usability improvements in TeamSpeak. So it does not surprise me even a small bit.

    Though, SSH query was a surprise. But lack if public key authentication was not.

    P.S.
    A simple "format=JSON" parameter to all commands turning response format to JSON would do.
    You don't have to thank me for idea. It was floating in the air.

  7. #7
    Join Date
    June 2018
    Posts
    16
    Thanks for all your answers, this will be a longer post as i will answer each of you.

    Quote Originally Posted by RandomHost View Post
    TeamSpeak has always been good at finding excuses to keep the status quo even if that means living in the past and I'd be surprised if we'd ever get to see something from you guys that's actually truly innovative or even "well designed".

    JSON is the de-facto standard data-interchange format of the web and instead of actually thinking about the suggestion and where it may come from and how it could be done the first thing we get is "Sorry that is not possible" and the feeling that you didn't even understand the context this suggestion was made in.

    You probably don't get what people are actually using your "query interface" for. It was designed as just that: a query interface like Telnet. You could get a bit of information from it and it was never designed to be used for automation. But what are people supposed to use to get information from and send information to their TeamSpeak server then when a "telnet look-alike" is all we got?

    Imagine what could be done if the TeamSpeak server actually had a true API with a standardized response format that most tools could parse out of the box rather than implementing their own parsers for your home-grown response format.

    That's what people are talking about. It's 2018 now and TeamSpeak still doesn't have a proper interface to talk with web applications. Really?
    On your point that we will always find good excuses not to do something: I can not answer the general case, as i am not not teamspeak as a whole. But for the answer that prompted this reaction, please consider that i have given honest and, in my eyes justified, reason, why changing the format of the query system to json is not a good idea. If you find my reasoning faulty, please tell me why, and we can have a civil discussion.
    I also have to disagree with you that json is THE de-facto standard. That honor should fall to the ml-language-family. It is everywhere.
    As for your suggestion for adding a rest api to the teamspeak server: That was not asked in the question, nor have i rejected such an api. It is in fact on our to do list, going so far as already adding some needed fundamentals to make the implementation happening. If you think this an important feature. Please make a post and tell us what your idea is. We are, in all honestly, interested in any and all suggestions.

    Quote Originally Posted by numma_cway View Post
    If you are developing a web application, the data format is the smallest step you have to take by spending 5 minutes on it. And maybe another 5 minutes when you find out that serversnapshotdeploy -new has an output format different from all other functions. If your browser cannot connect to raw TCP, why would you bother about the output format being not a format that is supported by the browser?

    I doubt the query interface was ever made to be used by hand, e.g. being unable to correct spelling mistakes (until 3.3.0 fixed it only in SSH) makes it unfeasable for any use other than maybe creating a virtual server.

    Maybe one could be able to change the data format to JSON by some command.
    The query interface was initially made with little regards for readability for non programmers, this is true. But things don't always end up where there have started, and i think the query interface has the potential to be more then a poor mans rest interface. As for adding a mode switch that changes the syntax of the query: i don't see the benefit of this.
    Json is not a syntax that lends itself well in interactive environments like the query. Should all the communication be two whole json objects that start with '{' and end with '}',
    or should the server only respond in json and the request remain the same? There are a lot of open questions, and the benefit is still not defined, except for 'it uses json'.

    The reason why we added escape character handling only to the ssh shell is simple: We didn't wanted to break everyone's scripts, and tools by making the existing query a proper telnet connection.

    Quote Originally Posted by ANR Daemon View Post
    You know, by this point in time (read: after ten years of observing TS3 development), I already have a very, very low expectation of potential features and usability improvements in TeamSpeak. So it does not surprise me even a small bit.

    Though, SSH query was a surprise. But lack if public key authentication was not.

    P.S.
    A simple "format=JSON" parameter to all commands turning response format to JSON would do.
    You don't have to thank me for idea. It was floating in the air.
    I am sorry, if we have disappointed so far.
    The reason why we did not include public key authentication in the release of ssh query is one of time and complexity. Implementing the ssh query turned out to be a much larger undertaking that initially expected and 'nice to haves' like public key authentication had to be left out for now. We instead focused on improving the speed and stability of the query. Additionally the public key authentication has problems when it comes to its implementation. How should the ssh keys be generated and saved? Inside the database? Using query commands? If yes, how can we bootstrap this process? If no, maybe add another file like the black/whitelist that contains the keys? What about multiple keys per client?
    As you can see there are a few open questions and i personally wanted to see if the community has an actual interest and if yes, what suggestions it has, before we decide on a course of action.

    As for the "format=JSON" suggestion: First, thank you for your suggestion. Second, how should json be implemented in an interactive environment and what benefits will json have?

  8. #8
    Join Date
    December 2004
    Location
    RF
    Posts
    2,996
    JSON is not usable in interactive environment (although it MAY be more readable at times, but that's incidental property), but it's very much welcome in a non-interactive one.
    It fits your current specification in several ways at once:

    1. It has a clearly defined format suitable for streaming;
    2. It has consistent encoding that forbids explicit control characters within stream (pretty printing is an optional feature), although it allows for whitespaces, the format itself guards them well;
    3. The JSON stream can be decoded on the fly with no backtracking, means, pull parsers are easily implemented.
    4. It has explicit UTF-8 support.

    The only downside of the format that I often stumble upon is that it has no binary payload support. But you already work around it by base64-encoding binary payloads.

    On the receiving end, the main characteristic property of JSON is that it can be easily decoded and validated by proven and tested tools. Almost every given language has means to encode and decode JSON, in several different ways at times.
    C++/C#/Lua/Perl/PHP/Python/Ruby… even JavaScript and PowerShell have JSON support!

    What to lack of pubkey support… I can always ssh -W and stuff manual login into connection properties auth script. Less than ideal, but works like a charm.

  9. #9
    Join Date
    November 2017
    Location
    Cologne, Germany
    Posts
    155
    Quote Originally Posted by mmuenchow View Post
    On your point that we will always find good excuses not to do something: I can not answer the general case, as i am not not teamspeak as a whole. But for the answer that prompted this reaction, please consider that i have given honest and, in my eyes justified, reason, why changing the format of the query system to json is not a good idea. If you find my reasoning faulty, please tell me why, and we can have a civil discussion.
    I think we got a communication problem here. Maybe it would have been wiser to say that the query interface is not the right place for JSON and suggest an alternative approach which could potentially satisfy that request. The way your post is written it sounded more like a general refusal of JSON as a whole because "it's not possible".
    Quote Originally Posted by mmuenchow View Post
    I also have to disagree with you that json is THE de-facto standard. That honor should fall to the ml-language-family. It is everywhere.
    Maybe it depends on what you are trying to do. For me, things like good old XML are just a bit too heavy-weight in most cases. Sure, you got all the fancy DTD validation stuff and can do a ton of things with it but how often do you actually need that and how many people actually take the time to take full advantage of what XML can do?

    I'm not talking about enterprise level super reliable triple-checked financial transaction data or whatever here, I'm talking about small things like a list of users on my TeamSpeak server or thinks like Amazon's Alexa Voice Service (which uses JSON).
    Quote Originally Posted by mmuenchow View Post
    As for your suggestion for adding a rest api to the teamspeak server: That was not asked in the question,
    Actually, it wasn't even a question. It was a very vague suggestion and I assumed it to be more of a general feature request to add something "like" a query interface that actually speaks JSON. Maybe we should all put a bit more effort into our wording here in order to avoid these misunderstandings.
    Quote Originally Posted by mmuenchow View Post
    nor have i rejected such an api. It is in fact on our to do list, going so far as already adding some needed fundamentals to make the implementation happening. If you think this an important feature. Please make a post and tell us what your idea is. We are, in all honestly, interested in any and all suggestions.
    The problem with that "To Do" list is that it probably exists as a physical or digital item somewhere in your office but the general public doesn't have a clue what you are planning since TeamSpeak as a company has the habit of making a big secret out of those things. You are giving us little to work with so it's kinda natural for us to assume that "Sorry that is not possible" means "Not going to happen" instead of "We consider adding something similar in the future" especially since you guys are IMHO really not the best at making people feel that their feedback is truly appreciated.

    So to summarize all of this, there is chance of a new API being added some time in the future (whenever that may be) that is going to be aimed towards being more of an actually API rather than a "telnet command line you can script if you REALLY want to".

    PS: I'm not trying to insult anyone, but I think TeamSpeak could really use a community manager of some sort who knows how to deal with customer feedback, who acts as a proxy between developers and customers, who knows the product, can help to clear up misunderstandings and moderate where appropriate.

  10. #10
    Join Date
    January 2019
    Posts
    1

    new TS query system over SSH

    Hi,

    Could you please explain me about the working of new TS query system over SSH, and if it's secured password, ssh key ?

    Thank you!
    Last edited by dante696; January 25th, 2019 at 10:35 AM. Reason: merged

  11. #11
    Join Date
    June 2008
    Posts
    17,711
    It's quite simple.

    - The whole connection is secured, the password itself has nothing to do with that. Its send over that secure connection.
    - You use a SSH client to connect instead of using a Telnet client.
    - The server generates a new key (ssh_host_rsa_key) on first start or/and allows you to add your own instead.
    When sending me private messages: Please make sure to include reference link to your forum thread or post.

    TeamSpeak FAQ || What should i report, when i open a client thread?

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Server ban system / new identy
    By Reeze in forum General Questions
    Replies: 2
    Last Post: August 4th, 2010, 05:55 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •