I’m one of the developers responsible for the chat system at Twitch. We’re hoping to significantly increase our activity on these forums to help support developers making awesome features using chat.
This year we’ve spent most of our time reducing the number of dropped messages from chat. We hope you’ve noticed the significant improvements that have been made towards that goal. Although there is still more to do, sending messages is more reliable now than ever.
Soon, message reliability will no longer be the most critical issue with the chat system. We’re starting to plan what we can do to improve the chat system further. We hope to answer questions, collect your feedback (complaints, suggestions, requests) on the current state of the chat system, as well as notify you of changes (proposed or pending) to the chat system that may affect you.
So, myself and others on the team will be checking these forums regularly and I wanted to introduce myself so that we can start the discussions and continue improving chat.
Not getting those blaring alarm sounds telling me the chat is derping has been great for my sanity, thank you!
Thanks to these stability fixes I can now join more chats as well, since I don’t have to rejoin them as often.
On the note of improvements to chat I personally feel getting one-to-one communication through chat would be awesome.
As the site grows the amount of messages in chat grows with it, and trying to reduce the unnecessary messages, like bot replies and triggers, would be a great benefit.
Like, say you do @moobot !command in chat, only Moobot will read it, and it will reply with @moocat Command response without anyone else actually seeing any of that.
To reduce dev time this could even be some sort of web client side ignore, where it would ignore messages not meant for you. Then it would already be affected by channel boundaries, slow mode, message rate limiting, and moderator status out of the box, without any extra work required. Although having it be truly one-to-one would make it better scale to something like sending a welcome message to every user that joins.
Just imagine a chat being as normal while a giveaway is being conducted through chat
I have some things I’m wondering about regarding the rate limits for chat.
If you send more than 20 commands or messages to the server within a 30 second period, you will get locked out for 8 hours automatically. These are not lifted so please be careful when working with IRC!
In what scope is this? Per connection, per IP, per channel?
Mods can type 100 messages in a channel before hitting this rate limit. If a message is sent to a room in which your global message count is already too high, you will be disconnected.
This hints that the rate limits are separate for each channel…? (and with disconnected, I guess this is the same 8h lock-out)
JOINs are rate-limited to 50 JOINs per 15 seconds. Additional JOINs sent after this will cause an unsuccessful login.
What exactly does it mean with unsuccessful login? I’m already logged in, does it mean that the JOIN is unsuccessful, or am I disconnected? Does this also apply the 8h lock-out?
Different wording on each point; “locked out”, “disconnected”, “unsuccessful login”
Good to have input from someone who actually knows this stuff.
Edit: Also, is there a connection limit for opening multiple connections to the chat from the same IP?
Thank you Brildum for your hard work! The recent stability of the chat has not gone unnoticed I assure you of that.
I definitely second Moocat’s private message idea. Something like that would be fantastic.
I’d like to expand on what Sunspots mentioned. I see these type of questions pop up all the time regarding message/API rates. It would be nice if all this information was in one place and clearly explained. (If it is disregard this.) Things like: message rate limit, command rate limit, join/part rate limit, IRC ping limits, API limits, and if they’re per ip, username, public or authorized api requests, etc. plus all available IRC commands.
edit: oh and of course would really like twitchnotify sub messages to come back to the base/default twitchclient
Our code to deal with abuse (rate-limiting, etc) is subject to change periodically, but we should definitely have some better docs. I think it may be useful for us create a github chat docs repository specifically focused on the chat API. Twitch can push changes, but most importantly, you all can send pull-requests and/or issue reports to help us determine when the docs aren’t correct or vague.
The 20/100 message rate-limit is per connection, but the determination of whether 20 or 100 is used is per-message. Consider the case where a mod of channel ‘test’ sends 21 messages to room #test in 21 seconds. Since they are a mod on channel ‘test’, this will be allowed (100 message rate-limit). However, if their 21st message was instead to a different room ‘not_a_mod_room’ where the user is not a mod, their connection will have 21 messages (over the 20 limit for this particular message) and they will be IP banned. This is an artifact of this code originally being written with the assumption that connections only joined a single room, but its good to note this case for bots managing multiple rooms.
Regarding the JOINs being rate-limited, both authentication and JOINs are rate-limited together. A single IP is currently allowed only 50 JOIN/auths every 10s and if this limit is exceeded, you will be disconnected. This does not apply the 8h lockout.
There is currently no connection limit, but we plan on adding this.
So lets do some thinking with this line, because Twitch has already added so many limits to chat that this is seriously getting ridiculous.
Connections (auths) per second
Joins per second
Messages per second
There’s also a per-connection server-side message queue that kicks you out if 20 messages collect on the server, meaning that the number of channels you can join per connection without ill effects is rather low, especially if there’s any lag spikes.
Couple that with the sheer number of Twitch channels and the amount of channels bots are currently in, and we now have a blaring issue.
What are we noticing? Well, it looks like you’re trying to make running large-scale bots exponentially harder (or potentially kill them entirely). The only way to scale currently is horizontally with multiple connections.
There’s no doubt that even if you keep adding limitations there will always be ways around them. It’s simple for me to just buy a huge block of IPs.
If the goal in all of this is to prevent abuse, you will find out that this does nothing to prevent abuse. Maybe there’s the sheer chance you’ll lessen it, but lets be honest here: abusers utilize proxies and hacked machines. The only people you’re hurting here are the third-party developers.
A fair point, but I think that adding restrictions to avoid abuse isn’t a problem by itself, only if it limits legitimate uses (which it certainly does currently, as you mention). We have discussed solutions for this and likely will create a thread to get feedback from bot developers.
One of our current thoughts is allowing bot developers to apply for increased limits, making it more difficult for malicious users to affect chat while allowing bot developers to avoid having to jump through hoops to make their bots work.
There is currently no connection limit, but we plan on adding this.
There are already a ton of hurdles you have to jump through just to get to the starting point of connecting, joining a channel and then sending a message. That’s literally weeks of work just to get through those chat restrictions put in place there.
Now with this proposed change chat developers will have to spread everything over thousands of proxies just to get to where we are today…
Is Twitch comfortable with having the credentials of a user that is a moderator in hundreds of thousands of chats get sent over unsecured proxy connections? Cause that’s where this will lead us, and that’s just one of the many issues with this.
And as @night says, this wont stop abusers cause they are literally sitting on the proxies, cause that’s what they do. Just see where your account creation restrictions got you, with these people easily making hundreds of thousands of Twitch accounts.
As the others have said, this would basically kill off all multi-channel bots on Twitch. There would need to be some kind of exemption for legitimate bots.
Abusers would still get around it since they’re sitting on all the proxies, and they don’t have any need for security nor low latency on their chat connections.
Legitimate white listed user’s growth would be limited severely by the limits set by the white list.
Legitimate users that cannot restrict themselves to the white list (or lack of white list) would have to use the cheapest possible solutions, most likely setting themselves up for a huge security disaster, and the service’s latency would take a huge hit, making the service considerably worse for users.
Other users with shared IPs, like college campuses, European ISPs, and some VPNs will be unable to connect to chat.
A white list would have to be moderated by Twitch personnel, which has not worked out in the past.
To lessen the effect of the restrictions all chat services would have to restrict who can use their service. That means moderately big channels that send a decent amount of messages would have to pay, and smaller users below say 100 viewers wouldn’t even be eligible for the service.
Due to the additional restrictions it would be highly unlikely that anyone in their right mind would start developing anything for chat, which would hinder any sort of competition. More than likely there will be just one service in the end, since everyone else has thrown in the towel, and the users will have no choice.
Hi,
I just spent some time on the ~27k people channel with no slow mode/facebook bullshit, chat wasnt going full-apeshit, but Lines-per-minute was quite high, balancing on the “impossible to focus on 1 message” point (~2 messages per sec?)
When watching it I came up with idea of - if the activity on the chat is high, it could enter mode when it refresh chat every time the new screen of messages is written. You could do this default mode in range it functions bests, but disableable in the options/profile option and channel options.
To make animation look better you could make sliding-down line where “image” refresh, kinda like: http://i.stack.imgur.com/AfeCw.png . You could also make arrows up - and - down make you jump 1 screen of histroy earlier, and one screen of history later (or make “pause chat” button/keyboard shortcut).
I agree with Night and Moocat, the rate limits are not the answer. The people who abuse and spam in chat will find away around the restrictions, and the legit users and mods will suffer. I have an idea that might work though, and I’m in the process of coding it into my bot right now; a user threat system.
Here is the short and dirty of it…
A spammer starts a script to join 1000 channels (within the rate limit numbers)
Knoobot is in use in 300 of those channels and flags that user Threat Level 1 for xx minutes for joining xx channels quickly.
The user starts spamming whatever (within the msg rate limits)
Knoobot flags this user as threat lvl 2 or higher and bans the user from all 300 channels for xx hours.
My hopes are that the system would block spammers before they reached 20 of the 300 channels the bot is in. The numbers would have to be adjusted and I plan on adding a streamer/moderator notify and approval system for the ban.
If twitch chat devs worked with us bot devs, we could create a anti-spam API that could have a shared threat system with our bots doing the reporting back to twitch.
Benefits:
No rate limit on registered bots,
Less spam making it into chat,
Little to no management on Twitch’s side,
Better bot performance for us
Sometimes registered bots spam chat at rates that can only be explained by a bug, so rate-limits are not only to prevent abusers but also to protect the chat system from good users too.
Re: your threat level feature
It seems very similar to our rate-limiting code (the logic is slightly different, but its a rate-limit: message-rate/user channel-rate/user) except its managed by third-party developers rather than Twitch, so its far more risky for Twitch imo.
You do present an interesting point though. Perhaps we can leverage third-party developers to notify Twitch of abusers automatically to enable easier detection/resolution of abuse.
I agree. We all make coding mistakes or bad logic sometimes, so the rate-limits are necessary even for registered bots. (Just maybe not as strict a rate-limit as we have now)
The difference between the current rate limits and what I propose is we will be able to distinguish the spammers from other viewers and prevent them from entering all of our bot monitored channels after they spam just a few. We could then share the threat alert info through twitch to the other bots so they can block them from all of their monitored channels as well.
Again, just an idea, and I appreciate any feedback. I will be adding this feature to my bot prior to launch to give my streamers some additional protection.
I had my bot doing this for about a day before the admin team was complaining (when spam bots were at an all-time high in chat, aka vine bots). It then became my understanding that admins already have similar tools that report names of some of these abusive bots, but that they don’t actually automate any bans and all accounts must be manually reviewed. If you can figure out a way to automate these bans, and if that’s such a impossibly feat even just make it a chat lockout instead of an account ban. Perhaps an x reports in y minutes triggers a chat lockout would be a good solution.
Putting site bans or lockouts in the hands of third parties is a bad idea imo. And Twitch should/are already monitoring this themselves.
On the note of rate limits, the API has no rate limits, and a developer can also create a bug here where the rate of requests degrades the performance of the API for everyone, but there are already safe guards in the API that should prevent this (by monitoring and revoking a client_id) The same safe guards could be used for chat.
And even if by some chance a bot had a bug that made it spam in chat, it could easily be removed by the broadcaster, just by simply removing, unmodding, or banning the bot.
And I can count several times in the past 2 years that the site has died because of a bug in third-party applications overloading our servers. That said, I definitely believe that third party developers should not be required to think about the rate-limits when implementing their applications. But I also believe that Twitch should protect our systems to prevent them from falling over.
One idea I’ve had is to partition a server specifically for registered bots and remove the rate-limits. If that server gets overloaded, it mostly affects other bots (not users). If that server falls over, we can remove access for a bot periodically and notify the developer.
The server partition is a great idea. +1 from me. @brildum would it be possible to “flag” viewers (not streamers or mods) who are spamming but staying in the rate-limits?
Detecting abuse is a very challenging problem. Spammers can easily use multiple accounts, multiple IPs, format their messages differently, etc…
Our goal is to eventually have better abuse detection, but we haven’t had time to prioritize it, instead our admin team has built their own tools to aid the fight against spammers, etc.
If different registered applications could have different rate limits, so that normal use is bound to one application and not every application, then an application could work within its own normal rates, and it would be rejected automatically in situations where a bug or abuse makes it have abnormal rates (in which case it would go over the set rate limit)
Higher rate limits could be given on a case by case basis, based on projected userbase or current peak rate.
You could then anticipate load, and give rate limits within the available resources.
I believe Google has something like this, where you can see how much you’re using within your application’s custom rate limit, and you can then expand this limit if you wish.
It’s a logical idea, but with a single point of failure it would have to be monitored and actually cared for by Twitch. Just looking at how the chatdepot service was entirely down for several days it seems very likely that if this were to happen to the bot server it wouldn’t be cared for cause it’s “not affecting most users”. And users shutting down bots so they can spam some streamer’s chat is nothing new (by DDoSing mostly)
