Periodic message lag spikes in group cluster

Chat and IRC
1

This is starting to get on my nerves here. Since the mass migration of everyone’s main chatrooms to AWS last week, I’ve noticed significant lag with group chat servers (and whispers) for at least every minute, regardless of how often I sent messages or whispered. It would take about a full second or more to be able to see my bot’s messages and whispers. (Yes, I run a bot, but mainly for my group chatrooms.) My friends in chat noticed roughly the same amount of delay too, and I even notice the same lag of at least that much time with other bots, including, but not limited, to “tpp”, the current main bot in the TwitchPlaysPokemon stream.

I was quite frustrated by this (because of how poorly Twitch is maintaining group chat servers the past month or so, IMO and no offense) that I decided to do some tests with my username and bot logged in at the same time. Using and my IRC client HexChat and my bot’s program, I first found out that USERSTATE messages are sent at the same time either user sees the message. For about every minute or so, I noticed my messages would take about a half-second on average to show up. Then, I was able to implement a way to measure USERSTATE lag, both in my bot’s code and in HexChat as a plugin. Here’s what I got on an empty group chatroom with no users joined there, and yet the result is the same in all of my other chatrooms:

KG6XTSt.png585×875 19.2 KB

The result is lag spikes for least every minute or more as I originally suspected, depending on how often I sent messages. Some smaller lag spikes within that 1-minute interval occur too as I continued to chat after the tests, up to about 3/4 of those lag times. I also got the same result with different IPs, implying that the entire group server cluster is affected and others might notice it with bots that normally have relatively low latency with the servers. I was also aware that this gets at its worst during the evening hours (U.S. time), and I was able to get a full second of USERSTATE lag once!

And here’s the result in the main chat cluster, not surprisingly:

I have a really fast and stable cable internet BTW, so I think this is more of a server-side issue to me.

But as you can see here, the lag times in the group cluster compared to the main cluster is so great it never got any faster as main chat did like before chatdepot.twitch.tv (the server responsible for maintaining everyone’s group chatrooms) went down for almost a full day earlier this month (March). So I concluded that something is making those processing delays every minute or so. I’ve emailed support about this 3 days ago and haven’t got any response from them as of writing. I’m pretty disappointed and still annoyed by this, although I do understand and respect Twitch’s plans to ditch group chat in the future entirely.

Does any of you still using group chat notice this same kind of lag too? And will it ever be fixed eventually? Any suggestions?

TL;DR: Group chat (and whispers, because they probably use group servers to get them sent) is lagging for a week now, and I’ve confirmed it by measuring time between sending a message and getting USERSTATE in return, as shown in the screenshots above. (Getting USERSTATEs from Twitch after sending every message in IRC chat is completely normal, as described here.) Now it’s about to become annoying, at least for me, and I’m curious what’s causing this to happen.

2

OK, I dunno why no one is paying attention to this very thoughtful and respectful report/complaint, but as of now I’m starting to see the lag getting worse. Is this never going to get fixed because group chat is about to be ditched for good soon?

3

You know what, I’m totally fed up with this and nobody is listening. Thanks a lot Twitch, I give -1 star for the performance of the group server cluster now.

4

Hey there!

I am a fairly new user here, but reading through several threads and also finding yours I currently understand that the Chat servers are being migrated, here is a link to the thread:

PSA: Chat servers are going to migrate to AWS EC2 servers

I am not sure if this effects what you are saying but I thought as I believe it fits into the context of what you are complaining it may help out.
<3

5

I only know that this started to appear every day since that time, and as already stated in that thread group chat isn’t moving to AWS, so I’m quite perplexed how the migration somehow affected group chat and whispers in an unexpected way. This is why I posted my findings on group chat lag and how it’s likely not going to be fixed.

And after my initial post, I’ve heard of reports that some whispers sometimes got eaten up, so apparently this situation has started to escalate even further.

6

Hi,

I’d like to thank you first off for a great post, along side supporting information. We really appreciate well thought out posts like this!

Group chat is not yet being processed on AWS. Some components of our chat infrastructure are located on AWS now (as you have mentioned). Additional round trips per request increase the latency to send a message, connect, etc. If this was the cause of these specific spikes you have noticed, we have a plan to reduce them :smile:

Do you have additional information about the latency you have recorded? Do they happen periodically (every 30 seconds for example), or do certain actions always take longer the first time? Some increased delay might be due to our caching infrastructure not having a value, or whether they require a write vs. a read.

Thanks,
Xangold

2 Likes
7

Thanks for reaching out! I noticed at least every 60 seconds or more and sometimes within that 60 seconds, regardless if you talked more than once per minute or not, these spikes occur. They start to come in at least 60 seconds after creating a connection to the server, no matter how many channels you join or what kind of CAPs you initially requested. Strangely, pings and pongs are perfectly normal, getting lag times within the range of 40-80 ms between the servers and my location, which is pretty normal for me, although I got even lower ping times before the migrations began. I can connect to this cluster fine, yet I can stay connected for many hours or even days like before the migration began. (And I’m not saying that I’m having any issues connecting to the new AWS servers and staying connected.)

Unfortunately, the only way to measure this kind of lag is to write a script, whether be for HexChat, a custom-made bot script, etc., that measures the time between the exact moment you hit the Enter/Return key and getting the USERSTATE reply. If you want to take a look at the source code for my HexChat Python script that does this and try it out, I’ll gladly provide it in this thread.

8

Update: I connected to group chat for the first time today just now, and to my amazement whisper lag with my bot appears to be eliminated! In addition, group chat messages no longer appeared late as described in my earlier posts! I’ll keep you guys updated if this annoying lag comes back anytime soon. Thank you so much Twitch Staff, including you @xangold!

So to conclude this topic for the time being, what caused this kind of lag to appear? Nevermind, I guess the migration to AWS fixed it after all

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.