PSA: Chat servers are going to migrate to AWS EC2 servers

There were some issues over the weekend and we transitioned everyone back to the main cluster.

This morning, we’ll collect a list of tasks to complete before we start moving non-Staff accounts over to the AWS cluster. After we have that, we’ll get a better idea of when we can expect the first batch of partners to transition.

Before we transition any partners, we’ll enable the new cluster on the staff accounts again.

Is there any way to get a list of impacted users made public (When you are about to launch), so we can validate that our processes correctly switch over (so automatic detection vs what is actually expected) - That way we can flag/give feedback about issues we see as third parties?

We’ve resolved a couple issues with some commands not working. We’re working on making subs notifications and the SERVERCHANGE message work.

According to our current progress, we may have a chance to transition a small number of partners over this week.

Does this mean some logic/RAWs may have changed between main to AWS?

Once a channel gets moved to the new servers do we have to rejoin the channel on the new server to get the messages or can we already join now?

I currently run a chatlog bot for a friend, so can I just run a second instance of my bot connected to the new servers to basically have no downtime whenever the channel gets moved?

You can join now.

What will happen to irc.twitch.tv? Is it possible to make irc.twitch.tv point to the new DNS once all chats are moved?

Also would it be possible to keep us updated on when everyone will be moved 100%?

finally SSL for raw tcp, added SSL socket cluster for AWS and just parse the cluster on manual rejoin. Shouldn’t be any problem for people to move over :slight_smile:

My own migration plan is to join all 3 clusters (main, event, aws).
That avoids needing to call /chat_properties for every room join, and avoids needing to worry about which cluster to join/when to join a cluster.

@scottrc We’ll keep this thread updated.

You can use these:
irc.chat.twitch.tv:80 (raw, no ssl)
irc.chat.twitch.tv:443 (raw, ssl)
irc.chat.twitch.tv:6667 (raw, no ssl)
irc.chat.twitch.tv:6697 (raw, ssl)
irc-ws.chat.twitch.tv:80 (websockets, no ssl)
irc-ws.chat.twitch.tv:443 (websockets, ssl)

Would it be possible to get a heads up the day before (weekday before?) the migration happens?That way if you plan to do it at night or over the weekend I can plan to be on watching to make sure my processes migrate smoothly? Would love you long time for it <3

Just an update, I’m deploying a change to drop messages that are “sent from the wrong side of the AWS fence” in preparation for the migration.

In other words, a message sent to a channel on the “aws” cluster to a server not on AWS (e.g. irc.twitch.tv) will be dropped. Similarly, a message sent to a channel not on the “aws” cluster to a server on AWS (e.g. irc.chat.twitch.tv) will be dropped.

Bots can do one of two things to make sure they send messages to the right server:

  1. Check the “cluster” field in https://api.twitch.tv/api/channels/CHANNEL_NAME/chat_properties and connect to the right server per channel.
  2. Connect to all clusters as @xangold mentioned above (both tmi.twitch.tv and irc.chat.twitch.tv)

Thanks!

Hey everyone,

We’re planning to shift a small number of partners to the new cluster later today for testing. Be sure to implement one of the two solutions @arcoraven posted above! If we run into any problems, we’ll reach out to help get you connected.

1 Like

Thanks for the heads up!

Do you have a list of who, or what type of criteria would qualify a user for the migration?

We’re still gathering a list of volunteers – we expect around 10-20 will be included in this test. Once this test concludes, we’ll start migrating people over in larger numbers. So, no list of users yet, but if you’re connected to the new cluster you’re good to go!

From a partners perspective - what are the benefits of this change. Specifically, are there pros that would validate opting into the testing over the possible downsides?

There’s no benefits to you (only to Twitch), but there are downsides (like third party tools like bots or alerts being incompatible). If you use any of my tools, I’ve spent today integrating support for this change so it should be a seamless switchover for anyone who opts in.

Great, thanks for clarifying

There are some advantages:

  • Some spammers might not have connected yet (brief seclusion)
  • The ensure they won’t “randomly break” in the future when we do switch everyone.
  • Of course as night mentioned, some 3rd party tools currently aren’t working with new servers yet.
  • You get to help us in our process, and have a direct feedback.

Update: a wider rollout is starting today!
We’ll be moving as many as 200 partners to the new servers. Let us know if you have any trouble getting your bots connected!

Glad to see chat is finally getting SSL. Now add IPv6 (for chat AND the website)