Over the last 2 weeks, we’ve been noticing some strange behavior on the /follows endpoint.
For some of the overlays we develop we use a custom follower notification service. When the overlay boots up we grab the current latest 25 followers and cache them. Then we set up a timer and keep on checking the /follows endpoint for new data. Once we receive data we do a deep data equality check to see if there is any difference in the array. This approach has worked for a while now, but on the October 13th, we started getting weird issues where we would have over 13000 followers reported per cast which was wrong. We identified that the issue was caused by the API response differing on a random number of consecutive calls.
For example the first 8 requests will receive the same response, but the 9th one will be different even though there was no changes in the followers for the channel. Most notably, the created_at property of the individual follow object will be off by 1 second and the user object’s property values will change from “” to null. It’s almost like the response is being load balanced on to another server which is responding with a different data model. This then causes our deep equality check for the local cache to fail and basically adds new 25 followers where they don’t exist, and then on the next request where the API response is standardized again, we add another 25 followers, essentially offsetting our follower metrics by 50 each time.
This was tested on a number of different twitch accounts both partnered and non partnered, with a number of different Client IDs, different request timer interval settings (anywhere between 10 seconds and 8 minutes) and different systems, to determine that the issue wasn’t on our end. We even rewrote the whole follower detection module to check we weren’t doing something wrong. Finally we wrote the test bench to see if we get the same issue, and it confirmed that we will randomly (sometimes it will take 4 consecutive requests, sometimes 9, sometimes 50 or 200) get different responses from the API even though there were no changes in followers.
An example of the API response differing is as follows:
API request #8
{ "created_at": "2016-07-28T17:10:44Z", "_links": { "self": "https://api.twitch.tv/kraken/users/someusername/follows/channels/somechannel" }, "notifications": false, "user": { "_id": 1234, "name": "someusername", "created_at": "2012-04-01T15:51:14Z", "updated_at": "2016-08-18T19:28:20Z", "_links": { "self": "https://api.twitch.tv/kraken/users/someusername" }, "display_name": "SomeUserName", "logo": null, "bio": null, "type": "user" } }
API request #9 (30 seconds later)
{ "created_at": "2016-07-28T17:10:45Z", "_links": { "self": "https://api.twitch.tv/kraken/users/someusername/follows/channels/somechannel" }, "notifications": false, "user": { "display_name": "SomeUserName", "_id": 1234, "name": "someusername", "type": "user", "bio": "", "created_at": "2012-04-01T15:51:15Z", "updated_at": "2016-08-18T19:28:20Z", "logo": null, "_links": { "self": "https://api.twitch.tv/kraken/users/someusername" } } }
From that example, it’s clear that the user model for the 2 responses differs but also, that the created_at time is 1 second later on the 9th response compared to the 8th response.
I couldn’t find this sort of behavior documented anywhere, and like I mentioned, it was not an issue before October 13th.
Thanks in advance.
