Get all followers for a lot of broadcasters simultaneously

I have a list of 1300 broadcasters I’ve been gather for research. On average they have roughly 113,000 followers. I’m attempting to recreate the pagerank algorithm to rank these broadcasters for a research paper. I was wondering what would be the best way to gather the names of all the followers for each broadcaster. Then get the number of followers each of them have. I know the “rate” is 1 call/second and this would require millions of api calls so I was wondering what would be the best way to go about this. Would it be to make x amount of calls then sleep for x seconds and repeat or is there a better way?

Thanks,
Sato

When using the offset parameter:

You’re limited to 1,700 individual followers from either side of a channel’s follower list.

Twitch API documentation reference

Here’s an example list of calls to make:

https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=100
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=200
...
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=1400
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=1500
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=1600

Only goes to offset 1600 with a limit of 100. And then in ascending order:

https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&direction=ASC
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=100&direction=ASC
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=200&direction=ASC
...
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=1400&direction=ASC
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=1500&direction=ASC
https://api.twitch.tv/kraken/channels/CHANNEL_NAME/follows?limit=100&offset=1600&direction=ASC

So that’s something like 18200 calls. If each call is called consecutively and takes 1 second, it would take a little over 5 hours. At a more realistic 132ms per call, it could take 40 minutes.

Edit: Use the cursor param instead.

FYI all followers can be accessed when using the cursor parameter instead of offset so the limit doesn’t really exist anymore.

I’m using the cursor to get all the followers. I’m not using offset.
With 1300 broadcasters at 113,000 followers on average I’m looking at 146.9 million calls overall.

Oh, welp. I’ve never been able to get cursor to work so I’ve completely forgotten about it.

In my bot, for limited use, I do sleep a second between calls, to be safe, when trying to rebuild follower lists. However, with the number of calls you are doing, you may want to do it more efficiently. I assume you are doing something with the data after you retrieve it from Twitch, possibly some JSON extraction or something and then placing into a database.

Did you consider taking the system time, in milliseconds, after the call to Twitch, then again after you have performed any data operations, and then sleeping for the difference (or not at all if the data processing takes longer than a second)? That could at least reduce your “downtime” that is incurred with sleeping if you are processing data, or doing some other unit of work, in between calls to Twitch.

I am sure too, if you took all the correlated data, that you would find some users that overlap from the channels and then reduce your queries some, I don’t know by how much, but I would think some.

I’m also sure there is overlap but I still need the name of every follower per broadcaster. The overlap only helps when I have to make an api call on the followers to determine how many followers they have. I can do a toy example to see how long it takes to add get the follower name and add it to a set by list comprehension.

This is a lot of data. Are you storing it? Make sure you’re within our storage guidelines of our Developer Agreement. The best way is to make some requests and then have a cooldown period as others have mentioned.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.