I have a list of 1300 broadcasters I’ve been gather for research. On average they have roughly 113,000 followers. I’m attempting to recreate the pagerank algorithm to rank these broadcasters for a research paper. I was wondering what would be the best way to gather the names of all the followers for each broadcaster. Then get the number of followers each of them have. I know the “rate” is 1 call/second and this would require millions of api calls so I was wondering what would be the best way to go about this. Would it be to make x amount of calls then sleep for x seconds and repeat or is there a better way?
So that’s something like 18200 calls. If each call is called consecutively and takes 1 second, it would take a little over 5 hours. At a more realistic 132ms per call, it could take 40 minutes.
I’m using the cursor to get all the followers. I’m not using offset.
With 1300 broadcasters at 113,000 followers on average I’m looking at 146.9 million calls overall.
In my bot, for limited use, I do sleep a second between calls, to be safe, when trying to rebuild follower lists. However, with the number of calls you are doing, you may want to do it more efficiently. I assume you are doing something with the data after you retrieve it from Twitch, possibly some JSON extraction or something and then placing into a database.
Did you consider taking the system time, in milliseconds, after the call to Twitch, then again after you have performed any data operations, and then sleeping for the difference (or not at all if the data processing takes longer than a second)? That could at least reduce your “downtime” that is incurred with sleeping if you are processing data, or doing some other unit of work, in between calls to Twitch.
I am sure too, if you took all the correlated data, that you would find some users that overlap from the channels and then reduce your queries some, I don’t know by how much, but I would think some.
I’m also sure there is overlap but I still need the name of every follower per broadcaster. The overlap only helps when I have to make an api call on the followers to determine how many followers they have. I can do a toy example to see how long it takes to add get the follower name and add it to a set by list comprehension.
This is a lot of data. Are you storing it? Make sure you’re within our storage guidelines of our Developer Agreement. The best way is to make some requests and then have a cooldown period as others have mentioned.