Different Results GET Reuests

API
1

I am running into an issue where when I try to make many requests with the API I will get a different result every time.

I built a simple python function that simply tries to get all the pagination links for a user.

def get_pagination_links(page_url, pagination_arr):
headers = {
‘Client-ID’: ‘MY_ID’
}

# Make a URL request with the predefined headers above
r = requests.get(page_url, headers=headers)

# Store the request in a json/dictionary object
jsonobj = r.json()

if 'pagination' in jsonobj:
    cursor = jsonobj['pagination'].get('cursor')
    pagination_arr.append(cursor)
    get_pagination_links("https://api.twitch.tv/helix/videos?user_id=31723226&after="+cursor, pagination_arr)

Then i just print the length of the array I have been building which is just a list of all the pagination links.

Here is where it gets weird. The code will run just fine but I get a different number of links each time.

The first time I get like about 30, but if i run it again right away I get 2.

There must be a number of requests you can make in a given time. Do anyone know what the problem is, or some kind of workaround?

2

I cannot replicate.

I called the URL https://api.twitch.tv/helix/videos?user_id=31723226 a handful of times and got 20 records each time.

Are you saying on page one you get 20 and page two you get less than 20, and then call page three and get more results?

3

I am saying sometimes I get 32 pages and other times i get 4.

I keep getting 32 though if i wait for like 10 minutes, so i think it is time related.

Also the call you are doing there only does a single page and defaults to 20 videos on the page. That is why you get 20 I am trying to get all of the pagination links.

ALSO THANKS!

4

Ah!

I’ll run some spot tests. I miss interpreted your inital message.

But that is odd.

5

I’m up to 55 pages and still going with 20 records per page.

on a second run it’s being a little sluggish about it, some pages just take longer to return than others. (It jsut got up to 59 pages and still going, just every few pages it goes sluggish)

I wonder if you are getting a “timeout” and not logging the timeout.

So you are getting four pages and the fifth page is timing out and you are not catching the timeout as a timeout?

Edit: I got to 287 pages

6

I was running in increments of 100.

So 287 * 20 = 5740
and for me
32 * 100 = 3200

So you got more than i did.

Not sure why that is the case.

I am not checking for a timeout… I will check that. Good catch!

7

At 100 per page I get 57 pages and the 58th has 35. So 5735

(at 20 per page the last page was 15 records)

Code: nodeJS

const got = require('got');

const client_id = 'testClientIDthatIhadtohand';

var counter = 0;
function fetch(p) {
    counter++;
    var url = 'https://api.twitch.tv/helix/videos?user_id=31723226&first=100';
    if (p) {
        url += '&after=' + p;
    }
    got({
        url,
        headers: {
'Client-ID': client_id
        },
responseType: 'json'
    })
    .then(resp => {
        console.log('Got', counter, resp.body.data.length, resp.body.data[0].id);
if (resp.body.pagination) {
    fetch(resp.body.pagination.cursor);
}
    })
    .catch(err => {
        console.error(err);
    });
}

fetch('');
8

Really strange. I am wondering if it is a python limitation.

I added this block.

Make a URL request with the predefined headers above

try:
    r = requests.get(page_url, headers=headers, timeout=3)
except Timeout:
    print('The request timed out')

It made no difference and i never got an exception.

9

I ran your code and i was able to confirm your results. I was actually planning to switch to nodejs anyways so I think I am just going to abandon the python code and chalk it up as a learning experience.

THANKS AGAIN!

one more add:
I was able to get my python code to work. It was not timing out, but it was just getting a bad response. So i basically added a while loop that just keeps trying until it gets a good response. Sometimes it just has to try a few times.

s_code = 429
while s_code != 200:
# Make a URL request with the predefined headers above
r = requests.get(page_url, headers=headers, timeout=1)
s_code = r.status_code

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.