Helix API Pagination Issues

Hey all. Hopefully I am just missing something here, but I am trying to paginate in reverse. From what I can tell you should just be able to input the cursor, but for some reason the cursor that is returned is always an after cursor. Let me explain.

When you do a follower query, you get a cursor like eyJiIjpudWxsLCJhIjoiMTUwNjE1MjYzNTUzODA3ODAwMCJ9.

Upon decoding the Base64 string, you can see that it gives you a before (b), and an after (a) key {"b":null,"a":"1506152635538078000"}. To use this one of the keys must be null, and the key that is populated must coincide with the appropriate call. ie. If you call a before, the b key must be populated, and the a key must be null, and vice versa.

Now, we can do some magic on it, and flip the values around before sending it but the value wont give back the results you want. ie. A cursor returned is always the after value, so if you take that value, and place it in the before call, it will return the exact same results.

So my question is this: Why doesn’t the api detect a before call, and populate the b key for you instead? Also, what is this key? Is it an unknown ID on the database entry that we have no way of ever returning? A timestamp of some kind? Knowing what this is might allow me to create the value I need myself, and not rely on getting the cursor from the previous call.

Any clarification on this would be amazing.

At a guess

A is the next page
B is the previous page

If you were on page 2+

Yes it looks like a timestamp, But there shouldn’t be any reason for you to be decoding the cursor, to paginate you just send the cursor, the cursor is configured by the database for a snapshot of the results when you made the request for page 0. [citation missing]

Do NOT go around encoding your own cursors, you won’t get any results as the API will go to the DB and say “hey give me the page for this cursor” and the DB will go “I have no idea about this cursor, it’s not in my cursors table/cache”

The next call, uses the cursor of the previous call in order to get the next page.

For followers though, you might want to look at Webhooks instead?

1 Like

You would think that would be the case, but I have checked the cursor on multiple pages, and only the a key is populated.

I agree. I was more so doing it to try to understand what is going on. Seems strange that the b key is never populated.

I guess there is no way to query for followers, and then query again later to only get new followers. I was thinking you may be able to store the cursor, but that seems like a no go (I had no idea the cursors were generated on a per call basis). I also thought maybe you could query followers, and stop once you get one you have already stored locally, but this doesn’t account for someone unfollowing, and then following again.

Seems like the only sure fire way to get new followers (without reading them all every time) is to use web hooks, which means you must have a web server storing them all. Since I was coding something that was meant to be run locally, I may have to rethink a few things. I could have a web server catching follows, and then query that web server with my app instead I guess. It’s just an extra layer I was hoping to avoid.

This still has issues of being out of date if for some reason the server crashed, and before it could reboot someone followed. You would likely need to periodically still loop through them all to double check that (or handle that on server boot to double check your data is still good).

Thanks for the info!

Thats what most people do. And store a list of people we “sent a notification to our overlays for in the last 24 hours” to account for unfollow/refollow

This is a mistake in the documentation for User Follows, right now only forward pagination is supported.

I’ll look into this :slight_smile:

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.