Paging Consistency

API
1

The Issue

So now that yet another API has been announced even though it seems like v5 has only just been worn in, I have a few concerns with paging in general. In the past, and even right now, almost every endpoint handles paging differently, in a few regards.

Requesting Each Page

Some endpoints such as get-channel-followers have a cursor paging parameter; a nice, easy string to keep paging until it is empty. However, many other endpoints such as get-channel-videos don’t have a cursor paging parameter to follow, instead it only has an offset parameter. This behavior isn’t inherently bad. With the pages that return a model with a total field, getting all pages/elements is trivial, but not all models have one, an example of this is the get-all-teams endpoint. In the scenario where there is no cursor paging parameter to follow and no total field in the model to calculate the number of pages to be returned, the only option left is to keep incrementing offset until an empty page is returned. However, this leads into another major issue, the offset behavior in different endpoints.

Offset Behavior

Because endpoints are worked on by separate teams, the same parameters in different endpoints can behave extremely different. There seems to be some “maximum” that each endpoint enforces, some endpoints enforce this as an arbitrary cap and others as the amount of elements that can be returned. What’s even worse than this is how some endpoints behave when this max is exceeded. Some endpoints return a blank page, some return an error, some return the previous valid page requested, and some even overflow/wrap around to the beginning and return what appears to be a “valid” page. In situations where you have to poll the endpoint until an empty/null page is returned and there is no way to tell how many pages should be returned, this can lead to some very bad situations as you might expect.

Solution

What can be done to combat these issues is what I hope is obvious. standardize the way endpoints handle paging and paging parameters. Personally, I believe each endpoint should have have cursor and offset parameters, and each page result should include a _total field just like get-channel-followers. The behavior of the offset isn’t of much concern to me, just make it consistent.

TL;DR

Every endpoint handles paging differently, which causes a lot of unnecessary head aches. Every endpoint should be standardized to handle paging and parameter behaviors the same way to make everyone’s lives easier.

This conceptually should also extend to the entire API, and not just paging.

2

Isn’t that exactly what they are doing? The post on the new API even specifically mentions pagination as one of the examples and that for consistency all endpoints will use a cursor for it.

3

In theory, yes. But they’ve made claims like this before and not much has come of it. Besides, I figured it wouldn’t hurt to bring to light what is an issue rather than have the dev team go on a witch hunt.

4

I mean, the blog post specifically says

Creating a consistent experience

Because we want to provide a consistent experience with our API, we made some decisions that apply across the board:

  • All APIs will use a cursor for pagination. Once you understand pagination for one endpoint, you should understand it for all endpoints.
  • Parameters will always be in the query string. This will allow flexibility for us to support new functionality without changing the base route.
  • We will have bulk GET calls for our resources. You can request up to 100 users, streams, or other base resources. (Note: bulk calls are not available for specific relationships, such as whether or not user X follows user Y).
5

I think it’s safe to assume you didn’t read my first reply.

6

Your entire post is about lack of consistency, you even bolded where the solution to your complaint is to standardize and make it consistent. The blog post specifically says they are standardizing pagination so it’s the same for all endpoints. What am I missing? There’s really no “witch hunt” as you call it as they’ve already called out the inconsistency as something to fix.

7

No, they are not lying in the blog post, the pagination is consistently using cursors.

8

That’s good to hear. Now I guess it’s just a waiting game for it to go live.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.