HTML response with 502 error from API end-point

API
1

Hello.

At Jul 30 20:21:33 UTC I’ve got the HTML response for API request:

Jul 30 23:21:33 : I, [2020-07-31T00:21:33.214532 #24791]  INFO -- request: GET https://api.twitch.tv
/kraken/streams/112353016?stream_type=live
Jul 30 23:21:33 : I, [2020-07-31T00:21:33.214627 #24791]  INFO -- request: Accept: "application/vnd.
twitchtv.v5+json"
Jul 30 23:21:33 : User-Agent: "Faraday v1.0.1"
Jul 30 23:21:33 : Client-ID: "secret"
Jul 30 23:21:33 : Authorization: "OAuth secret too"
Jul 30 23:21:33 : I, [2020-07-31T00:21:33.402674 #24791]  INFO -- response: Status 502
Jul 30 23:21:33 : I, [2020-07-31T00:21:33.402837 #24791]  INFO -- response: connection: "keep-alive"
Jul 30 23:21:33 : content-length: "138"
Jul 30 23:21:33 : server: "awselb/2.0"
Jul 30 23:21:33 : content-type: "text/html"
Jul 30 23:21:33 : date: "Thu, 30 Jul 2020 21:21:33 GMT"
Jul 30 23:21:33 : x-served-by: "cache-sea4479-SEA, cache-hel6820-HEL"
Jul 30 23:21:33 : x-cache: "MISS, MISS"
Jul 30 23:21:33 : x-cache-hits: "0, 0"
Jul 30 23:21:33 : x-timer: "S1596144093.223654,VS0,VS0,VE175"
Jul 30 23:21:33 : strict-transport-security: "max-age=300"
Jul 30 23:21:33 : I, [2020-07-31T00:21:33.402904 #24791]  INFO -- response: <html>
Jul 30 23:21:33 : <head><title>502 Bad Gateway</title></head>
Jul 30 23:21:33 : <body bgcolor="white">
Jul 30 23:21:33 : <center><h1>502 Bad Gateway</h1></center>
Jul 30 23:21:33 : </body>
Jul 30 23:21:33 : </html>
Jul 30 23:21:33 : #<Thread:0x000055af31f7a158 /home/my_project/src/twitch/channel.rb:37 ru
n> terminated with exception (report_on_exception is true):
Jul 30 23:21:33 : /home/my_project/.rbenv/versions/2.7.1/lib/ruby/2.7.0/json/common.rb:156:in
`parse': 783: unexpected token at '<html>\r (Faraday::ParsingError)

So, my code had the exception when tried to parse this HTML as JSON.

You have no rights to send any HTML from JSON API end-points.

2

502 errors are generally server errors meaning issue is on Twitch’s end. Just try again later. You said this was on the 30th at a very specific time. Is it still happening? If not ignore it.

3

It’s also worth modifying your code to test the HTTP response code first, before trying to JSON parse.

5xx’s codes normally won’t return as JSON response but a Generic “everything is super broken” HTML message, because everything is super broken. So just retry the request.

And you should catch any JSON parsing errors, as while most 4xx responses will return a JSON blob describing the error, they might not always.

Same for 200’s sometimes you’ll get a 200 but something went wrong and you didn’t get JSON back, so you should code to account for the possibility of not getting JSON back.

4

I don’t want to ignore it. Yes, it’s not happening right now. But it was, and then it can be in the future. I don’t want to get such exceptions anymore.

5

It’s non-sense. It’s also worth modifying your, Twitch, code to return JSON template (from nginx or something else) instead of HTML. For API end-points. It’s not too hard.

Why? Where is this “normal” described? Why HTML responses becomes “normal” for API end-points?

Why there is an HTML message at API end-point with JSON convention? Documentation says there will be JSON response. Is there any information about exclusive cases with HTML format?

So just change low-level error template from HTML to JSON.

Any documentation about these cases?

It’s also non-sense. You’re saying me “just retry a request”. And what next? What if something changed and there will be always “just retries”? These cases, HTML responses, are exception for JSON API, they should not be there in any time. If you could make HTML low-level error — you can make JSON template in the same way. So, please, do it. For API end-points (by domain, servers, anything else, as you want).

6

I am not Twitch, I am not Twitch staff, just another community developer.

If I get a 5xx error what I do next depends on what data I was trying to get, if it’s streams I’ll retry, if it’s something for an extension, I’ll report an error to the user/viewer and ask them to reload the page.

Most of the 4xx errors will be authentication errors, generally speaking, and you get a JSON blob back with a “message” describing the problem. It’s not directly documented no

Add a retry counter to your code and fail/error out after 2/3 goes?

Whilst you could in theory, when the server is 5xx it just wants to die, so the less work the server does the better for the server.

What happens if the server errors whilst attempting to generate the JSON?

If you want to raise this as a feature request (return JSON when nginx is dying/reading a bad response from the backend). You can make this request on the uservoice

https://twitch.uservoice.com/forums/310213-developers

7

When doing web dev, you will ALWAYS receive errors. You can’t avoid it. You need to know what each error means. There is a standard that the web follows. Not everyone uses them properly, but in general you can code based on the standard. When I do web dev, I always assume 1. The data doesn’t exist. And 2. I will have intermittent connection issues to the server or web in general.

1 Like
8

Not “just”, mister moderator.

Sorry, but I don’t care what you do in your projects. I’m waiting for changes from Twitch side.

I’ve asked about “they might not always.” Now you’re just repeating your words without this part.

Yes, it’s OK. But what if I make a request every 2 minutes? And such 502 downtime was for 10 minutes? My program will stop again. And this is bad. I want to receive JSON from Twitch instead of HTML when Twitch have to send JSON, this is not too hard.

Server can’t “want”. It died, like stopped.

JSON response will be lesser than HTML one.

Meh… is it generating the current HTML response? I don’t think so. Just plain text HTML file, you know? So… we can have the same plain text JSON file, can’t we?

OK, but for what is this portal, mister moderator?

Yes. But there may be more or less of them. We can affect on it.

Not at all, but again, we can avoid some of them, resolve some problems, you know? Like… if you have often internet-connection problems on your hosting — you can migrate to another hosting. Or if you have an HTML response where you have to have JSON response — you can rewrite HTML to JSON.

For what these words? I know. We’re not discussing “what does this error mean?”, OK?

What “a standard”?

Sorry, but I don’t care what do you always assume. And this discussion about HTML response at JSON end-point, you know? Not about data existence or connection issues in general. I’m tired of repeating, this is simple subject.

9

Yes the expected response from the API is JSON, but there is no guarantee that it’ll be the case all the time as there may be unexpected errors.

Because it’s not guaranteed to be JSON 100% of the time you as a 3rd party dev should design your app to be able to gracefully handle unexpected responses, such as HTML.

This is a discussion forum, as has been previously mentioned if you want to request a feature you should go through the proper process which is submitting it to uservoice.

Even if twitch was to make changes for the specific error you encountered it doesn’t mean that there may not be an error that for some reason results in a non-JSON reaponse, so don’t neglect proper error handling and best design practices by expecting Twitch, or any API, to conform to your specific apps lack of error handling. Continuing to argue with other 3rd party devs here on the forum won’t fix your code, or make changes to Twitchs API.

10

Yes, nothing is guaranteed at 100%, but we can change this simple response. I’m discussing this subject, this change, and I’m tired to repeat.

One more time: is it mentioned somewhere in Twitch API documentation?

No, I should not. API have to response with JSON when it declares that it responses with JSON every time when it can. I’m not discussing internet connection issues or something like this, but if an API server can response with HTML — it can response with JSON. So, let’s change it.

I see another information here:

image
image758×241 18.2 KB

And this is not “a feature”, this is the error (or “bug”, if you wish).

Yes, it doesn’t, but I’m tired to repeat, I’m not about some theoretical error, OK? I’m about specific error. JSON for 502 response instead of HTML. That’s it. It’s simple. I hope you know understand me.

I want to resolve any error properly, in the most correct way. If there is an authentication error — I’ll handle it in my application and refresh tokens. If there is an internet connection issue — I’ll discuss it with my hosting/provider. If there is an HTML response from JSON API — I’ll enforce to change it on API server side.

I’m surprised how well you see the future. I don’t want to argue with 3rd party devs. I didn’t call them. But if they (you) come and try to convince me that I should change my code instead of correct fix on server side — I’ll argue with this. I don’t see correct arguments, only some theoretical issues and non-100% connections and uptime. But I don’t care about this, I’m here discussing one specific problem, error.

11

No, and as there’s no explicit guarantees on the type of response for errors you shouldn’t just assume everything should be JSON.

Looks to me from the data you posted originally that the response declared that it wasn’t JSON, the response to your request specifically said it was text/html so if your app just assumes its JSON and errors when it isn’t, that’s on you. If the content type was erroneous then that’d be Twitchs fault, but you got what it said.

If you believe the service that is returning that reaponse is in error, then you should submit an issue on the twitchdev github so it can be tracked. If on the other hand the service is doing what its designed to do despite your app not gracefully handling it, the it would be a feature request to change the currently designed functionality of that service.

Again, please stop assuming that responses will always be JSON unless its explicitly guaranteed to do so. The intended responses will be JSON, as documented, but unexpected errors haven’t been explicitly guaranteed to be any specific format for all cases.

You are free to do whatever you like with your own code, even if it does go against best programming practices. While we are 3rd party devs, many of us have been working with this API for many years and some of us have worked for Twitch, so we’re trying to advise you on the best ways to properly submit feature requests or error reports as well as how you could handle such issues on your end and in the process develop better error handling to result in an overall better app design and reliability of your app. If you choose to take the advice of this community or not is up to you.

12

It’s better to mention this in the documentation. But this subject still not about guarantees, but about specific error.

This is correct point, thank you. We can forget about JSON parsing error, it’s actually on me, but the problem still in HTML response. There should not be HTML response. Yes, it’s the server error, but it can be changed, as I wrote.

OK, I’ll try to post this issue and on User Voice, and on GitHub project, but funny there are no links to them in the “Getting help” section of docs:

image
image777×320 20.5 KB

API service designed to send HTML errors? Seriously, this is intentional behavior? I don’t think so. I think Twitch developers just left default and forget about this, because these are (I hope) rare cases.

I know. And this is unexpected error, yes. But… we can just change it, so let’s do it. And then resolve other specific, not unknown theoretic, errors properly. Maybe another change on the server side, maybe some catches at a client side.

I didn’t asked “how can I handle such errors?”. I know how to handle, I know that I can handle, I know that this is a good practice generally. But I have specific error and I’m sure that it’s better to resolve it on the server side.

13

The links are on the support page. Support | Twitch Developers

14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.