Experiencing connection failures/timeouts

We are experiencing an increased percentage of connection failures manifesting as TCP connection timeouts starting the afternoon of 2020-08-31 UTC. It appears to be specific to the route between SFO2 region of Digital Ocean and Twitch. We are using both Helix and v5 APIs and it appears to be affecting both. We are in contact with both Twitch and Digital Ocean support and it would help to know if anyone else is seeing similar issues whether from Digital Ocean or not. We are seeing the error rate hit as high as 10% since the start of this event where it was basically 0% prior. If anyone has experienced an increase number of TCP timeouts recently, please reply with network provider and any other details.

Thank you.

1 Like

+1 to that. Since 14:00 GMT yesterday, timeouts on all of our DO server calls to both Kraken and Helix. Everything runs perfectly locally. Haven’t done a deploy to master in 4 days. Our whole team is perplexed.

Error: connect ETIMEDOUT 151.101.190.214:443
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1136:16) {
errno: -110,
code: ‘ETIMEDOUT’,
syscall: ‘connect’,
address: ‘151.101.190.214’,
port: 443,
config: {
adapter: [Function: httpAdapter],
transformRequest: { ‘0’: [Function: transformRequest] },
transformResponse: { ‘0’: [Function: transformResponse] },
timeout: 0,
xsrfCookieName: ‘XSRF-TOKEN’,
xsrfHeaderName: ‘X-XSRF-TOKEN’,
maxContentLength: -1,
validateStatus: [Function: validateStatus],
headers: {
Accept: ‘application/vnd.twitchtv.v5+json’,
‘Client-ID’: ‘uoeyr64g1fynhsxbk7pnrr0mpucqjs’,
‘Content-Type’: ‘application/json’,
‘User-Agent’: ‘axios/0.18.1’
},
method: ‘get’,
url: ‘https://api.twitch.tv/kraken/streams?limit=100&channel=178459325,69189731,107141613,72453759,46719889,497666638,505086336,558971599,156646113,128491537,55950529,107165481,155634426,55477452,136199166,466348910,261118209,98631551,136934594,61668056,195120927,164042693,80442978,100893832,76893617,509626139,39368774,70505920,159118549,162426488,541325824,146686955,481553752,545177741,115086735,160831813,61704989,534819210,513696111,561558090,176349089,76928023,199247895,549065311,51372748,212011889,417141922,177131047,27028017,137717000,494437923,450740972,475523772,40412324,196791941,84574069,245291553,559180652,121324082,136558367,149971393,116262923,157118766,121788608,188758350,433736781,228567610,450797526,93196435,138633370,243161978,227322800,214650393,132988257,82817901,155609330,42952712,81943100,78811572,109759319,171537283,58852502,47268643,189934952,453883574,271763919,178826215,234771615,248488988,69642126,163370287,41732376,495412072,200050051,253102067,554671664,127023080,554887662,160026472,447637957&stream_type=live’,
data: undefined
},
request: <ref *3> Writable {
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: false,
decodeStrings: true,
defaultEncoding: ‘utf8’,
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
bufferedRequest: null,
lastBufferedRequest: null,
pendingcb: 0,
prefinished: false,
errorEmitted: false,
emitClose: true,
autoDestroy: false,
errored: false,
bufferedRequestCount: 0,
corkedRequestsFree: {
next: null,
entry: null,
finish: [Function: bound onCorkedFinish]
}
},
writable: true,
_events: [Object: null prototype] {
response: [Function: handleResponse],
error: [Function: handleRequestError]
},
_eventsCount: 2,
_maxListeners: undefined,
_options: {
protocol: ‘https:’,
maxRedirects: 21,
maxBodyLength: 10485760,
path: ‘/kraken/streams?limit=100&channel=178459325,69189731,107141613,72453759,46719889,497666638,505086336,558971599,156646113,128491537,55950529,107165481,155634426,55477452,136199166,466348910,261118209,98631551,136934594,61668056,195120927,164042693,80442978,100893832,76893617,509626139,39368774,70505920,159118549,162426488,541325824,146686955,481553752,545177741,115086735,160831813,61704989,534819210,513696111,561558090,176349089,76928023,199247895,549065311,51372748,212011889,417141922,177131047,27028017,137717000,494437923,450740972,475523772,40412324,196791941,84574069,245291553,559180652,121324082,136558367,149971393,116262923,157118766,121788608,188758350,433736781,228567610,450797526,93196435,138633370,243161978,227322800,214650393,132988257,82817901,155609330,42952712,81943100,78811572,109759319,171537283,58852502,47268643,189934952,453883574,271763919,178826215,234771615,248488988,69642126,163370287,41732376,495412072,200050051,253102067,554671664,127023080,554887662,160026472,447637957&stream_type=live’,
method: ‘get’,
headers: {
Accept: ‘application/vnd.twitchtv.v5+json’,
‘Client-ID’:REMOVED
‘Content-Type’: ‘application/json’,
‘User-Agent’: ‘axios/0.18.1’
},
agent: undefined,
auth: undefined,
hostname: ‘api.twitch.tv’,
port: null,
nativeProtocols: {
‘http:’: {
_connectionListener: [Function: connectionListener],
METHODS: [
‘ACL’, ‘BIND’, ‘CHECKOUT’,
‘CONNECT’, ‘COPY’, ‘DELETE’,
‘GET’, ‘HEAD’, ‘LINK’,
‘LOCK’, ‘M-SEARCH’, ‘MERGE’,
‘MKACTIVITY’, ‘MKCALENDAR’, ‘MKCOL’,
‘MOVE’, ‘NOTIFY’, ‘OPTIONS’,
‘PATCH’, ‘POST’, ‘PROPFIND’,
‘PROPPATCH’, ‘PURGE’, ‘PUT’,
‘REBIND’, ‘REPORT’, ‘SEARCH’,
‘SOURCE’, ‘SUBSCRIBE’, ‘TRACE’,
‘UNBIND’, ‘UNLINK’, ‘UNLOCK’,
‘UNSUBSCRIBE’
],
STATUS_CODES: {
‘100’: ‘Continue’,
‘101’: ‘Switching Protocols’,
‘102’: ‘Processing’,
‘103’: ‘Early Hints’,
‘200’: ‘OK’,
‘201’: ‘Created’,
‘202’: ‘Accepted’,
‘203’: ‘Non-Authoritative Information’,
‘204’: ‘No Content’,
‘205’: ‘Reset Content’,
‘206’: ‘Partial Content’,
‘207’: ‘Multi-Status’,
‘208’: ‘Already Reported’,
‘226’: ‘IM Used’,
‘300’: ‘Multiple Choices’,
‘301’: ‘Moved Permanently’,
‘302’: ‘Found’,
‘303’: ‘See Other’,
‘304’: ‘Not Modified’,
‘305’: ‘Use Proxy’,
‘307’: ‘Temporary Redirect’,
‘308’: ‘Permanent Redirect’,
‘400’: ‘Bad Request’,
‘401’: ‘Unauthorized’,
‘402’: ‘Payment Required’,
‘403’: ‘Forbidden’,
‘404’: ‘Not Found’,
‘405’: ‘Method Not Allowed’,
‘406’: ‘Not Acceptable’,
‘407’: ‘Proxy Authentication Required’,
‘408’: ‘Request Timeout’,
‘409’: ‘Conflict’,
‘410’: ‘Gone’,
‘411’: ‘Length Required’,
‘412’: ‘Precondition Failed’,
‘413’: ‘Payload Too Large’,
‘414’: ‘URI Too Long’,
‘415’: ‘Unsupported Media Type’,
‘416’: ‘Range Not Satisfiable’,
‘417’: ‘Expectation Failed’,
‘418’: “I’m a Teapot”,
‘421’: ‘Misdirected Request’,
‘422’: ‘Unprocessable Entity’,
‘423’: ‘Locked’,
‘424’: ‘Failed Dependency’,
‘425’: ‘Too Early’,
‘426’: ‘Upgrade Required’,
‘428’: ‘Precondition Required’,
‘429’: ‘Too Many Requests’,
‘431’: ‘Request Header Fields Too Large’,
‘451’: ‘Unavailable For Legal Reasons’,
‘500’: ‘Internal Server Error’,
‘501’: ‘Not Implemented’,
‘502’: ‘Bad Gateway’,
‘503’: ‘Service Unavailable’,
‘504’: ‘Gateway Timeout’,
‘505’: ‘HTTP Version Not Supported’,
‘506’: ‘Variant Also Negotiates’,
‘507’: ‘Insufficient Storage’,
‘508’: ‘Loop Detected’,
‘509’: ‘Bandwidth Limit Exceeded’,
‘510’: ‘Not Extended’,
‘511’: ‘Network Authentication Required’
},
Agent: [Function: Agent] { defaultMaxSockets: Infinity },
ClientRequest: [Function: ClientRequest],
IncomingMessage: [Function: IncomingMessage],

},
response: undefined
}

Thank you for confirming this issue. I’m having a hard time convincing either side that this problem is real. I have run tests from these networks/regions:

DO-sfo2 - has timeouts
DO-sfo3 - has timeouts
DO-nyc1 - no timeouts
EC2-us-west-2c - no timeouts
EC2-us-west-1 - no timeouts

1 Like

That squares, all of our DO regions are SFO2s.

EDIT:

Hey Eddie, thought you’d be curious - from DO support:

"Hey Mike,

The MTRs aren’t reporting any network issues to the Twitch API. You also aren’t the only customer reporting this issue – we actually investigated this for several hours over the last few days and we haven’t found any cause for this on our end – our routing to Twitch hasn’t changed in well over a week, and we haven’t implemented any changes in the san francisco networks either.

I’d recommend reaching out to Twitch and getting the MTR from their services back to your Droplet to find out if that reveals anything. This does look like it’s most likely caused by something on their end.

Thanks,"

Edit 3:

Did you just open a bug ticket to get your server MTR’d by Twitch?

Thanks Mike. I got basically the same response from DO over the last couple of days. I’m still waiting on Twitch to provide mtr back to my droplet. What’s interesting is I am able to reproduce this using netcat and simply doing a tcp connection is timing out about 5-10%. It just occurred to me that if I run the netcat test against another server that’s not twitch, I can at least see if the problem is specific to Twitch.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.