[SOLVED] Viewer.html/js Communicate with PubSub

Here’s the progress so far. I have ChromaSDK lighting effects sending from common.js and broadcasting on the PubSub. Viewers receive the PubSub broadcast messages which display on their Chroma enabled devices.

Common.js sends compressed data (5kb) every 1 second which is able to fit 10 animation frames in each PubSub broadcast.

I still need to know a few things.

1 Are there common.js callbacks for time so that I can sync the lighting effects with the video timestamp?

2 Is it possible to include PubSub data in the archived video so that video playback can still play lighting effects?

===

Below was my process for figuring out the examples and getting things to work…

===

Here’s the PubSub example.

Is it possible for an extension’s Viewer.html on the Twitch stream to talk with the PubSub?

It seems when I include the PubSub example, that creates an Auth button in the Viewer.html but the iframe is not allowed to redirect to authenticate.

I’m hoping that the Viewer.html can communicate with the PubSub. The broadcaster will be sending data via PubSub. And then the viewers will receive the data from PubSub which will talk with their viewer.html page.

It looks like the Broadcaster Viewer.html/js needs to communicate with my EBS. And the EBS needs to push to the PubSub.

Can the Viewer.html/js receive PubSub messages without the EBS for the viewer watching the stream?

I just noticed common.js which has auth and context events that I can pass to the EBS to authorize and send to PubSub for the broadcaster.

Looks like with common.js and the auth callback, I can detect if the Twitch viewer is the broadcaster.

if(window.Twitch.ext) {

  window.Twitch.ext.onAuthorized(function(auth) {
    console.log(auth);
    var parts = auth.token.split(".");
    var payload = JSON.parse(window.atob(parts[1]));
    var streamer_id = payload.channel_id;
	console.log("broadcaster", payload.role == 'broadcaster');
  });

Okay and here’s a modified common.js that can get pubsub messages. There’s no need to hardcode the clientId/token because that info comes across in the auth event.

/*
Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

    http://aws.amazon.com/apache2.0/

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
*/

/*

  Set Javascript common to all extension views in this file.

*/

// Source: https://www.thepolyglotdeveloper.com/2015/03/create-a-random-nonce-string-using-javascript/
function nonce(length) {
    var text = "";
    var possible = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
    for (var i = 0; i < length; i++) {
        text += possible.charAt(Math.floor(Math.random() * possible.length));
    }
    return text;
}

function heartbeat() {
    message = {
        type: 'PING'
    };
    $('.ws-output').append('SENT: ' + JSON.stringify(message) + '\n');
    ws.send(JSON.stringify(message));
}

function listen(topic, auth) {
    message = {
        type: 'LISTEN',
        nonce: nonce(15),
        data: {
            topics: [topic],
            auth_token: auth.token
        }
    };
    $('.ws-output').append('SENT: ' + JSON.stringify(message) + '\n');
    ws.send(JSON.stringify(message));
}

function connect() {
    var heartbeatInterval = 1000 * 60; //ms between PING's
    var reconnectInterval = 1000 * 3; //ms to wait before reconnect
    var heartbeatHandle;

    ws = new WebSocket('wss://pubsub-edge.twitch.tv');

    ws.onopen = function(event) {
        $('.ws-output').append('INFO: Socket Opened\n');
        heartbeat();
        heartbeatHandle = setInterval(heartbeat, heartbeatInterval);
    };

    ws.onerror = function(error) {
        $('.ws-output').append('ERR:  ' + JSON.stringify(error) + '\n');
    };

    ws.onmessage = function(event) {
        message = JSON.parse(event.data);
        $('.ws-output').append('RECV: ' + JSON.stringify(message) + '\n');
        if (message.type == 'RECONNECT') {
            $('.ws-output').append('INFO: Reconnecting...\n');
            setTimeout(connect, reconnectInterval);
        }
    };

    ws.onclose = function() {
        $('.ws-output').append('INFO: Socket Closed\n');
        clearInterval(heartbeatHandle);
        $('.ws-output').append('INFO: Reconnecting...\n');
        setTimeout(connect, reconnectInterval);
    };

}

if(window.Twitch.ext) {

  window.Twitch.ext.onAuthorized(function(auth) {
    console.log("auth", auth);
    var parts = auth.token.split(".");
    var payload = JSON.parse(window.atob(parts[1]));
    var streamer_id = payload.channel_id;
	console.log("broadcaster", payload.role == 'broadcaster');
    if (auth.token) {
        connect();
        $('.socket').show()
        $.ajax({
            url: "https://api.twitch.tv/kraken/user",
            method: "GET",
            headers: {
                "Client-ID": auth.clientId,
                "Authorization": "OAuth " + auth.token
            }})
            .done(function(user) {
                $('#topic-label').text("Enter a topic to listen to. For example, to listen to whispers enter topic 'whispers."+user._id+"'");
            });
			
      $('#topic-form').submit(function() {
        listen($('#topic-text').val(), auth);
        event.preventDefault();
      });
    }
  });
  
  window.Twitch.ext.listen('broadcast', function (topic, contentType, message) {
    console.log("topic", topic);
    console.log("contentType", contentType);
    console.log("message", message);
  });

  window.Twitch.ext.onContext(function(context, contextFields) {
    console.log("context", context);
    console.log("contextFields", contextFields);
  });
  
  window.Twitch.ext.onError(function(err) {
    console.error("err", err);
  });
  
}

Can the window.Twitch.ext.onAuthorized(function(auth) { authorization be used to authorize the PubSub? It would seem like the Viewer common.js would already be authorized since it is running on the Twitch site???

Next question??? What topic should be used?

And I’ll see about sending data on the PubSub to see if other viewers receive it.

Common.js can broadcast to PubSub directly with the following method:

window.Twitch.ext.send('broadcast', 'application/json', json);

Thanks,

~Tim Graupmann

It works!

In the end I didn’t even need a PubSub websocket. I was able to use the common.js events.

I used the auth callback to determine if it’s the broadcaster or not.

And then window.Twitch.ext.send is able to broadcast directly to the PubSub.

The window.Twitch.ext.listen callback receives the broadcasted messages in common.js.

Walla!

Basically all as documented by the Javascript helper send and listen functions:

PubSub has a 6k limit. Does Twitch have any built-in JS compression options?

I found this one which compresses 33684 bytes down to 11348:

With 7zip (the application) I can get 33k down to 5.9k. I need the same level of compression in JS.

Before Compression 18702
TestCompression.html:15 After Compression 10200
TestCompression.html:17 After Decompression 18702

Thanks,

~Tim Graupmann

FYI Pako had the best compression. I’m able to achieve 10 frames per second sending data across PubSub using only 5kb of bandwidth.

Here’s the post where I discovered the JS compression library.

Progress…

I do see one other issue and that is the PubSub broadcast messages are way faster than the video stream.

Is there a way a way to synchronize PubSub messages with the time index of live video?

Also is there a way for videos to save with the PubSub data so it can be played back with the archived videos?

It seems to me that the context callback should have had a video timestamp. It has whether the video has paused. The time index would be handy here but it’s missing…

hlsLatencyBroadcaster	number	Number of seconds of latency between the broadcaster and viewer.

That sounds like that might help keep the lighting effects in sync with the video.

Possibly, every viewers delay is different though, so your Extension could cache/delay pubsub processing of a message in a queue, until the right moment time.

No, but something could be written, extension side, but extensions don’t run/play on vod’s right now for that sort of thing.

Perhaps but that will be different for every user, and it’s only available in a video extension now a panel extension

Does this require the user to download something? Or is the keyboard listening to Chrome/WebBrowser? As users are not supposed to download something for the extension.

Does this require the user to download something? Or is the keyboard listening to Chrome/WebBrowser? As users are not supposed to download something for the extension.

If the user has Razer Synapse installed, with a Chroma enabled device they will also have the ChromaSDK which also provides a secure REST server API that controls the device lighting.

Broadcasters would have an extra backend server running to get the device data.

Viewers just need the Twitch extension which talks to the secure REST server to play lighting effects.

Perhaps but that will be different for every user, and it’s only available in a video extension not a panel extension

I see the boilerplate has a viewer extension example.

https://github.com/twitchdev/extensions-samples/tree/master/boilerplate/frontend

Is there an example of a video extension somewhere?

Excellent. I have the viewers seeing Chroma effects synchronized with the video.

common.js

  window.Twitch.ext.onContext(function(context, contextFields) {
    //console.log("context", context);
    //console.log("contextFields", contextFields);
    if (context != undefined &&
      context.hlsLatencyBroadcaster != undefined) {
      receiveLatency = (context.hlsLatencyBroadcaster - 1) * 10; //100 ms intervals
	}
  });

I used a receive buffer so when PubSub messages are received I add some blank entries so that messages are received at the right time.

		  //add latency
		  while (receivedBuffer.length < receiveLatency) {
            receivedBuffer.push(undefined); //100ms intervals
		  }
		  //cap the buffer length to maintain sync
		  while (receivedBuffer.length > 0 &&
            receivedBuffer.length > receiveLatency) {
            receivedBuffer.splice(0, 1);
		  }

I have a function being called on a 100ms interval which displays the Chroma animations.

function showChromaState() {
  if (receivedBuffer.length > 0) {
    var json = receivedBuffer[0];
    receivedBuffer.splice(0, 1);

The interval is setup in the auth callback so I know if it’s a viewer or broadcaster.

  window.Twitch.ext.onAuthorized(function(auth) {
    //console.log("auth", auth);
    var parts = auth.token.split(".");
    var payload = JSON.parse(window.atob(parts[1]));
    //var streamer_id = payload.channel_id;
	isBroadcaster = payload.role == 'broadcaster';
	//console.log("broadcaster", isBroadcaster);
	if (isBroadcaster) {
      $('#user-access').text("Designation: Broadcaster");
      setInterval(function() { sendChromaState(); }, 100);
	} else {
      $('#user-access').text("Designation: Viewer");
      chromaSDK.init();
	  setInterval(function() { showChromaState(); }, 100);
	}
  });

I’ll use the latency callbacks to manage the receive buffer and that will keep things in sync.

The receive buffer should never be longer than the latency in seconds.

The animations play starting at the latency - 1 second because each PubSub message is 1 second long.

And that keeps the video in sync.

So I’m currently detecting if the broadcaster is watching the video to send PubSub messages from the viewer.html/JS.

But really I should be using the extension backend to broadcast PubSub messages. That way the broadcaster won’t have to worry about keeping the video browser tab in focus to push the Chroma data.

It might be nice to also make an OBS plugin so that when “Start Streaming” is enabled that it immediately starts sending Chroma data.

In the meantime when streaming starts, I’ll make it so the broadcaster only has to open the viewer browser tab once to authorize the backend.

Otherwise, the extension backend would need a front-end to enter the user streaming key. And I’d have to use the API to check if the user is streaming, authorize, and broadcast PubSub messages.

I’m on the fence about making an OBS plugin to broadcast the Chroma data on PubSub…

As per the diagram, I’ll make it so the Front-end JavaScript HTML triggers the Extension Backend Service to broadcast the Chroma data over PubSub.

Ideally, it will be the broadcast software that triggers the Extension Backend Service to broadcast the Chroma data over PubSub.

Solved via - [SOLVED] Backend Service: Broadcast to PubSub from Backend using Auth.Token JWT