Pass raw video stream or video texture to video component extension

Hey there!

I was wondering if there was anyway to get a reference to the actual streamed video in an extension. For example, I’d like to pass the video as a texture to three.js or something similar to do fun video effects on the video. I tried embedding the stream through the iframe embed route, but it doesn’t appear that it’s working and I suspect it’s because iframes are blocked in general for extensions for security reasons.

Is there any way to achieve what I’m trying to do?

I can think of a few ways, but it won’t be in real time.

What you describe would work better as a OBS/XSplit plugin/source rather than in browser/extension/clientSide I think.

Do you mean non-realtime as it would have a delay from the main stream or that it would only work with VODs? I think non-realtime would be fine for my usecase as long as it’s live.

Can you talk more about what you’re thinking as well w/r/t OBS source?

If it’s running in OBS you have real time access and arguable better access to the computer in order to access the video and/or display

Hmm I don’t think that would work though because then any interaction can only be done from the streamer’s computer right?

What’s the non-realtime methods you’re thinking of?

Not sure I’ve never tried to do what you want to do. Most of the things I’ve dabbled with are too heavy for client side, which means punting it server side and then out of date/way out of sync.

But depends what sort of interaction you want from viewers. I was just thinking you were making a pretty overlay which wouldn’t need/have any interaction (except maybe picking which filter)

On second thoughts what you describe sounds a little like the SnapChat extension, but that will be running a local compoenent to inject the “cat ears”/whatever and access the video stream locally and then thats in the live stream. Which is probably the most performant/works on all viewer computers.

I don’t think that an extension is able to break out of the iframe to see whats behind it, might means you will need a local component to capture the video/display on the streamers computer.

It seems like in general you can’t have an iframe in an extension at all, which may be the core issue. I think what I really need would just be the ability to get the stream from the channel the extension is running on, and basically pass that to the js code or use in a video tag or otherwise.

Currently the rules and security prevent that.

And would mean if people are using your extension, the video would be essentially loaded and streamed twice

Edit: This post

Covered some ideas on syncing data from a local data sender. With an extension front end. Might be of interest?

Hmm I definitely see some overlap of ideas there, but I think my usecase is still slightly different. The idea here is that any given viewer could have a different experience of the stream based on how they were interacting with the stream. For example if the stream video was mapped to a 3D cube, any given user could rotate the cube differently, independent of action by the streamer.

So said differently, no local data sending besides a stream.

Also looking at the rules I now see that you can’t use iframes (makes sense).