Firstly, I apologize for my ignorance; I’ve never run a twitch stream and know little about it. My question is this:
Is it possible, and if so, how would I go about streaming a video game with big numbers on the screen, grabbing screen caps very often (I’m thinking the max frequency I would want would be 1 second), analyzing the data (image detecting to figure out what numbers are being displayed), doing some arbitrary math with those numbers and then outputting them onto my stream’s overlay.
Sorry for the run-on sentence; allow me to sum up the major points of my question:
How do I programmatically grab screen caps of my video stream (what language would I use) and how would I transmit data from that program to an overlay. I admit I’m really in way over my head here; is there a specific streaming computer program that would allow for something like that easily? Are overlays even implemented in the streaming program or elsewhere?
Hopefully someone out there has a lot of knowledge and is feeling patient and generous with me. Thanks very much for reading.
Frequently grabbing images of that size and then running image detection on them in real-time seems like a very, very heavy task to do, cpu-wise. It would require a third-party library like Asprise or Tessaract and probably more than a little experience in programming.
Streaming programs or the plugins that generate the overlay don’t interact with the image data in the way that you would need, as far as I am aware.
There is probably a better way to accomplish what you are trying to do than to grab the numbers AFTER they are being rendered on the screen. What exactly are you trying to do?
I’ll do things in whatever way is most efficient/makes it possible. I’d like to be able to stream Super Smash Bros matches, grab the character damage %s in real time and output the +/- to the overlay (showing who is in the lead by how much damage).
(Go to the link below if you aren’t familiar with what smash bros looks like).
Since the game is run on Wii U it rules out everything but actually grabbing screenshots when it streams to your PC. But looking into it I feel that it’s definantly is doable - given that the existing librarys are good enough to read the numbers the way they are presented and that you are experienced/determined enough to pull it off.
From what I found it would be easiest to do this in C# with the .NET 2.0 Framework - a language I’m not so familiar with. I don’t know how you would do this in detail but found a tutorial that points you in the right direction. The Asprise library for Optical Character Recognition is also available for this language.
If the absolute position of numbers dynamically vary as players are eliminated you need a way to locate them, which could be difficult. Hopefully this is not the case, so that you can manually input the number of players in the beginning - which would correspond to the positions of a set of bounding boxes for capture. (Ideally you would only grab these rectangles and not the entire screen.)
When the players take damage there is short animation that make the numbers unreadable as they increase, but this should not be a problem since you continually update and would eventually capture the numbers while they are still.
However, this seems to me like a huge endevour - especially if you are not familiar with programming. Considering that the difference between two numbers is something most people can do in their head I wouldn’t say that this feature is worth your time, unless you’re mostly in it for the challenge. For more questions regarding this I think StackOverflow or similar is a better forum. Your project isn’t really related to Twitch API, even though you intend to use it for Twitch.
I am familiar with programming, and while I feel that this project is probably about 1 notch above my current skill level, it would be a passion project for me, which I feel would motivate me enough to see it through. While it’s obviously possible for people to figure out the damage difference in their head, I really feel that it would add a lot to the Smash streaming community, which is large and growing, and if I could get the system up and running, I feel pretty confident that I could get it spreading through the community, which would be very satisfying.
Thank you very much for pointing me in the right direction.
Have a look at OpenCV. That seems to be what you’re after. We have a similar system on TrumpSC’s channel, with card draw/game event detection in Hearthstone using OpenCV.
Good luck!