r/Python • u/toastyCSboi • Jan 20 '20
I wrote a Python program to play Google Dinosaur Game using Gesture Recognition
150
u/Changer_ Jan 20 '20
cool, you could expand by making a handsign to voice for deaf people
101
u/toastyCSboi Jan 20 '20
Absolutely! That was actually my initial idea (for a hackathon). Though I ended up making this. I'll totally be working on that!
32
Jan 20 '20
[removed] — view removed comment
9
u/toastyCSboi Jan 20 '20
That's such an amazing idea! I will try to contact a professor of physiology/kinesiology in order to talk more about this!
9
Jan 20 '20
Just going to say it here. There's a charity that works to do exactly this kind of stuff called Special Effect. They'd be pretty interested in this kind of stuff too.
2
u/b4xt3r Jan 20 '20
I've always wanted to make something for quadriplegics that took the whole "eye mouse" thing and incorporated facial gestures that the user could teach the system by moving the various facial muscles to create complex gestures. Something like "open email/open new email" goes from "eye move blink eye move blink eye move blink eye move blink......" to "double eyebrow raise -> both eye close" (that make take you back to the main screen) "double eyebrow raise" (new "input" I guess), "eyes to right -> blink left" would be the application layer instruction. I don't know. The system would be able to learn from the gestures that the user could perform rather than forcing the user to use a very limited number of input to conform the user to he system (it should be conform systems to user IMHO).
114
u/mfitzp mfitzp.com Jan 20 '20
That's quite an "expansion".
"Hey I made flappy bird" "You could expand it by building an aircraft."
20
14
Jan 20 '20
I'll get right on that as soon as you hand me 10,000 labeled images of hand signs.
2
1
u/Changer_ Jan 20 '20
tbh you could easily set up something similar to what mozzilla deepspeech does instead of voice and word it would be pictures/or video and word. where people upload pictures/or video and put words to them, make it a community thing. I'm sure there are lots of people who are interested in this.
1
Jan 21 '20 edited Jan 21 '20
You are absolutely right. Sounds like something that deserves a research grant. As others have shown, there are already proofs of concept. The models and techniques required already exist.
Sounds like we should ask r/datasets
4
u/fulgerul46 Jan 20 '20
I made a few months ago a handsign to text for a hackathon, it went pretty good tbh
1
Jan 20 '20
Did you take letters as input, or words? I have only found datasets for letters. For obvious reasons, being limited to only one letter at a time can make communication tedious.
3
u/fulgerul46 Jan 20 '20
letters, just made it as a proof of concept with a small frontend, but if you are interested you can read this for words https://medium.com/tensorflow/getting-alexa-to-respond-to-sign-language-using-your-webcam-and-tensorflow-js-735ccc1e6d3f
3
Jan 20 '20 edited Jan 20 '20
Very cool, thanks.
Edit: Looks like they collected their own data. Just imagine what we could do with a larger, better dataset. 👌
37
Jan 20 '20
Could you please share the code? Or briefly explain how you did it?
87
u/toastyCSboi Jan 20 '20 edited Jan 20 '20
Here's the link to my GitHub Repo: https://github.com/Kshitij05/GestureRecognitionHackEd2020?files=1 (ClenchRex.py is the best working code)
As for the explanation - I use OpenCV to find Convex Hull and Convexity Defects, which helps differentiate an open hand and clenched fist. This code helped me a lot to understand gesture recognition: https://github.com/Sadaival/Hand-Gestures/blob/master/gesture.py
After I could detect gestures, I just used PyAutoGUI for keyboard inputs!
13
7
u/JoHnO674sh0t678 Jan 20 '20
Thanks, this is gonna help a lot, I'm inspired to try this out in c# as I'm more familiar with that, will be good practice
2
9
u/InternetGreninja Jan 20 '20
Do you have a sign for ducking?
7
u/toastyCSboi Jan 20 '20
I tried to use the "Victory" (or two fingers raised) sign for ducking - it was extremely buggy. I will try to make some changes to my code in order to be able to duck properly.
2
u/jayjaytdt Jan 21 '20
Hey man keep the control at duck by default, Rest will follow
2
u/toastyCSboi Jan 21 '20
I tried that actually. Down arrow Key + Space just resulted in an action from Down arrow key.
4
15
u/rexarski Jan 20 '20
I see Tim Hortons I upvote.
3
u/toastyCSboi Jan 20 '20
I spend way too much on Double Doubles than I would like to admit
1
u/rexarski Jan 21 '20
Me too! I remember at some time CIBC had a crossover credit card with Tim Hortons basically you could have free double doubles with monthly credit card spending. That should’ve saved a lot of money to be honest.
13
u/BananaFace765 Jan 20 '20
Canada gang where you at?
7
u/rexarski Jan 20 '20
Been in Toronto for 5 years. Used to stay up at a local Tim Hortons for CS projects. Definitely can relate.
4
u/BananaFace765 Jan 20 '20
I only go to Tim's if I am either hungry, or looking to satisfy my caffeine addiction.
0
u/SilverLion Jan 20 '20
Tim Hortons has the worst coffee ever, only way to make it drinkable is to smother it with cream and sugar. McDonalds is where it's at
1
u/rexarski Jan 21 '20
I find it the only way to keep me warm on a cold winter morning though.
2
u/SilverLion Jan 22 '20
Fair. I bought a cheap french press + grinder and started making my own coffee, so much better and I've never looked back. And most cost-effective.
3
Jan 20 '20
Great work man. What laptop model is it? Looks like envy though. I’m looking for a good performance laptop that involves app or game development? Looking for portability and by no means I’d go for macs. Please do suggest me. Thanks
3
u/toastyCSboi Jan 20 '20
It's an HP Spectre x360 (i7, 8 GB). I'm pretty satisfied with it - though I wish I got the one which comes with Nvidia graphics.
3
Jan 20 '20
thats awesome. way to go thnking outside the box and making somthing unique. im trying to learn the code but i cannot do anything becuase i seem to be missing the drive to code. it sucks becuase i want to but i feel like im forcing myself a bit.
1
u/toastyCSboi Jan 20 '20
Thanks so much! I totally relate to that - I would suggest taking a break or coding something really small and easy, so you feel motivated once you're done!
2
u/TheHammer_78 Jan 20 '20
Do you use haar trasnformation or simply do you watch at the expansion of the hand blob?
2
u/toastyCSboi Jan 20 '20
I didn't use Haar transformations here, though that would have been cooler. I used convex hull to detect convexity defects.
2
2
Jan 20 '20
How did you do the recognition for if your hand was open or closed?
1
u/toastyCSboi Jan 21 '20
I found convexity defects using OpenCV
1
Jan 21 '20
I'm stuck at opencv. I tried to make a program that searches for open eyes but there is no boolean in the program that is set to false if not faces And I need to get to closed eyes.
2
u/Oskarzyg Jan 20 '20
cv2, numpy, pyautogui? Could you name all modules please?
1
2
2
u/oktilubmidav Jan 21 '20
Lol I saw you consistently moving your hand when walking by...good to see you finally got it working
1
2
u/NubNuboso Jan 20 '20
Nice, but the down arrow?
3
u/ExHax Jan 20 '20
Divide the screen into two then detect the left hand for it
3
u/toastyCSboi Jan 20 '20
I tried another gesture for ducking - it was too buggy. But the idea of using the other hand sounds awesome!!
8
7
6
u/69shaolin69 Jan 20 '20
I made things like this too
Instead of using cascades I used machine learning and image processing.
6
5
5
5
4
3
2
2
2
2
2
2
2
2
2
2
2
2
2
2
u/Mikz_thegreat Jan 20 '20
THIS IS AWESOME! I just started learning python.. but how did you do THAT
2
u/Kali_Torvalds Jan 20 '20
This is what my College instructor means when he says "There are middle schoolers that will smoke your asses!"
2
2
u/69shaolin69 Jan 20 '20
Yeah I wonder what people at the cafe thought about a guy waving at his laptop
Future is weird hah
Good job btw, i love pyautogui and open-cv
And ofc Tims.
2
u/cakes42 Jan 20 '20
Pls do this on a pixel 4 so I can use the soli radars for something else that's not skipping songs.
2
2
2
2
2
2
2
2
2
2
2
2
1
u/JaimeRojas332 Jan 22 '20
What python version does it need to run? I'm getting some trouble with cv2, using python 3.7.5
1
u/JaimeRojas332 Jan 22 '20
Already solved my problem, I was using a 64 bits python version so I need to install opencv for 64 bits as described in this video : https://www.youtube.com/watch?v=Z78zbnLlPUA&list=PLQVvvaa0QuDdttJXlLtAJxJetJcqmqlQq
1
1
Jan 24 '20
This could honestly be great for the medical field. rehab for hand/finger movement due to injury
1
1
-1
u/abdalla_noah Jan 20 '20
you could have used javascript and ml5.js with p5.sj and coded it directly in the browser. it would be much easier. regardless, really good job!
1
u/toastyCSboi Jan 20 '20
I tried to deploy this game on a website, though I'm a beginner at JS. I will definitely learn more and try to do that!
4
u/danielmbcn Jan 20 '20
Or he could have used directly his brother to hit the keyboard as the OP made the gestures in front of him! It would be much easier!
68
u/toastyCSboi Jan 20 '20
Link to Github repository: https://github.com/Kshitij05/GestureRecognitionHackEd2020?files=1 (The final code is in ClenchRex.py)