r/Python • • Jan 20 '20

I wrote a Python program to play Google Dinosaur Game using Gesture Recognition

3.9k Upvotes

116 comments sorted by

68

u/toastyCSboi Jan 20 '20

Link to Github repository: https://github.com/Kshitij05/GestureRecognitionHackEd2020?files=1 (The final code is in ClenchRex.py)

2

u/[deleted] Jan 21 '20

[removed] — view removed comment

15

u/mingping101 Jan 21 '20

I don't think its connected. He's just taking a video through his phone

2

u/shaburushaburu Feb 03 '20

You can connect phone to the laptop using droidcam through WiFi and also through USB (need USB bugging to be turned on by going into the Android developer settings) And then changing the videocapture(0) to videocapture (1) in the python code

cv2.VideoCapture( variable )

Where variable can be 0 for webcam 1 for external cam and a direct file path like ("E:\video.mp4")

1

u/Java_Ethio Jan 21 '20

His pc has a camera, the phone is to show us it is real

150

u/Changer_ Jan 20 '20

cool, you could expand by making a handsign to voice for deaf people

101

u/toastyCSboi Jan 20 '20

Absolutely! That was actually my initial idea (for a hackathon). Though I ended up making this. I'll totally be working on that!

32

u/[deleted] Jan 20 '20

[removed] — view removed comment

9

u/toastyCSboi Jan 20 '20

That's such an amazing idea! I will try to contact a professor of physiology/kinesiology in order to talk more about this!

9

u/[deleted] Jan 20 '20

Just going to say it here. There's a charity that works to do exactly this kind of stuff called Special Effect. They'd be pretty interested in this kind of stuff too.

2

u/b4xt3r Jan 20 '20

I've always wanted to make something for quadriplegics that took the whole "eye mouse" thing and incorporated facial gestures that the user could teach the system by moving the various facial muscles to create complex gestures. Something like "open email/open new email" goes from "eye move blink eye move blink eye move blink eye move blink......" to "double eyebrow raise -> both eye close" (that make take you back to the main screen) "double eyebrow raise" (new "input" I guess), "eyes to right -> blink left" would be the application layer instruction. I don't know. The system would be able to learn from the gestures that the user could perform rather than forcing the user to use a very limited number of input to conform the user to he system (it should be conform systems to user IMHO).

114

u/mfitzp mfitzp.com Jan 20 '20

That's quite an "expansion".

"Hey I made flappy bird" "You could expand it by building an aircraft."

20

u/EnergeticStoner Jan 20 '20

It's just a couple of wings though.

14

u/[deleted] Jan 20 '20

I'll get right on that as soon as you hand me 10,000 labeled images of hand signs.

1

u/Changer_ Jan 20 '20

tbh you could easily set up something similar to what mozzilla deepspeech does instead of voice and word it would be pictures/or video and word. where people upload pictures/or video and put words to them, make it a community thing. I'm sure there are lots of people who are interested in this.

1

u/[deleted] Jan 21 '20 edited Jan 21 '20

You are absolutely right. Sounds like something that deserves a research grant. As others have shown, there are already proofs of concept. The models and techniques required already exist.

Sounds like we should ask r/datasets

4

u/fulgerul46 Jan 20 '20

I made a few months ago a handsign to text for a hackathon, it went pretty good tbh

1

u/[deleted] Jan 20 '20

Did you take letters as input, or words? I have only found datasets for letters. For obvious reasons, being limited to only one letter at a time can make communication tedious.

3

u/fulgerul46 Jan 20 '20

letters, just made it as a proof of concept with a small frontend, but if you are interested you can read this for words https://medium.com/tensorflow/getting-alexa-to-respond-to-sign-language-using-your-webcam-and-tensorflow-js-735ccc1e6d3f

3

u/[deleted] Jan 20 '20 edited Jan 20 '20

Very cool, thanks.

Edit: Looks like they collected their own data. Just imagine what we could do with a larger, better dataset. 👌

37

u/[deleted] Jan 20 '20

Could you please share the code? Or briefly explain how you did it?

87

u/toastyCSboi Jan 20 '20 edited Jan 20 '20

Here's the link to my GitHub Repo: https://github.com/Kshitij05/GestureRecognitionHackEd2020?files=1 (ClenchRex.py is the best working code)

As for the explanation - I use OpenCV to find Convex Hull and Convexity Defects, which helps differentiate an open hand and clenched fist. This code helped me a lot to understand gesture recognition: https://github.com/Sadaival/Hand-Gestures/blob/master/gesture.py

After I could detect gestures, I just used PyAutoGUI for keyboard inputs!

13

u/ChillRedditMom Jan 20 '20

Spectacular!

7

u/JoHnO674sh0t678 Jan 20 '20

Thanks, this is gonna help a lot, I'm inspired to try this out in c# as I'm more familiar with that, will be good practice

2

u/lollipas Jan 21 '20

How did you got that program to work with the dino game?

9

u/InternetGreninja Jan 20 '20

Do you have a sign for ducking?

7

u/toastyCSboi Jan 20 '20

I tried to use the "Victory" (or two fingers raised) sign for ducking - it was extremely buggy. I will try to make some changes to my code in order to be able to duck properly.

2

u/jayjaytdt Jan 21 '20

Hey man keep the control at duck by default, Rest will follow

2

u/toastyCSboi Jan 21 '20

I tried that actually. Down arrow Key + Space just resulted in an action from Down arrow key.

4

u/Tengoles Jan 20 '20

Came to ask this

15

u/rexarski Jan 20 '20

I see Tim Hortons I upvote.

3

u/toastyCSboi Jan 20 '20

I spend way too much on Double Doubles than I would like to admit

1

u/rexarski Jan 21 '20

Me too! I remember at some time CIBC had a crossover credit card with Tim Hortons basically you could have free double doubles with monthly credit card spending. That should’ve saved a lot of money to be honest.

13

u/BananaFace765 Jan 20 '20

Canada gang where you at?

7

u/rexarski Jan 20 '20

Been in Toronto for 5 years. Used to stay up at a local Tim Hortons for CS projects. Definitely can relate.

4

u/BananaFace765 Jan 20 '20

I only go to Tim's if I am either hungry, or looking to satisfy my caffeine addiction.

0

u/SilverLion Jan 20 '20

Tim Hortons has the worst coffee ever, only way to make it drinkable is to smother it with cream and sugar. McDonalds is where it's at

1

u/rexarski Jan 21 '20

I find it the only way to keep me warm on a cold winter morning though.

2

u/SilverLion Jan 22 '20

Fair. I bought a cheap french press + grinder and started making my own coffee, so much better and I've never looked back. And most cost-effective.

3

u/[deleted] Jan 20 '20

Great work man. What laptop model is it? Looks like envy though. I’m looking for a good performance laptop that involves app or game development? Looking for portability and by no means I’d go for macs. Please do suggest me. Thanks

3

u/toastyCSboi Jan 20 '20

It's an HP Spectre x360 (i7, 8 GB). I'm pretty satisfied with it - though I wish I got the one which comes with Nvidia graphics.

3

u/[deleted] Jan 20 '20

thats awesome. way to go thnking outside the box and making somthing unique. im trying to learn the code but i cannot do anything becuase i seem to be missing the drive to code. it sucks becuase i want to but i feel like im forcing myself a bit.

1

u/toastyCSboi Jan 20 '20

Thanks so much! I totally relate to that - I would suggest taking a break or coding something really small and easy, so you feel motivated once you're done!

2

u/TheHammer_78 Jan 20 '20

Do you use haar trasnformation or simply do you watch at the expansion of the hand blob?

2

u/toastyCSboi Jan 20 '20

I didn't use Haar transformations here, though that would have been cooler. I used convex hull to detect convexity defects.

2

u/JMH5909 Jan 20 '20

Code plz

1

u/toastyCSboi Jan 21 '20

I have posted the link in the comments!

2

u/[deleted] Jan 20 '20

How did you do the recognition for if your hand was open or closed?

1

u/toastyCSboi Jan 21 '20

I found convexity defects using OpenCV

1

u/[deleted] Jan 21 '20

I'm stuck at opencv. I tried to make a program that searches for open eyes but there is no boolean in the program that is set to false if not faces And I need to get to closed eyes.

2

u/Oskarzyg Jan 20 '20

cv2, numpy, pyautogui? Could you name all modules please?

1

u/toastyCSboi Jan 21 '20

I used OpenCV, NumPy, PyAutoGUI and math

1

u/Oskarzyg Jan 21 '20

Nice! Wish I could have a working laptop to code on! Good Work!

2

u/malingeringGit Jan 20 '20

did you use a sensor or just image process’d the camera input?

2

u/toastyCSboi Jan 21 '20

Just image processing!

2

u/oktilubmidav Jan 21 '20

Lol I saw you consistently moving your hand when walking by...good to see you finally got it working

1

u/toastyCSboi Jan 21 '20

Lmao, yeah I probably weirded out a bunch of people. Thanks!

2

u/NubNuboso Jan 20 '20

Nice, but the down arrow?

3

u/ExHax Jan 20 '20

Divide the screen into two then detect the left hand for it

3

u/toastyCSboi Jan 20 '20

I tried another gesture for ducking - it was too buggy. But the idea of using the other hand sounds awesome!!

8

u/chensformers Jan 20 '20

Clever and fun exp

6

u/69shaolin69 Jan 20 '20

I made things like this too

Instead of using cascades I used machine learning and image processing.

6

u/scythoro Jan 20 '20

Awesome job!

I watched this way too long waiting for the game to get harder.

5

u/appinv Python&OpenSource Jan 20 '20

Cool saw someone make by smiling just the other day! Saved!

5

u/[deleted] Jan 20 '20

Pushing the limits of technology right here

4

u/iggy555 Jan 20 '20

Lol cool

3

u/[deleted] Jan 20 '20

Sometimes it's the little things that are the best. Such a cool idea. Props!

2

u/belizean33 Jan 20 '20

Cool bro! This has so many practical applications

2

u/reinaldo866 Jan 20 '20

Unknowncheats wants to know your location.

2

u/BlasphemousToenail Jan 20 '20

That is really cool!! Well done!

2

u/KarlIvanov Jan 20 '20

Good work!

2

u/[deleted] Jan 20 '20

Nice

2

u/[deleted] Jan 20 '20

Cool

2

u/jec4r Jan 20 '20

amazing, congratulations bro!

2

u/colesupreme Jan 20 '20

Now to duck!

2

u/oculusshift Jan 20 '20

this is pretty cool

2

u/Mikz_thegreat Jan 20 '20

THIS IS AWESOME! I just started learning python.. but how did you do THAT

2

u/Kali_Torvalds Jan 20 '20

This is what my College instructor means when he says "There are middle schoolers that will smoke your asses!"

2

u/next2zero Jan 20 '20

And this is how you get those absurd hi-scores

2

u/69shaolin69 Jan 20 '20

Yeah I wonder what people at the cafe thought about a guy waving at his laptop

Future is weird hah

Good job btw, i love pyautogui and open-cv

And ofc Tims.

2

u/cakes42 Jan 20 '20

Pls do this on a pixel 4 so I can use the soli radars for something else that's not skipping songs.

2

u/[deleted] Jan 20 '20

This is so cool.

2

u/Nomekop777 Jan 20 '20

How do you duck

2

u/b4xt3r Jan 20 '20

Rock on! That is fantastic!

2

u/[deleted] Jan 21 '20

wicked cool!!

2

u/LinhNguyen221 Jan 21 '20

Tim Horton's -> Canada :)

2

u/ivanksy_ Jan 21 '20

This is really cool !

2

u/JohnnyWobble Jan 21 '20

lmao I have the same computer

2

u/_Cosmic_Joke_ Jan 21 '20

That’s pretty cool!!!

2

u/realRohitYadav Jan 21 '20

Whoa! This is amazing.

2

u/ambarish_k1996 Jan 21 '20

I am really impressed, nice job and keep up the good work

2

u/dnouvel Jan 20 '20

😎 Coool!

1

u/JaimeRojas332 Jan 22 '20

What python version does it need to run? I'm getting some trouble with cv2, using python 3.7.5

1

u/JaimeRojas332 Jan 22 '20

Already solved my problem, I was using a 64 bits python version so I need to install opencv for 64 bits as described in this video : https://www.youtube.com/watch?v=Z78zbnLlPUA&list=PLQVvvaa0QuDdttJXlLtAJxJetJcqmqlQq

1

u/geppaaa Jan 22 '20

which library did you use for gesture recognition?

1

u/[deleted] Jan 24 '20

This could honestly be great for the medical field. rehab for hand/finger movement due to injury

1

u/[deleted] Jan 20 '20

Good job Isaac.

-1

u/abdalla_noah Jan 20 '20

you could have used javascript and ml5.js with p5.sj and coded it directly in the browser. it would be much easier. regardless, really good job!

1

u/toastyCSboi Jan 20 '20

I tried to deploy this game on a website, though I'm a beginner at JS. I will definitely learn more and try to do that!

4

u/danielmbcn Jan 20 '20

Or he could have used directly his brother to hit the keyboard as the OP made the gestures in front of him! It would be much easier!