40
Mar 22 '19
does this use any form of machine learning? I'm a non tech
59
u/alkasm github.com/alkasm Mar 22 '19 edited Mar 22 '19
No, this isn't using machine learning.
The program is using color-based thresholding to find the hand (the black/white mask of the hand shown) and then finding the contours (outline) of the hand. From there it finds the convexity defects, which are where the outline falls into the inner shape, and counts how many of those there are. So if you hold up 5 fingers there's 4 big convexity drops towards the center (the spaces between your fingers). If you did a Spock hand gesture for e.g., this would likely register it as "3." This is also why the video shows very widespread fingers, to find those convexity defects clearly.
This is not a robust method, color-based thresholding is very sensitive (obviously) to the color of your skin, the lighting, and the background, and assumes it's the only thing of that color in the frame. Still, it's a fun example of something rather simple to do in OpenCV that actually doesn't take a ton of computer vision knowledge to try.
This is absolutely possible and not hard to do with ML as well, but the author here is using classical CV techniques.
The author says that the first script helps "train" your machine and that is absolutely nonsense. What the code does is just hard threshold a training image (which I'll note that they don't have the rights to as they've taken a Shutterstock image with the watermark still on it...). It's on you, the user, to take a capture of your hand and provide threshold values that work well for it.
Source: am computer vision engineer and read the source code.
2
26
Mar 22 '19
No itâs using a computer vision library called OpenCV
31
u/declanaussie Mar 22 '19
OpenCV uses machine learning for some stuff so maybe. I donât know enough about OpenCV to say yes for sure though, although it seems likely in this situation.
14
u/alkasm github.com/alkasm Mar 22 '19
Yep, OpenCV has a deep neural network module (
cv2.dnn) and also has support for some simpler ML stuff like haar cascades and etc.However, this code isn't using any of that and isn't ML-based.
5
0
u/yeezybillions Mar 22 '19 edited Mar 22 '19
Computer vision is a type of machine learning.
Edit: my understanding of computer vision was wrong
6
u/alkasm github.com/alkasm Mar 22 '19 edited Mar 22 '19
Not quite---computer vision has been around before the resurgence of ML and has a rich history of interesting work related to image processing and geometry. Typically the non-ML stuff is called "classical computer vision" and the geometric stuff is called "multiple view geometry." Nowadays a lot of computer vision is powered by ML for tasks like recognition, classification, tracking, detection, pose estimation, and so on. But there are other parts of CV that aren't.
2
u/yeezybillions Mar 22 '19
It sounds like youâre a lot more knowledgeable about ML and CV than me, but my understanding of machine learning is: itâs any model that learns and adjusts parameters from the data to achieve a particular goal (typically minimize some error function), rather than being explicitly hard-coded. I was taught that even simple linear regression can be considered machine learning. I donât consider myself an expert in ML or CV by any means though, so please correct me if Iâm wrong; maybe this is just a semantics issue.
4
u/alkasm github.com/alkasm Mar 22 '19 edited Mar 22 '19
I think that's a fine definition for the purposes here. But many things in computer vision aren't learned/parameterized by an error function.
For e.g., there's a technique called a "stroke width transform" to detect text in images. It takes an input of edges in the image (computed with a Canny edge detector or similar), then it finds the smallest distance between the edges. Areas in the image that are of close to constant width can be classified as text and run through an OCR program to read the text. The stroke width transform requires no training, doesn't have any parameters to tweak.
Here's an image of what I mean by stroke width and distance between edges: https://images.app.goo.gl/gC3Bs7igcMMvmUqz6
And here's an example running on an image.to detect possible letter candidates: https://www.researchgate.net/profile/Bowornrat_Sriman2/publication/283026077/figure/fig2/AS:305510722097159@1449850840987/Example-of-the-Stroke-Width-Transform-result-on-Thai-and-English-scripts.png
These letter candidates can then get filtered out based on size and aspect ratio and location and so on.
Actually I don't even know why I gave this example, the OP here is an example of non-ML based computer vision.
2
u/Han-ChewieSexyFanfic Mar 22 '19
No, not necessarily. Computer vision is a problem, machine learning is one possible approach to solve that problem.
2
u/Ph0X Mar 22 '19
This case, it's quite literally taxing the pixels, using a brightness/color threshold to figure out which pixels are "wall" and which are "hand" (top right), then creates a convex hull around it.
Both of those are very basic algorithms and have zero machine learning, statistical analysis or AI involved.
3
u/Deadshot_0826 Mar 22 '19
It does as the first step in running the program is âtrainingâ the computer to recognize the gestures.
8
u/alkasm github.com/alkasm Mar 22 '19 edited Mar 22 '19
The author's description and their code doesn't line up. There is no ML in the code, and no training.
1
u/Deadshot_0826 Mar 23 '19
The original question is if this program âuses machine learning in any wayâ, and from what I understand it is. The first step is training it if you look in the repo. If you it isnât could you explain why not? Thanks.
2
u/alkasm github.com/alkasm Mar 23 '19
It isn't. See my longer answer for how it works. There is no training. The "training" is a script that you can modify and find the color threshold values that work for your hand. In other words, you train it! Not a program! Lol
1
1
4
2
u/Killerjayko Mar 22 '19
Could this potentially be used for things like translating sign language?
10
u/jblo Mar 22 '19
Currently, no.
ASL/BSL, etc rely far too heavily on facial and other non manual markers, along with tone and speed dictating much of what is being said in Sign Language. You'd need multiple systems like this all running in concert, with thousands of hours of ML applied to begin really translating.
1
2
u/bradfordmaster Mar 22 '19
In addition to what the other poster said, techniques like this are rarely robust enough in real world environments. This is a great project, but look at the video -- we've got a perfectly well lit room with a plain white background and very clear separated fingers. This is still a hard problem to solve, but very unlikely to work from a random angle video taken from a smartphone with who knows what in the background
2
u/EmperorDeathBunny Mar 22 '19
Does it still count correctly if you hold out your thumb and index finger?
4
1
1
u/jjbugman2468 Mar 22 '19
I remember seeing a series of Facebook posts by a guy who made a gesture-detecting calculator a year or two ago. Basically this, plus more poses for calculation signs
1
1
u/jkuhl_prog Mar 23 '19
In some languages, three is done with a thumb and two fingers, can it recognize that?
1
1
1
0
0
0
Mar 22 '19
[deleted]
5
u/alkasm github.com/alkasm Mar 22 '19 edited Mar 22 '19
IDK what their nationality has to do with it, but I could easily see it being a common project in computer vision courses. It's pretty easy to do, relatively little code, and teaches you what you can do with contours.
2
13
u/juliangalardi Mar 22 '19
Any repo/git?