r/AskComputerScience • u/kool2015 • Jun 27 '20
Is the Algorithm for the deep Learning too complex or easy? In this picture it seems not too complex
16
5
u/tikhonjelvis Jun 28 '20
The algorithm for training and using deep neutral networks is pretty simple, although there's more to it than covered in that diagram. 3Blue1Brown has a good explanation. (Note that it requires a bit if basic calculus and linear algebra.)
What makes deep learning a difficult field—and an active area of research—is not the core algorithm itself. Instead, it's mostly about understanding and using neutral nets:
What kind of data do you need to train on?
How do you need to preprocess the data?
How should the neutral net itself be configured? (There are a lot of different architectures.)
How do you interpret the output and figure out why the network produced the result it did?
Solving real problems with deep learning requires a lot of high-level knowledge and expertise on top of just understanding the basic algorithms. (In fact, you can probably get away without really understanding the core algorithms in a lot of contexts because you'll be using an existing framework like TensorFlow to do the heavy lifting.)
There is also research going on in improving the algorithm itself, trying to make it perform better at scale, to the point where companies are developing custom computer architectures and chips dedicated to drop learning. Projects like that, as you might imagine, are by no means simple! (Although they are a lot simpler than general-purpose chips like most CPUs.)
4
u/postisapostisapost Jun 27 '20
Ah, but how do you decided on adequate weights for the hidden layers, esp without a priori knowledge of Presidents?
That's the beauty of backprop.
Update: the algorithm is abstract enough that one can specify any constraint or optimizer, and it will work. So, it could be as simple or complex as the problem domain requires.
2
u/lookayoyo Jun 28 '20
Yes but different structured neural networks will perform differently for different tasks. Sure you can generate a model that looks good with certain data sets, but the hard part is testing different models and see what works best. Harder yet, figure out why one structure works best. One thing this image obfuscates is that it doesn’t explain what it does. Maybe the engineer guessed that it is identifying shapes and features, but the system is really a black box. The engineer just sees a series of linear equations and won’t actually have much of a clue as to why it functions the way it does.
3
u/This_Kvothe Jun 28 '20
As alluded to in the other comments this picture is an extremely over-simplified view of deep learning. It does not in any way indicate the complexity of the calculus required for a network like this to make a decision.
1
1
48
u/kedde1x Jun 27 '20
There's waay more to it than the picture leads one to believe. For instance, all the lines have a unique weight, which has to be accounted for. The values at each layer are computed by the values of their previous layer and their respective weights and something called an activation function.
The picture also doesn't explain how the learning actually works. It is not just a matter of forwarding a bunch of values through it. You need to train it first. Without training, it cannot know anything and the output is seemingly random. The way to do this is to feed it an input that you know the answer to. Then see what it says. Most likely it will be wrong. Then you measure exactly how wrong it is on each node in the output layer, and propagate this error back throughout the network, adjusting the weights along the way according to the error. Then you repeat this for thousands or even millions of different inputs. This uses stochastic gradient descent. This is called the backpropagation algorithm and is essential to why neural networks work. And it is not apparent in the picture.
Then there's another problem with the example shown in the picture. It shows a regular feedforward neural network, which works great for very simple tasks like distinguishing between digits or even letters. But that's it. More complex tasks, like image recognition as shown in the example, are not suitable for these kinds of regular neural networks. Instead, you'd need a different kind of network. For image recognition, convolutional neural networks are really popular. In such a network, each layer is not a collection of nodes, but a 2D plane of nodes. The algorithms to feedforward and backpropagate then uses convolution between layers. My point is, the example would not be feasible in the network shown.
Deep learning and neural networks are really complex and hard to understand. I think the picture works fine for laymen to kind of grasp at the idea of how it works; but if you really want to understand, you need to do a lot of reading because it is definitely a complex topic.