r/MachineLearning • u/themathstudent ML Engineer • Oct 05 '17
Discussion [D] Deep Learning vs Bayesian Methods
https://medium.com/@sachin.abeywardana/deep-learning-vs-bayesian-7f8606e1e7812
u/asobolev Oct 05 '17 edited Oct 05 '17
Bayesian models don't have to be linear. Bayesian approach is a grounded way to incorporate prior knowledge into the model.
Deep Quantile Regression does not tell you how your model is uncertain in its predictions (epistemic uncertainty), it only models ground truth (aleatoric) uncertainty. Will your uncertainty estimates blow up if you train the network on a single observation?
Bayesian approach does not constrain you to mean estimates or symmetric uncertainty bounds. Given enough samples of a parameter of interest, you can estimate any statistic, whereas in the Deep Quantile Regression this statistic is baked into the neural net, hence it's a one-trick pony (this could make it more efficient, though).
Efficient and scalable Bayesian Inference is still an active topic of research. It basically started just 3-4 years ago, give it some time.
That said, I agree with your conclusions (except with the choking on 100s – Stochastic VI scales just as good as Deep Learning does).
6
2
1
u/multivariateg Oct 05 '17
"Here’s my main qualm with Bayesianists, they simply cannot commit to an answer"
The "answer" is the parameter that maximises the posterior distribution. It just so happens that in the process of Bayesian inference you get the full distribution (and therefore an idea about how sure you are) as part of the inference process. Compare this to deep learning, you get your answer that maximises a particular function, but without any of the associated (un)certainty about your answer without doing something expensive like bootstrapping.
0
Oct 07 '17
[deleted]
1
u/themathstudent ML Engineer Oct 07 '17
I'm just going to reply to the very last comment since you seem to have made up your mind about me. The paper was published in AAAI (so yes peer reviewed), and the code is referenced in the paper. https://www.aaai.org/ocs/index.php/AAAI/AAAI15/schedConf/presentations
4
u/HugoRAS Oct 07 '17
Fair enough.
I haven't "made up my mind about you" --- All we see is your article. Wrong's wrong. When I was young, I might have done the same.
Mistakes happen.
Just make sure you learn from them.
7
u/chrisorm Oct 05 '17
This is a troll post right?
"GANs effectively take a random vector and project it into a higher dimensional space which emulates the distribution of a given dataset. I won’t be surprised if the same principle is used to sample from a high dimensional posterior."
If only there was something GAN-like that gave us an explicit approximation to the posterior which we could easily sample from....
Oh yeh, wait, that other really famous deep generative model, VAEs.