https://www.kdnuggets.com/2018/06/why-data-lake-matters.html

I'm at Super Computing '15, and the trend here is similar to what I've seen at other conferences: Spark and Hadoop style tools perform drastically slower for science problems than MPI does, and this includes the fault tolerance aspects, as well (i.e., MPI based checkpoint-restart).

A common argument is that it's easier to code Spark and Hadoop applications than it is MPI codes, but there's a caveat here. To get even remotely comparable performance, a tremendous amount of work has to be put into tuning the parameters and configurations.

The other problem is that it's very hard to get Spark and Hadoop codes to scale for NON-EMBARRASSINGLY parallel algorithms. Pretty much 100% of Spark and Hadoop workloads are embarrassingly parallel, meaning the "mappers" operate without ever having to talk to each other.

What a farce. Who uses these shitty tools?

My weekend read. Do you know why building a data-driven organization is always perceived as a challenge? It's because most of the organizations who strive to be data-driven now, didn't start off with the culture of data. And this transformation of cultural rust and resistance is a painful exercise. Every organization has a plan (+strategy) to execute, but still it takes years to make it happen.

https://www.linkedin.com/posts/saurabhg-data-science_strategy-data-leadership-activity-6763868075105378304-_FVR

Hey /r/bigdata

I hope this is the right sub for this.

Our community created a data mining test for those in the know to test their skills and prove their knowledge. Take a look and let me know what you think. Post your score below to see how you stack up!

So you think you know data mining?

The test does ask for users to create a free account but info is kept completely private and we don't spam. Some people thought this was a data mining effort in and of itself, but we just wanted to see if the experts here think the test is up to snuff. Thanks!