r/learnpython • u/Matheos7 • Nov 28 '20
Data analysis for beginners
Hi all,
What would you recommend as the best route to start learning towards data analysis with Python (I will add that I'm not planning on becoming data analyst/scientist within few months or so - I started learning Python 4 months ago, got the basics, not interested in building websites, creating games, but I would like to explore data; I also already started learning SQL 2 months ago. Really taking it easy and want to see where it all leads to, but extremely happy that I'm learning new skill).
Should I start with Matplotlib? Or Numpy? Or Pandas?
Any other tips or recommendations?
Thank you for your answers in advance!
45
u/MikeDoesEverything From zero to Data Engineer: https://tinyurl.com/2p8wd6y7 Nov 29 '20
As somebody who is a scientist and worked with data scientists, aside from learning the techniques I would also try and find something out about how to approach problems. I'll list the mistakes data scientists made when we were working together:
Assuming any trend was a fact. The DS team were very new, to be fair, and were finding what was essentially random correlation in data and trying to attribute causation to it. They would have discovered the vast majority of their reasoning did not have any scientific basis to it if they simply asked the R&D team. That being said, they spent a lot of time trying to figure out the meaning behind trends. This is the classic case of causality does not equal correlation. Leading onto next point.
Not asking for any external input on their analyses. The DS team really should have been better at communicating and, instead, they were more keen on selling data analysis. We all went on these courses with little to no experience to use software for analysis although the whole thing was a mess. We had a lot of people drawing wild conclusions from small sets of data with zero skepticism because their findings made them look good and it all became very political very quickly. Goes nicely onto my next point.
Not questioning data. Slightly different from the first one in the sense the DS team never really challenged their own ideas or even sought to test it and back it up with experimental data. Their argument would be "we've run this simulation 5000 times on the computer and it has a 98+% correlation" although what happens in the program doesn't always reflect what happens in real life. Always be willing to accept challenges on your findings and also question your own data.
tldr: Talk to your subject matter experts about trends and see if they check out. Do not assume all missing data is bad. Do not assume all collected data is good. Always be asking questions about your own data.
20
u/foyslakesheriff Nov 29 '20 edited Nov 29 '20
Dude, you need to learn the math. I say this to you because it's easier than saying it to myself, and I need to hear it more often. Have a good handle on statistics, linear algebra, and calculus. It's not sexy but it's necessary to not waste your time in the long term. I hate hearing that because I love learning new Python frameworks. After learning the theory, the tools will make more sense. People might say that we have those tools so that we don't have to reinvent the wheel each time. That is true. But you still gotta know how a wheel works, otherwise you won't know how to diagnose and treat an issue when you get unexpected results.
If you don't understand linear algebra, then you won't understand why you're using Pandas. If you don't understand some regression models, scikit-learn won't be useful. Et cetera
It's not so much what you start with, because you will need to know all of those tools and use them together. Pandas uses numpy under the hood so to speak. Matplotlib will make your data output pretty. You might need SQL. I would start small, and create a few exploratory jupyter notebooks that use all those packages at least a bit.
Having said that, watch Andrew Ng's stuff.
3
u/Rainbow_Moonbeam Nov 29 '20
There's an edx micro masters series of courses by the university of San Diego which I'm working through. There's four in total: Python for Data Science, Statistics, Machine Learning, and Spark for Big Data. I'm halfway through the stats one at the moment.
The first course had a much stronger focus on "Python" than "data science" - kind of dipping your toes into it and seeing what sorts of things you could do. A bit of pandas, a bit of numpy, a quick intro to machine learning, and a few mini projects.
The second course is way more data science than python. It's almost all maths with a tiny bit of how to do the same thing but in python. It's pretty tough (micro masters assumes you have university level maths experience and I stopped studying maths at 18) but I'm looking forward to understanding data better and knowing what's relevant/statistically significant etc.
I enjoyed the first course but kind of felt like all my analysis was a bit silly and superficial. I could find correlations but I didn't know if it was important. I could copy the way the teachers used machine learning but didn't know if I was using the right model for the task or how to find more in depth analysis. I'm hoping that after I finish the next few courses, I'll be able to understand this all better.
17
u/nojustlurkingty Nov 28 '20
Python Crash Course (book) section 1 (python basics) and then Project 2 (data analysis)
Covers numpy, matpotlib, and data visualizations
I think matpotlib, numpy, and pandas are equally as important
6
3
u/MrCoderModer Nov 29 '20
I would say learn pandas as well! Pandas is going to give us the skills to start manipulating data in Python. I would say watch Kevin Markham.
https://www.youtube.com/playlist?list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y
3
u/SandyPointRigs Nov 29 '20
The Francois Chollet book on Keras & machine learning is free online and very good. Lots of examples and things to try & learn from.
3
u/pugwala Nov 29 '20
Data analysis begins with data cleansing, and data cleansing is the daily grind for most data scientists. “Garbage in, garbage out” as it is said, is the first rule. 100% of the data cleansing tutorials and walkthroughs that I have found through Google, Medium/Towards Data Science, StackOverflow, and YouTube were never sufficient for my data cleansing needs. Most tutorials start with highly curated datasets/sources to begin with and then go from there. Develop your own data cleansing pipeline and use it with all your needs. Issues to consider:
bias hard baked into the datasets (are they biased to a desirable outcome?)
sparsely filled data/records (to impute or not impute, convert NaNs, drop/ignore - all of which adds additional bias to your data and results)
multiple language support (commonly on the same webpage)
markup & script languages
email addresses & hyperlinks
numbers, conversions, dates, timeframes & varying data types
text processing issues: anagrams, stopwords, capitalization, domain specific terms, uni/bi/trigrams
categorical features
scaling
And so much more. Create your own data cleansing pipeline and get to really know your own data.
Become familiar with Pandas, but don’t build your processing pipelines around it. Pandas can be frustratingly inconsistent in methods and functions. Keep in mind that one way to think about Pandas is that it is a container of arrays, and as such may be evaluated in lots of very familiar and consistent ways.
RandomForest is your friend. I always say “no data scientist gets fired for using RandomForest”. This is a global truth. There are a lot of algorithms and models that a far superior, and some data scientists will poo-poo its use, but I always respond saying “it’s computationally efficient and reliably verifies which insights are valuable with almost zero effort”. Your director, or cost center owner, will love you.
Scikit-Learn is a great API toolkit to build upon. Yes, many of the algorithms are a bit dated and newer boutique versions provide better results. It it gets you a long way down the road of good, better, best.
Lastly, remember that 12 data scientists in a room will have 12 different opinions of why there approach is superior. Unless you are in academia, don’t let perfection get in the way of good.
I hope this helps.
2
u/Zack_jeff Nov 29 '20
Hey. So I’ve been a data scientist for 3 years now and for data you need all of the libraries you just listed: NumPy, Pandas, Matplotlib, seaborn.
But I would definitely start with pandas because that’s what you mainly use to even access the data, or you can use os
You can explore your data needs with only learning pandas. Pandas has a wide variety of tools to help you with this.
2
u/_Hawks_20 Nov 29 '20 edited Nov 29 '20
Follow this order for learning - numpy, pandas, metplotib. Link to resources:
Numpy - https://youtu.be/QUT1VHiLmmI
Pandas - https://www.youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS
Metplotib - https://www.youtube.com/playlist?list=PL-osiE80TeTvipOqomVEeZ1HRrcEvtZB_ Or Learn all in this video - https://youtu.be/r-uOLxNrNk8
2
u/my_password_is______ Nov 29 '20
https://www.kaggle.com/learn/overview
kaggle, they have free courses and tons of data sets to work on
also
https://www.youtube.com/playlist?list=PLGLfVvz_LVvQjNJr85J4U_lxDg8vgqvcO
https://www.youtube.com/playlist?list=PLGLfVvz_LVvSVgVCsPWLr961id7kRv1wt
https://www.youtube.com/playlist?list=PLGLfVvz_LVvQy4mkmEvtFwZGg1S38MUmn
2
2
u/kuthedk Nov 29 '20
lol. you're going to end up using all of those that you asked about. so you best start learning how to use all of them. no one is more important than the other. they have different functions and you need to learn when to use them and when one is better than the other for a given situation.
try to start to analyze the data that you are interested in, and find the ways that help explain the data that you are looking at. All of those things you have asked about in regards to Python are called libraries. They are basically extensions of code that someone already spent a long time working on to make sure that they work, in other words, they are your tool box.
1
u/mightykev Nov 29 '20
I’m starting out as well actually going to a program at uni. Found it’s more about the concept than the tool. I’m having hard time on the statistics theory, linear algebra concepts vector, dimensions, than the mechanics in python or R.
1
1
u/antirabbit Nov 29 '20
If your goal isn't just cleaning data, but extracting some statistics/trends, I would look at beginner statistics courses/guides. That can help you focus more when you have a large amount of data but only need to show the most "important" parts/trends.
Also, learning matplotlib is useful if you want to visualize data in Python. I usually write a separate R script for that, since most graphs are much, much easier to make in R, but you can basically find an example for most kinds of graphs from matplotlib's examples page here.
Also, learning SQL/a Python wrapper to SQL can help with working with larger datasets (reasonably up to 10s of GB on a personal computer). I usually use Postgres/the psycopg2 library. I'd recommend just using SQLite3 to start off with.
1
u/Far_Inflation_8799 Nov 29 '20
These python libraries are all “Branches of the same tree” - pandas mathplitlib and bumpy seaborn etc I guess pandas is the starting point
1
1
u/Sharmila_Zaman Nov 29 '20
of course, you should go with Panda. This is the best data analysis tool with python.
1
u/cheyen0609 Nov 29 '20
Data analysis, know your data, how to spot anomalies, how to cleanse the data. Understand the data source. Then reporting is (to me) about knowing how to piece the data together to get useful info out.
1
u/tmk_g Dec 01 '20
Check out the resources like Datacamp, Stratascratch and Leetcode. There you can test and improve your SQL and Python skills. Also, they provide exercises on Numpy and Pandas. Besides, you should also improve your logical skills, Mathematics, Modeling, and Tableau skills.
1
u/boris13579 Jan 07 '24
I started to learn data Analysis from this course:
https://www.udemy.com/course/dav-using-python-ag/?couponCode=ANI-77-003
You have to use coupon ANI-77-003 to get the discount.
It is a very good course, which helped me a lot to grasp data analysis and visualization.
God bless,
Boris
-1
Nov 28 '20
I would recommend Pandas, a lot of my videos have them and a lot of different coding scenarios feed off them, you'll learn a lot..
97
u/ectomancer Nov 28 '20
Watch pandas youtube videos such as 11 video series
https://www.youtube.com/watch?v=ZyhVh-qRZPA&list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS