r/learnpython • • Nov 28 '20

Data analysis for beginners

Hi all,

What would you recommend as the best route to start learning towards data analysis with Python (I will add that I'm not planning on becoming data analyst/scientist within few months or so - I started learning Python 4 months ago, got the basics, not interested in building websites, creating games, but I would like to explore data; I also already started learning SQL 2 months ago. Really taking it easy and want to see where it all leads to, but extremely happy that I'm learning new skill).

Should I start with Matplotlib? Or Numpy? Or Pandas?

Any other tips or recommendations?

Thank you for your answers in advance!

330 Upvotes

35 comments sorted by

97

u/ectomancer Nov 28 '20

32

u/Matheos7 Nov 28 '20

Corey...that man cannot be praised enough. Learned a ton from his videos, will certainly watch this then.

What would you recommend after it? Matplotlib or NumPy? Or something else? Thanks!

16

u/Mondoke Nov 28 '20

On my experience, matplotlib is more useful than numpy, but it will depend on where you are going to work.

Sklearn is a very nice addition to have, but you need to have a background in statistics (which, for data analysis, is already relevant).

Also, there are some non python skills that you will need to have, because the companies you'll work will have a data structure of some kind. So, knowing Excel to an intermediate level and SQL is a big plus. For Excel, knowing some power query will make some tasks like fuzzy matches way easier than using python.

6

u/gottsc04 Nov 29 '20

Pandas definitely seems to be the most popular data science tool, it has loads of documentation and you can just do so much with it.

If you know R, for some reason I found matplotlib fairly similar in its basic uses. Granted I was learning them both at the same but in different courses. I'm still learning python but I think numpy is better suited for some machine learning uses. Take what I say with a grain of salt though! Good luck

4

u/synthphreak Nov 29 '20

IME you don't actually need to know much numpy directly unles you will be manually coding up math. Math aside, most other libraries do the hard numpy work for you under the hood, such as pandas which is essentially just a wrapper around numpy.

Matplotlib is a great one to know if your analysis will involve plotting, which it should. Others might recommend seaborn, but I don't know seaborn, and I have been perfectly fine. Matplotlib vs. seaborn is probably just a preference thing rather than anything objective. You can also do some simple plotting with just pandas.

As someone who analyzes large data sets for a living, I honestly am just fine knowing pandas. You can get a loooooooot of miles out of being fluent with pd.DataFrame. The cited Corey Schafer playlist is a good one for this. Another excelent resource, which you can access for free if you don't care about certificates, is this Coursera course (although I'd recommend paying to get access to the assignments which are super, super helpful).

2

u/[deleted] Nov 30 '20

Matplotlib is a great one to know if your analysis will involve plotting, which it should. Others might recommend seaborn, but I don't know seaborn, and I have been perfectly fine. Matplotlib vs. seaborn is probably just a preference thing rather than anything objective

Seaborn builds on top of matplotlib. It starts with changing a lot of the matplotlib defaults to be a bit more 'modern', then it adds more higher level functions for analyzing relationships in multiple dimensional data. But you need to know Matplotlib to fully understand seaborn, as you can still interact with plt using the new seaborn settings.

People should start with matplotlib, but seaborn has some niche uses in analysis that make it a good idea to learn.

1

u/synthphreak Nov 30 '20

Good points. Have some karma.

1

u/nojustlurkingty Nov 28 '20

And I also agree with this. Corey Schafer is the man and I've learned a lot being about halfway through this series

3

u/[deleted] Nov 29 '20

how would you compare that with the Kaggle course at https://www.kaggle.com/learn/pandas?

45

u/MikeDoesEverything From zero to Data Engineer: https://tinyurl.com/2p8wd6y7 Nov 29 '20

As somebody who is a scientist and worked with data scientists, aside from learning the techniques I would also try and find something out about how to approach problems. I'll list the mistakes data scientists made when we were working together:

  • Assuming any trend was a fact. The DS team were very new, to be fair, and were finding what was essentially random correlation in data and trying to attribute causation to it. They would have discovered the vast majority of their reasoning did not have any scientific basis to it if they simply asked the R&D team. That being said, they spent a lot of time trying to figure out the meaning behind trends. This is the classic case of causality does not equal correlation. Leading onto next point.

  • Not asking for any external input on their analyses. The DS team really should have been better at communicating and, instead, they were more keen on selling data analysis. We all went on these courses with little to no experience to use software for analysis although the whole thing was a mess. We had a lot of people drawing wild conclusions from small sets of data with zero skepticism because their findings made them look good and it all became very political very quickly. Goes nicely onto my next point.

  • Not questioning data. Slightly different from the first one in the sense the DS team never really challenged their own ideas or even sought to test it and back it up with experimental data. Their argument would be "we've run this simulation 5000 times on the computer and it has a 98+% correlation" although what happens in the program doesn't always reflect what happens in real life. Always be willing to accept challenges on your findings and also question your own data.

tldr: Talk to your subject matter experts about trends and see if they check out. Do not assume all missing data is bad. Do not assume all collected data is good. Always be asking questions about your own data.

20

u/foyslakesheriff Nov 29 '20 edited Nov 29 '20

Dude, you need to learn the math. I say this to you because it's easier than saying it to myself, and I need to hear it more often. Have a good handle on statistics, linear algebra, and calculus. It's not sexy but it's necessary to not waste your time in the long term. I hate hearing that because I love learning new Python frameworks. After learning the theory, the tools will make more sense. People might say that we have those tools so that we don't have to reinvent the wheel each time. That is true. But you still gotta know how a wheel works, otherwise you won't know how to diagnose and treat an issue when you get unexpected results.

If you don't understand linear algebra, then you won't understand why you're using Pandas. If you don't understand some regression models, scikit-learn won't be useful. Et cetera

It's not so much what you start with, because you will need to know all of those tools and use them together. Pandas uses numpy under the hood so to speak. Matplotlib will make your data output pretty. You might need SQL. I would start small, and create a few exploratory jupyter notebooks that use all those packages at least a bit.

Having said that, watch Andrew Ng's stuff.

3

u/Rainbow_Moonbeam Nov 29 '20

There's an edx micro masters series of courses by the university of San Diego which I'm working through. There's four in total: Python for Data Science, Statistics, Machine Learning, and Spark for Big Data. I'm halfway through the stats one at the moment.

The first course had a much stronger focus on "Python" than "data science" - kind of dipping your toes into it and seeing what sorts of things you could do. A bit of pandas, a bit of numpy, a quick intro to machine learning, and a few mini projects.

The second course is way more data science than python. It's almost all maths with a tiny bit of how to do the same thing but in python. It's pretty tough (micro masters assumes you have university level maths experience and I stopped studying maths at 18) but I'm looking forward to understanding data better and knowing what's relevant/statistically significant etc.

I enjoyed the first course but kind of felt like all my analysis was a bit silly and superficial. I could find correlations but I didn't know if it was important. I could copy the way the teachers used machine learning but didn't know if I was using the right model for the task or how to find more in depth analysis. I'm hoping that after I finish the next few courses, I'll be able to understand this all better.

17

u/nojustlurkingty Nov 28 '20

Python Crash Course (book) section 1 (python basics) and then Project 2 (data analysis)

Covers numpy, matpotlib, and data visualizations

I think matpotlib, numpy, and pandas are equally as important

6

u/acroporaguardian Nov 29 '20

Dont forget to learn statistics

3

u/MrCoderModer Nov 29 '20

I would say learn pandas as well! Pandas is going to give us the skills to start manipulating data in Python. I would say watch Kevin Markham.

https://www.youtube.com/playlist?list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y

3

u/SandyPointRigs Nov 29 '20

The Francois Chollet book on Keras & machine learning is free online and very good. Lots of examples and things to try & learn from.

3

u/pugwala Nov 29 '20

Data analysis begins with data cleansing, and data cleansing is the daily grind for most data scientists. “Garbage in, garbage out” as it is said, is the first rule. 100% of the data cleansing tutorials and walkthroughs that I have found through Google, Medium/Towards Data Science, StackOverflow, and YouTube were never sufficient for my data cleansing needs. Most tutorials start with highly curated datasets/sources to begin with and then go from there. Develop your own data cleansing pipeline and use it with all your needs. Issues to consider:

  • bias hard baked into the datasets (are they biased to a desirable outcome?)

  • sparsely filled data/records (to impute or not impute, convert NaNs, drop/ignore - all of which adds additional bias to your data and results)

  • multiple language support (commonly on the same webpage)

  • markup & script languages

  • email addresses & hyperlinks

  • numbers, conversions, dates, timeframes & varying data types

  • text processing issues: anagrams, stopwords, capitalization, domain specific terms, uni/bi/trigrams

  • categorical features

  • scaling

And so much more. Create your own data cleansing pipeline and get to really know your own data.

Become familiar with Pandas, but don’t build your processing pipelines around it. Pandas can be frustratingly inconsistent in methods and functions. Keep in mind that one way to think about Pandas is that it is a container of arrays, and as such may be evaluated in lots of very familiar and consistent ways.

RandomForest is your friend. I always say “no data scientist gets fired for using RandomForest”. This is a global truth. There are a lot of algorithms and models that a far superior, and some data scientists will poo-poo its use, but I always respond saying “it’s computationally efficient and reliably verifies which insights are valuable with almost zero effort”. Your director, or cost center owner, will love you.

Scikit-Learn is a great API toolkit to build upon. Yes, many of the algorithms are a bit dated and newer boutique versions provide better results. It it gets you a long way down the road of good, better, best.

Lastly, remember that 12 data scientists in a room will have 12 different opinions of why there approach is superior. Unless you are in academia, don’t let perfection get in the way of good.

I hope this helps.

2

u/Zack_jeff Nov 29 '20

Hey. So I’ve been a data scientist for 3 years now and for data you need all of the libraries you just listed: NumPy, Pandas, Matplotlib, seaborn.

But I would definitely start with pandas because that’s what you mainly use to even access the data, or you can use os

You can explore your data needs with only learning pandas. Pandas has a wide variety of tools to help you with this.

2

u/_Hawks_20 Nov 29 '20 edited Nov 29 '20

Follow this order for learning - numpy, pandas, metplotib. Link to resources:

  1. Numpy - https://youtu.be/QUT1VHiLmmI

  2. Pandas - https://www.youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS

  3. Metplotib - https://www.youtube.com/playlist?list=PL-osiE80TeTvipOqomVEeZ1HRrcEvtZB_ Or Learn all in this video - https://youtu.be/r-uOLxNrNk8

2

u/youaregorgousbooboo Nov 28 '20

Pandas is probably the most important

2

u/kuthedk Nov 29 '20

lol. you're going to end up using all of those that you asked about. so you best start learning how to use all of them. no one is more important than the other. they have different functions and you need to learn when to use them and when one is better than the other for a given situation.

try to start to analyze the data that you are interested in, and find the ways that help explain the data that you are looking at. All of those things you have asked about in regards to Python are called libraries. They are basically extensions of code that someone already spent a long time working on to make sure that they work, in other words, they are your tool box.

1

u/mightykev Nov 29 '20

I’m starting out as well actually going to a program at uni. Found it’s more about the concept than the tool. I’m having hard time on the statistics theory, linear algebra concepts vector, dimensions, than the mechanics in python or R.

1

u/219jw Nov 29 '20

Follow..

1

u/antirabbit Nov 29 '20

If your goal isn't just cleaning data, but extracting some statistics/trends, I would look at beginner statistics courses/guides. That can help you focus more when you have a large amount of data but only need to show the most "important" parts/trends.

Also, learning matplotlib is useful if you want to visualize data in Python. I usually write a separate R script for that, since most graphs are much, much easier to make in R, but you can basically find an example for most kinds of graphs from matplotlib's examples page here.

Also, learning SQL/a Python wrapper to SQL can help with working with larger datasets (reasonably up to 10s of GB on a personal computer). I usually use Postgres/the psycopg2 library. I'd recommend just using SQLite3 to start off with.

1

u/Far_Inflation_8799 Nov 29 '20

These python libraries are all “Branches of the same tree” - pandas mathplitlib and bumpy seaborn etc I guess pandas is the starting point

1

u/0161WontForget Nov 29 '20

This is an area I need to improve on so I’ll save this

1

u/Sharmila_Zaman Nov 29 '20

of course, you should go with Panda. This is the best data analysis tool with python.

1

u/cheyen0609 Nov 29 '20

Data analysis, know your data, how to spot anomalies, how to cleanse the data. Understand the data source. Then reporting is (to me) about knowing how to piece the data together to get useful info out.

1

u/tmk_g Dec 01 '20

Check out the resources like Datacamp, Stratascratch and Leetcode. There you can test and improve your SQL and Python skills. Also, they provide exercises on Numpy and Pandas. Besides, you should also improve your logical skills, Mathematics, Modeling, and Tableau skills.

1

u/boris13579 Jan 07 '24

I started to learn data Analysis from this course:

https://www.udemy.com/course/dav-using-python-ag/?couponCode=ANI-77-003

You have to use coupon ANI-77-003 to get the discount.

It is a very good course, which helped me a lot to grasp data analysis and visualization.

God bless,

Boris

-1

u/[deleted] Nov 28 '20

I would recommend Pandas, a lot of my videos have them and a lot of different coding scenarios feed off them, you'll learn a lot..