r/learnpython • • May 10 '20

Beginner Python data analysis project - critique greatly welcome!

Hi everyone,

I just finished my first Python data analysis project, and being a total beginner, I could really use some other perspectives as to the quality of my final product. I plan on building a portfolio of similar projects and using it for applying to social science research assistant jobs where some programming and analysis ability is highly preferred. Since I am entirely self-taught, I need some concrete examples of what I can do to share with potential employers.

With that said, while I'm sure there are areas that could be improved, would my project be worth sharing with some edits? Or is it actually just a total flaming trash heap that should be scrapped entirely? Positive critique and helpful suggestions appreciated!

Project

Edit: Wow, I did not expect to wake up to so many comments and upvotes! I really needed this boost, to be honest. I'm so grateful for everyone's kindness and suggestions, and I'm definitely going to go back and try and tweak some things! I hope I can pay it forward one day, but for now, to fellow noobs who are trying to set up their first project - just go do it! I basically just forced myself to create something after learning the basics, and it really helps the learning process to break things down and figure out how they work outside of a guided tutorial. Keep coding!

Edit 2: I've uploaded a new version of the project since this was posted, taking into account some of the suggestions everyone has made. It's still not perfect, but I think it's time to move onto learning with new projects.

361 Upvotes

68 comments sorted by

View all comments

18

u/Babs12123 May 10 '20

This looks really good! A few thoughts:

  • When reading in csvs to pandas I find it useful to specify the encoding and the type (usually auto set to UTF-8 and object personally). Particularly when you're working with data which contains some text and some numeric columns, it's helpful to be explicit to avoid any unexpected behaviour.

  • When naming variables be explicit with regards to the data type, e.g. instead of 'clergydata' I would call this 'df_clergydata'. When you end up with multiple different lists, dicts and dfs in your code it's very helpful to have all of this explicitly named (particularly when you come back to your code a month later).

  • When creating column names in your df you created several which contain capital letters and spaces (e.g. clergydata['Age range']). It's better and easier to only use lower case letters in variable names/column names where possible and to use underscores instead of spaces. This lets you access the column using clergydata.age_range instead of clergydata['Age range'] in lots of situations when manipulating your df, which is often much quicker and easier.

  • In cell 12 you manually specify the archdiocese abbreviation and name (e.g. LA, and Archdiocese of Los Angeles) for many different locations. It would be better to automate this somehow, to both improve clarity and also reduce the risk of error/inconsistency. I saw someone above suggested using a group by, which would work, or you could use a for loop to directly create your top19_cathpops and top19_dionames lists. If you're not clear how to do this let me know and I would be happy to clarify.

Most importantly your code works and answers some interesting questions, but the above points will make things more explicit (which is always better) and make your life easier.

7

u/synthphreak May 10 '20

This lets you access the column using clergydata.age_range instead of clergydata['Age range']

The flip side of doing it this way is that it conflates column names with built-in methods/attributes. If there is no conflict between them, you’re fine. But df’s have a LOT of built-in methods/attributes, many of which you probably don’t know about... I can’t tell you how many times I’ve named a column items, and then later wasted 30 minutes debugging my code only to find out that df.items is already a thing. By contrast, df[‘items’] will ALWAYS and ONLY ever return the item column. Just something to think about.

2

u/Babs12123 May 10 '20

Yeah this is a good point - I haven't encountered this with df column names but have with other variables and it is very irritating to debug.

If you're using non-generic variable/column names then it shouldn't happen often but I agree it makes sense to use your own judgement here.