Programming courses often focus heavily on understanding code, while paying far less attention to understanding the program state. But code does not exist in isolation. Its main goal is to change the program state, before ultimately producing some output.

To develop an accurate mental model of program execution, students need to understand both:

• the instructions being executed

• the values, references, and data structures those instructions create and modify

Reading code alone does not always reveal how the program state changes during execution. That is why I created 𝗺𝗲𝗺𝗼𝗿𝘆_𝗴𝗿𝗮𝗽𝗵: a tool that visualizes the state of a Python program as it changes, step by step.

It can help explain a wide range of introductory Python topics. Here are just a few examples:

• Loops, Lists and Dictionaries

• Python Data Model

• Function Calls

• Recursion

• Algorithms

• Classes

• Custom Data Structures

Instead of reconstructing the program state from print statements, students can now watch it change as each line executes. This makes unfamiliar concepts easier to understand and bugs easier to fix.

Help your students learn Python programming more thoroughly and easily.

See: more examples

I keep seeing the same question: "Do I really need statistics for data science?"

Short answer: Yes.

Long answer: You can copy-paste sklearn code and get models running without it. But you'll have no idea what you're doing or why things break.

Here's what actually matters:

**Statistics isn't optional** - it's literally the foundation of:

• Understanding your data distributions

• Knowing which algorithms to use when

• Interpreting model results correctly

• Explaining decisions to stakeholders

• Debugging when production models drift

You can't build a house without a foundation. Same logic.

I made a breakdown of the essential statistics concepts for data science. No academic fluff, just what you'll actually use in projects: Essential Statistics for Data Science

If you're serious about data science and not just chasing job titles, start here.

Thoughts? What statistics concepts do you think are most underrated?

When I started, I thought it was all Python, ML models, and building beautiful dashboards. Then reality checked me. Here are the lessons that hit hardest:

1. Collecting resources isn’t learning; you only get better by doing.

2. Most of your time will be spent cleaning data, not modeling.

3. Explaining results to non‑technical people is a skill you must develop.

4. Messy CSVs and broken imports will haunt you more than you expect.

5. Not every question can be answered with the data you have  and that’s okay.

6. You’ll spend more time finding and preparing data than analyzing it.

7. Math matters if you want to truly understand how models work.

8. Simple models often beat complex ones in real‑world business problems.

9. Communication and storytelling skills will often make or break your impact.

10. Your learning never “finishes” because the tools and methods will keep evolving.

Those are mine. What would you add to the list?

I’ve been fortunate in my career to have interviewed and screened hundreds & hundreds of Data Science and Analytics candidates at Amazon, Sony, and other top tech companies. The types of behavioural questions you get are often very similar in nature. I’ve rewritten a few example questions below so they capture the style of questions without giving away anything confidential from those companies.

Also, to start, one important thing to understand as you read through these is to always remember that hiring managers are not just looking for technical answers, with these types of questions they are looking for how you think, how to justify decisions, how you structure ambiguity, and how you connect analysis to real decisions or value or outcomes.

Anyway, here are five example questions that can be great for preparing if you're at that stage of the process.

1. A key engagement metric on your product dropped 12% week-over-week. Walk me through how you would investigate

For this type of questions, what I'm really looking for is structured thinking. Good candidates usually start by clarifying the metric, the scope, and the timeline. Then they break the problem down logically. Things like segmenting by platform, geography, user cohort, feature usage, release timing, seasonality, experiment changes, etc.

A big signal here is whether you naturally "dive deep" into the problem instead of jumping to conclusions. In other words, can you somewhat methodically narrow the problem space until you find the likely root cause.

2. A product change increased revenue but reduced user engagement. How would you decide whether to keep the change?

This one is more about trade-offs and business judgment. Good answers usually talk about defining the real objective first. Are we optimising revenue, retention, long-term growth, or something else? I've found that strong candidates will also talk about things like segmentation, longer-term impacts, and possibly running controlled experiments. It's nice here to see that you are not just reporting metrics but thinking about the long-term impact of decisions.

3. You launch a new feature but adoption is much lower than expected. How would you approach this?

This question is looking to see how you connect product thinking with analytics (and if you do this at all). For this one, good answers typically explore things like discoverability, user friction, onboarding flow, messaging, or whether the feature actually solves a real user problem. The strongest candidates also bring the "customer" into the discussion. In good analytics teams, you always start with the user or customer and work backwards to a solution, so it's nice to see candidates think in that way.

4. Tell me about a time when you had to make an important decision even though the data was incomplete

This type of question comes up quite often. Data Scientist & Data Analysts are not always operating in perfect analytical environments and so sometimes you need to combine partial data, domain knowledge, and judgment to move forward. I like to see whether the candidate can make sensible decisions when the answer isn’t obvious, and whether they maybe considered alternative viewpoints before committing (if that makes sense)

5. Tell me about a time you investigated a complex problem and uncovered the real root cause

This one is less about specific modelling or algorithms and more about analytical curiosity. Strong answers for me here, usually involve seeing how the candidate dug through multiple layers of data, maybe questioned assumptions, and eventually might have connected several signals together.

One final piece of advice from me, for anyone preparing for these types of interviews, is that, many candidates focus entirely on technical preparation, but the really strong candidates combine this with analytics, product thinking, and communication.

They explain their reasoning clearly, structure their approach logically, and constantly connect their analysis back to business outcomes. In other words, the goal is not just to show that you can analyze data or apply code or algorithms, it's that you can show how you use your tools/skills/concepts/the data to drive good decisions or create business value.

Hope that helps if you're prepping for interviews!

Now, 2026 has almost come. I know a lot of people have defined that target for this year to become a data scientist or an AI engineer. The fact is that all companies in IT are also hiring mostly from these two roles only. In linkedin, I have seen a lot of queries regarding how to get ready for Data Science interviews because this area of study is really growing, and thus I wanted to give you all an extensive preparation guide, as this year I changed my tech stack to data scientist. This list is based on my actual interview experiences, as well as the help that I got from Linkedin and reddit etc., as well as companies like InterviewQuery, and it provides information about what to expect when interviewing at various companies. Data science interviews are normally different according to the role and the company level:

1. Recruiter Screen: Resume chat, experience, and salary expectations.

2. Online Assessment: Often 2-4 SQL or coding problems.

3. Virtual Screen: 1-2 rounds, 45-60 mins – SQL, stats questions.

4. Final Round: Hiring manager or team fit. The big tech companies like FAANG prioritize the areas of product analytics and experimentation, whereas newly founded companies might concentrate on the whole ML project cycle instead.

CORE SKILLS YOU MUST MASTER: Programming You must be fluent in:

● Python

● NumPy

● Pandas

● Scikit-learn

Writing clean, readable, bug free code

Data transformations without IDE help

Expect:

● Data cleaning

● Feature extraction

● Aggregations

● Writing logic heavy code

SQL

Almost every Data Science role tests SQL. You should be comfortable with:

● Joins - inner, left, self

● Window functions

● Grouping & aggregations

● Subqueries

● Handling NULLs

Statistics & Probability:

● Probability distributions

● Hypothesis testing

● Confidence intervals

● A/B testing

● Correlation vs causation

● Sampling bias

Machine Learning Fundamentals. You must know:

● Supervised vs Unsupervised learning

● Regression & Classification

● Bias Variance tradeoff

● Overfitting / Underfitting

Evaluation metrics:

● Accuracy

● Precision / Recall

● F1-score

● ROC-AUC

● RMSE

FEATURE ENGINEERING & DATA UNDERSTANDING:

● This is where strong candidates stand out.

● Handling missing data

● Encoding categorical variables

● Feature scaling

● Outlier treatment

● Leakage prevention COURSES:

1.) IBM Data Science Professional Certificate: A full scale series of courses teaching Python, SQL, data analysis, visualization, machine learning, and capstone projects that are perfect for novices developing industry required skills through practical applications and a certificate that can be shared.

2.) LogicMojo DS course: Offers lessons on Python, statistics, machine learning, and data analysis. Useful as a reference for learning core problem solving and project development and interview preparation.

3.) Codecademy: Free, rigorous university level courses offering deep theoretical insights into statistics, probability, and ML ideal for mastering the mathematical rigor expected in advanced DS interviews.

PRACTICE PHASE — THIS IS CRITICAL

● Practice writing code in Google Docs or a plain text editor.

● Explain your approach out loud while coding, as if an interviewer is present.

● Prioritize medium to hard-level problems over easy ones.

● Simulate real interview conditions: time limits, no external help, and clean code only.

Recommended Practice Platforms:

● Kaggle (datasets, notebooks, competitions)

● Google Colab (ML experiments)

● UCI ML Repository (real datasets)

● GitHub (end-to-end DS projects)

By means of proper readiness and practice, any Data Science interview can be faced with confidence. It is advisable to support theories with practical skills, evaluate your setbacks, and slowly but surely improve your problem solving technique. Consistency alongside reflection is what brings success.

I want to know something new about python and where i can use all this but I don't have any experience.

Ever felt confused about what to learn, where to start, or how to get into Data Analytics or Data Science?

During my beginner phase, I felt the same way. There were hundreds of videos, tutorials and PDFs, and I honestly didn’t know what to learn first or where to begin.

So I decided to create a handbook that brings together the essential beginner-level knowledge you’d need to get started, all in one place.

📚 The Ultimate Data Science Handbook – Volume 1 (328 pages)

📖 Sample pages: Attached

🛒 Get the complete handbook: https://pdfcart.in/p/ultimate-data-science-handbook-vol1

What’s included?

A structured learning resource with detailed explanations, step-by-step code walkthroughs, practical examples, different types of graphs with explanations, real-world applications, practice exercises, chapter summaries and interview questions.

The idea was to bring together concepts that beginners often have to search for across multiple PDFs, notes and tutorials, so you can follow a more organised learning path without constantly switching between resources.

PS.The content is from multiple valuable resources that i learnt from and was compiled by me .. AI was used to make the pdf.

I feel like every data science discussion revolves around Python, R, SQL, deep learning, or the latest shiny model. Don’t get me wrong those are super important.

But in the real world, I’ve noticed the “boring” skills often make or break a data scientist:

1. Knowing how to ask the right question before touching the data

2. Being able to explain results to someone who doesn’t care about statistics

3. Cleaning messy data without losing your sanity

4. Spotting when a model is technically “accurate” but practically useless

So, fellow data peeps, what’s the one underrated skill you wish more people talked about (or that you learned the hard way)?

I want to learn data science and become a data analyst. Preferably from free online sources. Bear in mind I come from a mechanical engineering background. So I am not familiar with software or any programming language. The sources need to start from the most basic level because of it.

Thank you in advance.

Hello Everyone,

I have created a Data Science roadmap repository on GitHub.

This repository intended to provide a complete Data Science learning path to those who intersted in learning Data Science. In this repository, I gave preference to free resource. However, some valuable paid courses also included.

GitHub repo: https://github.com/durgeshsamariya/data-science-roadmap

Note: This repo is still under progress.

Background: As a senior data scientist / ML engineer, I have been both individual contributor and team manager. In the last 6 months, I have been full-time building AI agents for data science.

Recently, I see a lot of stats showing a drop in junior recruitment, supposedly “due to AI”. I don’t think this is the main cause today. But I also think that AI will automate a large chunk of the data science workflow in the near future.

So I would like to share a few thoughts on why data scientists still have a bright future in the age of AI but one needs to learn the right skills.

This is, of course, just my POV, no hard truth, just a data point to consider.

LONG POST ALERT!

Data scientists will not be replaced by AI

Two reasons:

First, technical reason: data science in real life requires a lot of cross-domain reasoning and trade-offs.

Combining business knowledge, data understanding, and algorithms to choose the right approach is way beyond the capabilities of the current LLM or any technology right now.

There are also a lot of trade-offs, “no free lunch” is almost always true. AI will never be able to take those decisions autonomously and communicate to the org efficiently.

Second, social reason: it’s about accountability. Replacing DS with AI means somebody else needs to own the responsibility for those decisions. And tbh nobody wants to do that.

It is easy to vibe-code a web app because you can click on buttons and check that it works.

There is no button that tells you if an analysis is biased or a model is leaked. So in the end, someone needs to own the responsibility and the decisions, and that’s a DS.

AI will disrupt data science

With all that said, I already see that AI has begun to replace DS on a lot of work.

Basically, 80% (in time) of real-life data science is “glue” work: data cleaning and formatting, gluing packages together into a pipeline, making visuals and reports, debugging some dependencies, production maintenance.

Just think about your last few days, I am pretty sure a big chunk of time didn’t require deep thinking and creative solutions.

AI will eat through those tasks, and it is a good thing. We (as a profession) can and should focus more on deeper modeling and understanding the data and the business.

That will change a lot the way we do data science, and the value of skills will shift fast.

Future-proof way of learning & practicing (IMO)

Don’t waste time on syntax and frameworks. Learn deeper concepts and mecanisms. Framework and tooling knowledge will drop a lot in value. Knowing the syntax of a new package or how to build charts in a BI tool will become trivial with AI getting access to code sources and docs. Do learn the key concepts and how they work, and why they work like that.

Improve your interpersonal skills.

This is basically your most important defense in the AI era.

Important projects in business are all about trust and communication. No matter what, we humans are still social animals and we have a deep-down need to connect and trust other humans. If you’re just “some tech”, a cog in the machine, it is much easier to replace than a human collaborator.

Practice how to earn trust and how to communicate clearly and efficiently with your team and your company.

Be more ambitious in your learning and your job.

With AI capabilities today, if you are still learning or evolving at the same pace, it will be seen later on your resume.

The competitive nature of the labor market will push people to deliver more.

As a student, you can use AI today to do projects that we older people wouldn’t even dream of 10 years ago.

As a professional, delegate the chores and push your project a bit further. Just a little bit will make you learn new skills and go beyond what AI can do.

Last but not least, learn to use AI efficiently, learn where it is capable and where it fails. Use the right tool, delegate the right tasks, control the right moments.

Because between a person who boosted their productivity and quality with AI and a person who hasn’t learned how, it is trivial who gets hired or raised.

Sorry, a bit of ill-structured thoughts, but hopefully it helps some more junior members of the community.

Feel free if you have any questions.

I've been working on a tutorial on Mathematics for Machine Learning, and there's something I've been thinking about for a while.

Whenever people talk about the mathematics required for Machine Learning, the usual answer is:

• Linear Algebra

• Calculus

• Probability and Statistics

And they're absolutely right. These are fundamental to understanding how Machine Learning works.

But I think there's a problem with how we approach teaching these subjects.

Most resources assume you already have a reasonably good mathematical foundation.

They'll start explaining vectors, matrices, derivatives, gradients, probability distributions, and all sorts of other concepts. But what if you're someone who struggled with mathematics in school? What if you barely remember algebra? What if equations have always looked more intimidating than they should?

Where do you start?

I think we often overlook this group of learners.

Not everyone entering Machine Learning has a strong mathematical background. Some are developers who want to transition into ML. Some are students who discovered an interest in AI but aren't particularly confident in mathematics. Others simply want to understand what's happening behind the models rather than treating libraries as magic boxes.

And telling these people to just learn Linear Algebra, Calculus, and Statistics isn't particularly helpful if they don't have the foundations to understand them.

So I decided to approach this differently.

Instead of jumping straight into Linear Algebra, I've started building a Mathematics for Machine Learning tutorial from the ground up.

We're starting with algebra.

Yes, algebra. Variables, expressions, equations, exponents, logarithms, functions, graphs, and all those things that many of us probably haven't thought about since school.

From there, we'll gradually move towards:

1. Algebra and mathematical foundations

2. Functions and graphs

3. Linear Algebra

4. Calculus

5. Probability and Statistics

6. Optimization and Numerical Methods

The idea isn't to turn this into a school mathematics textbook or make people solve hundreds of equations without understanding why.

I want to focus on:

• Intuition first: Understand what a concept actually means before introducing complicated notation.

• Derivations: Understand where mathematical formulas come from instead of memorizing them.

• Worked examples: Break down problems step by step.

• Visual explanations: Use graphs and visualizations wherever they help.

• Python experiments: Connect mathematical concepts to actual code.

• Machine Learning applications: Understand where and why these concepts appear in ML algorithms.

Ultimately, I want someone to be able to look at a mathematical expression in a Machine Learning paper or an algorithm and understand what's happening, rather than immediately reaching for Google to find out what every symbol means.

I also want to be clear that this isn't going to be a shortcut. Mathematics takes time, and there's no getting around that. The goal is to make the journey less intimidating and more understandable.

I'm building this primarily for people who want to get into Machine Learning, Deep Learning, and eventually LLMs but feel that their mathematical background is holding them back.

I've started publishing the tutorial here:

https://mathsforml.vercel.app/

It's a work in progress, and I'll be expanding it as I develop more chapters and topics.

My philosophy is simple: You shouldn't have to be good at mathematics before you start learning it. You just need a place that teaches it properly, from where you are right now.

I'm curious to hear from others:

• Did you struggle with mathematics when you started learning Machine Learning?

• Do you think most ML mathematics resources assume too much prior knowledge?

• What concepts do you wish someone had explained properly before you started learning ML?

I'd also appreciate suggestions on topics that should be included in the tutorial. Feedback on the existing material is very welcome!

I have been teaching probability and statistics to first-year graduate students and advanced undergraduates in data science for a while (10 years). 

At the beginning I tried the traditional approach of first teaching probability and then statistics. This didn’t work well. Perhaps it was due to the specific population of students (with relatively little exposure to mathematics), but they had a very hard time connecting the probabilistic concepts to the statistical techniques, which often forced me to cover some of those concepts all over again.

Eventually, I decided to restructure the course and interleave the material on probability and statistics. My goal was to show how to estimate each probabilistic object (probabilities, probability mass function, probability density function, mean, variance, etc.) from data right after its theoretical definition. For example, I would cover nonparametric and parametric estimation (e.g. histograms, kernel density estimation and maximum likelihood) right after introducing the probability density function. This allowed me to use real-data examples from very early on, which is something students had consistently asked for (but was difficult to do when the presentation on probability was mostly theoretical).

I also decided to interleave causal inference instead of teaching it at the very end, as is often the case. This can be challenging, as some of the concepts are a bit tricky, but it exposes students to the challenges of interpreting conditional probabilities and averages straight away, which they seemed to appreciate.

I didn’t find any material that allowed me to perform this restructuring, so I wrote my own notes and eventually a book following this philosophy. In case it may be useful, here is a link to a free pdf, Python code for the real-data examples, solutions to the exercises, and supporting videos and slides:

https://www.ps4ds.net/

I am a manager of one of top pharma companies in the states. An opportunity expanding my team came and was having conversation with HR. HR started requirement conversation with “No visa holders, US citizen or green card holder only due to the current political landscape”.

I learned people lying in their application like they wouldn’t need visa sponsorship when they actually need, to just see if they can get away with it. It’s sad but it will take a long time to find the right talent. I see a ton of applications coming in with international background.

Just wanted to inform folks the hiring sentiment in DS job market. It started.

20 years in data science. Master’s in the USA. Worked with large North American clients, big banks (JPM, HSBC, Equifax), then leadership roles at startups + Fortune 50 work.

Most people don’t fail in DS because they’re bad at math or Python.

They fail because they’re trained to: collect tools memorize algorithms chase courses

…instead of learning how to think like a data scientist.

Real DS is about: framing messy problems knowing when not to model understanding how wrong is “too wrong” explaining tradeoffs to non-technical people dealing with models breaking in prod

Almost no beginner course teaches this.

So I’m starting a small Data Science cohort.

Yes, beginners are welcome — but the goal is to train people to become real data scientists, not tutorial addicts or certificate collectors.

No bootcamp hype. No random courses. Just how the job actually works.

If this resonates and you want details, DM me.

Curious: what’s the worst DS course you’ve paid for? what do you wish you’d learned first?

Hi everyone,

I’m 19 (turning 20 soon) and I’m really passionate about getting into Data Science. Right now, due to some personal reasons, I can’t continue my degree, but I don’t want that to stop me from learning.

I’ve started learning Python and I’m planning to move into math/stats and projects next. My questions are:

• Does not having a degree make it impossible to get into Data Science?

• What’s the best path for someone like me who’s self-studying?

• Should I focus more on building projects, certifications, or freelancing skills?

I’d love to hear from people who’ve gone through non-traditional paths or have advice for someone in my situation. I’m really motivated to make this work, just need some direction.

Thanks so much 🙌

Back in 2017 I landed in Australia with two postgraduate degrees, a PhD candidature at University of Sydney, and zero commercial experience in anything.

The PhD fell apart. Over $200,000 in funding gone. I downgraded to an MPhil and started applying for jobs.

80 rejections later I still had nothing.

Recruiters kept saying the same thing. "Great background but we need someone with local commercial experience." I had more academic credentials than most people in the room and could not get an entry level job.

My wife was working in data. She looked at my situation one evening and said the tools are learnable, the market needs people, just start.

So I did. From absolute zero.

Here is what the actual sequence looked like for me, not what courses tell you, what genuinely got me from unemployed to Senior Data Engineer in six years.

Year 1: SQL and Excel only. Not because it was the perfect starting point. Because every single entry level data job I could apply for listed those two things. I stopped following learning roadmaps and started reading job descriptions instead. That one shift saved me probably a year of learning the wrong things.

Got a casual data management role. Small title. Real data. Real problems. That job was worth more than any course I ever took because it gave me context for everything I learned after.

Year 2: Power BI. The analyst roles I wanted all listed it. So I learned it while working. Not from a course start to finish. From a real dashboard I needed to build for an actual stakeholder.

Year 3: Python. Not for machine learning, not for AI. For automating the boring reporting work that was eating my Mondays. That practical reason made it stick in a way that six previous attempts at Python courses never did.

Year 4 and 5:SQL got deeper, data modelling, pipelines, moving from analyst work into proper data engineering. Picked up Azure tools on the job.

Year 6: MS Fabric and Databricks. Senior contractor level. These tools finally made sense because I had four years of context underneath them.

This is the part nobody says clearly enough. MS Fabric and Databricks are not beginner tools. But in the age of AI they can be learned faster now.

The thing that actually worked was simple. At every stage I asked one question. What does the next job I want actually need. Then I learned exactly that and nothing else until I had the job.

Two master's degrees never got me hired. Learning the right tool for the right role at the right time got me hired every single time after that.

Anyone else figure this out the hard way or did you find a smarter way in from the start?

Okay so this is a bit embarrassing to write out but here it is.

When I started trying to get into data analytics I did everything you are supposed to do. Finished three online courses. Built some projects. Put them on GitHub. Tailored my resume for every single application. Wrote cover letters that I genuinely thought were good. Applied to probably 80 roles over 18 months.

Nothing.

Well not nothing. A few interviews. But nothing that converted. And the feedback I kept getting was so vague it was almost useless. "We went with someone with more commercial experience." Okay cool, how do I get commercial experience if nobody gives me commercial experience. Classic loop.

The frustrating part was I was not being lazy. I was genuinely working hard. Like staying up late, redoing my resume every two weeks, reading every career advice thread I could find kind of hard.

But I was working hard in completely the wrong direction and I did not know it.

Hmm. So what actually changed things.

My wife said something one evening that sounds obvious in hindsight but genuinely had not occurred to me. She said stop reading career advice and start reading job descriptions. Find the twenty postings closest to what you want. Write down every tool and skill that appears more than three times. Learn exactly those things. Nothing else.

That was it. That was the whole insight.

Took me two weeks to do that exercise properly. Realised I had spent two months learning a tool that appeared in maybe three out of fifty postings I was actually targeting. Two months. Gone.

Shifted focus completely. Three months later I had my first data role.

Ahh and the other thing that wasted a huge amount of my time was applying broadly. I genuinely thought volume was the strategy. More applications equals more chances. Nope. It just means more time writing cover letters for roles you are not quite right for yet instead of actually getting right for the roles you actually want.

Six years later I am a Senior Data Engineer and I still use the same logic. Read what the market is actually asking for. Build toward that specific thing. Everything else is noise.

Curious if anyone else figured this out early or if you went through the same painful loop I did.

Hello, I'm looking for the best data science courses for beginners, all the way to intermediate/advanced levels, with Python. I have no problem with the course including AI/ML or any extra material. Websites like Udemy, Coursera, etc. No problem with paid courses.

Thank you for your help.

I think it would be greatly appreciated by people who are just starting out and are intimidated by this enormous field

I kept seeing beginners learn individual Pandas commands but still struggle with what order to actually use them in when working with a real dataset.

So I put together a simple workflow I use as a reference:

1. Load
read_csv() / read_excel()

2. Inspect before changing anything
head()
shape
info()
describe()
isna().sum()
duplicated().sum()

3. Clean

• standardize column names

• handle missing values based on what they actually mean

• remove genuine duplicates

• convert dates/numbers safely

4. Transform
assign()
map()
pd.to_datetime()
np.where()

5. Summarize
groupby()
agg()
transform()

6. Combine datasets
merge() / join() / concat()

The part I think beginners often miss is validation after the merge.

A query/script can run perfectly and still give the wrong answer if a many-to-many merge quietly multiplies your rows.

Useful checks:

validate="many_to_one"
indicator=True
compare row counts before/after
check key uniqueness before merging

A few other mistakes worth watching for:

• treating missing values as automatically equal to 0

• merging keys with different data types

• confusing count() with size()

• using median filling without understanding why values are missing

• assuming no Python error means the analysis is correct

I wrote a more detailed version with examples for Pandas, NumPy, GroupBy, merges and an end-to-end workflow.

Learn more: https://www.pangaeax.com/blogs/python-data-analysis-cheat-sheet/

Anything important you’d add to this workflow, especially something you learned the hard way when working with messy data?

Disclosure: This is a PangaeaX article and I’m connected with PangaeaX.