r/datacleaning • • Apr 04 '18

Data Science Interview Guide

Thumbnail
medium.com
14 Upvotes

r/datacleaning • • Jun 27 '15

16 Free Data Science Books [x-post from /r/machinelearning]

Thumbnail
wzchen.com
11 Upvotes

r/datacleaning • • Dec 16 '15

Bad data guide : problems seen in real-world data along with suggestions on how to resolve them.

Thumbnail
github.com
10 Upvotes

r/datacleaning • • Nov 26 '15

Open Refine - A open source tool from google to clean data and connect it with open data

10 Upvotes

Video Demo: youtube

More info at: OpenRefine


r/datacleaning • • Jul 14 '20

Full cleaning tutorials

9 Upvotes

So last week I found a YouTube video where a guy went through a full set data cleaned and wrangled it and asked the questions he was trying to answer. Let you try to clean and wrangle the data and then did it. It was a great video for learning. I was wondering if there is any other videos that you know of where some take a large set up data and cleans and wrangle and lets you try and wrangle it/clean ahead of time.

Ps I have found many tutorials of little training videos I am looking for large data sets and full working through all the steps as you tackle a real world problem!


r/datacleaning • • Dec 13 '19

Data Cleaning Guide: Saving 80% of Your Time to Do Data Analysis

Thumbnail
finereport.com
9 Upvotes

r/datacleaning • • Jan 15 '26

Working on an offline Excel data-cleaning desktop app

8 Upvotes

Hi everyone 👋 following up on my last post.

I’m continuing work on a desktop app for cleaning and organizing messy datasets locally (everything runs on your PC — no AI, no cloud).

Current capabilities include:

  • Detecting common data inconsistencies

  • Fast duplicate identification and removal

  • Column-level formatting standardization

  • Exporting cleaned data in multiple formats

I’ve added an Excel data preview and recorded a short clip showing the current flow. More improvements are in progress.

As before, I’d appreciate feedback from people who deal with real-world datasets — especially anything that would make this more practical in daily workflows.

Thanks.


r/datacleaning • • Sep 24 '20

Mindful data wrangling

Thumbnail
medium.com
8 Upvotes

r/datacleaning • • Jun 25 '19

Data extraction from scanned documents

8 Upvotes

I've been tasked with coming up with an automated way of processing a large number of scanned documents and extracting key data items from these docs.

The majority of these are scanned PDFs of varying quality and wildly varying layouts. The data elements im looking to extract are somewhat standardized. Some examples to illustrate : I need to extract client name and that might be recorded in the document as "Client : client X", "client name: client x", "CName: client X". Similarly, to extract invoice date I would look for "invoice date : mmddyyy", "treatment date : dd-MM-yy", "incall date - ddmmyyyy" etc..etc..

I've implemented a solution in R that :

  1. Converts a scanned pdf to PNG

  2. Uses Tesseract to run OCR

  3. Uses Regex to extract key data items from the extracted text (6 to 15 items per document, depending on the document type)

Each document type will have a slightly different way the data needs to be extracted. I have created functions to extract individual items e.g. getClientName(), getInvoiceDate() and then combine these into a list, so that for each document I get the extracted items.

The above works, for most of the simple docs. I can't help feel that regex is a bit unwieldy and might not generalize to all cases - this is supposed to be a process that will used across my organization on a daily basis. My aim is to expose this extraction service as an API so that users in my organization can send pdf, images or text and my API returns key data in JSON.

This is a very specific use case, but I'm hoping there are others out there that have dealt with similar scenarios. Are there any tools or approaches that might work here? Any other things to be mindful of?


r/datacleaning • • May 09 '17

How to Engineer and Cleanse your data prior to Machine Learning | Analytics | Data Science

Thumbnail
acheronanalytics.com
8 Upvotes

r/datacleaning • • Jan 12 '16

Show /r/datacleaning: R in Visual Studio

Thumbnail
youtube.com
8 Upvotes

r/datacleaning • • Oct 29 '25

Dirty/Inconsistent data (in-flight transforms, defaulting, validation) - integration layer vs staging DB

7 Upvotes

Your go-to approach for cleaning or transforming data in-flight during syncs - do you run transformations inside your integration layer, or push everything into a staging database first?


r/datacleaning • • Sep 02 '25

How much time do you spend cleaning messy CSV files each week?"

7 Upvotes

Working with data daily and curious about everyone's pain points. When you get a CSV with:

  • Duplicate rows scattered throughout

  • Phone numbers in 5 different formats

  • Names like "john SMITH", "Mary jones", "BOB Wilson"

  • Emails with extra spaces

How long does it usually take to clean? What's your current process?

Asking because I'm exploring solutions to this problem 🤔


r/datacleaning • • Dec 23 '25

I stopped fixing missing values. I started watching them.

6 Upvotes

I noticed something uncomfortable about how I handle missing values.

I almost never look at them.
I just:

  • drop rows

  • fill with mean / mode

  • move on and hope nothing breaks

So I built a tiny UI experiment that forces me to see the damage before I “fix” anything.

What it does:

  • upload a CSV

  • shows missing data column by column

  • visually screams when a column looks risky

  • lets me try a fill and instantly see before vs after

No rules. No schemas. No “AI knows best”.
Just: “Here’s what your data actually looks like — are you sure?”

It made me realize something:

Curious how others do this honestly:

  • Do you inspect first?

  • Or do you auto-fix and trust yourself?

I put the UI + code here if you want to see what I mean:

https://github.com/Abhinay571/missing-data-detector/commit/728054ff5026acdb61c3577075ff1b6ed4546333


r/datacleaning • • Feb 29 '24

Looking to create a "Clean Data" definition

6 Upvotes

Hi,

Just wondering what requirements or checklist items people would suggest for a definition of Clean Data ready to be used in machine learning? Akin to "tidy data", but for modelling. I.e.

  • There should be no string fields. All data should be either in a numeric form, or as a categorical data type etc

I know this will likely be opinionated, hence wanting to "crowd source" it 😃

Feel free to disagree with any statements, as I imagine there will be differences


r/datacleaning • • Jul 01 '23

How to get started with python for data analysis?

6 Upvotes

If you're embarking on the odyssey of studying Python data analysis, commence by acquiring a mastery of the rudiments of Python programming.

Once you've attained a level of proficiency with Python, plunge into the depths of indispensable libraries such as NumPy for numerical computation and Pandas for data manipulation. Engage in practical exercises utilizing authentic datasets to accrue experiential knowledge, and refine your prowess in data visualization employing Matplotlib and Seaborn.

Delve into the realm of statistical analysis using the comprehensive tools provided by SciPy, and contemplate augmenting your skill set with other pertinent libraries such as scikit-learn for machine learning. Engross yourself in online communities, undertake ambitious projects, and perpetually pursue learning and diligent practice to ascend to a zenith of expertise in Python data analysis—a gratifying pursuit that unveils the portals to unearthing invaluable insights from data.

To get you started, I will highly recommend you look at these articles.

Exploratory Data Analysis and visualization practical example:

https://link.medium.com/FYuBpTyvCAb

Data cleaning with python (a practical example)

https://link.medium.com/GBsdtEFvCAb

How to make data Visualization in python

https://link.medium.com/6rWH2nKvCAb

Python data cleaning made easy

https://link.medium.com/6rWH2nKvCAb

Sales Statistical analysis with python

https://link.medium.com/ZGx7NDRvCAb

https://link.medium.com/OidaOBUvCAb

Python Web App Development: Unleashing the Power of Simplicity and Flexibility

https://medium.com/@mondoa/python-web-app-development-unleashing-the-power-of-simplicity-and-flexibility-d34e9d1bd658

Enhancing Your Web Application with Python’s Data Analysis Tools

https://medium.com/@mondoa/enhancing-your-web-application-with-pythons-data-analysis-tools-2ecd0af29027

The Ultimate Python 3 Guide: Everything You Need to Know

https://medium.com/@mondoa/enhancing-a-comprehensive-python-3-tutorial-b8102f0cfcc4


r/datacleaning • • Nov 18 '15

How have you been obfuscating your data, or found it obfuscated?

6 Upvotes

In January, my friend Travis and I are doing a talk at Data Wranglers DC entitled ‘Black Hat Data Wrangling’ where we’ll be discussing different ways of obfuscating data, and how to make that data accessible.

We have a growing list of online and offline methods including:

  • No API

  • Javascript overlays

  • Rendering to images

  • Printing it all out, scanning it, faxing it and then scanning it again (yes, this has happened)

  • Snail mail a hard drive (people still do this too)

What methods of data obfuscation have you used or encountered? We’d love to add them to our talk.

Thanks for your help!


r/datacleaning • • Jan 31 '15

Great tool for cleaning .csv files: csvkit

Thumbnail csvkit.readthedocs.org
6 Upvotes

r/datacleaning • • Aug 19 '14

"An introduction to data cleaning with R"

Thumbnail cran.r-project.org
6 Upvotes

r/datacleaning • • May 28 '23

Textraction.ai released! Flexible entity extraction - no training needed

5 Upvotes

It can extract exact values (e.g. names, prices, dates), as well as provide ChatGPT-like semantic answers (e.g. text summary). Just describe the entities with a simple format:

  • description: a free text description of what you want to extract.

  • type: string / float / integer / string.

  • variable name: a descriptive variable name.

  • (optional) valid values: limit the output to a set of specific possible values.

Very impressive, it worked great on my data which consists of product descriptions and specs.

I like the interactive demo (https://www.textraction.ai/). The service is accessible also as an API for any commercial purpose via the RapidAPI platform: https://rapidapi.com/textractionai/api/ai-textraction


r/datacleaning • • May 30 '22

End-To-End Data Preparation with my new open source project: https://github.com/kuwala-io/kuwala

5 Upvotes

r/datacleaning • • Oct 11 '21

Data cleaning issues

5 Upvotes

To all the people working with data, Apart from the general issues like

- missing values, incorrect formats, trailing spaces, text case, etc

what are some issues you usually face while cleaning data in your organization


r/datacleaning • • Sep 18 '20

Data cleaning feedback

5 Upvotes

Hi All,

I have always been frustrated with data cleaning and the trivial errors I end up fixing each time. That's why, I am thinking of developing a library of functions that can come in handy when cleaning data for ML

Looking to understand what kind of data cleaning steps you repeat often in your work. I am looking into building functions for cleaning textual data, numerical data, date/time data, bash scripts that clean files.

Do any libraries already exist for this? I am used to writing functions from scratch for any specific cleaning I had to do eg correct spelling mistakes, filtering outliers, remove erroneous values.

Any help is appreciated. Thanks.


r/datacleaning • • Jun 16 '20

Can someone please help me differentiate between data wrangling and data cleaning?

5 Upvotes

Hi all! I’m currently researching data cleaning and trying to find good information on how it’s done, as there is not much literature/ guidelines from what I know. However, it seems people often say that data wrangling and data cleaning are the same thing, but I was warned against this and told not to bunch them together.

I know that they are different but it’s hard to find something that really lays out why. Can someone please explain the difference between them and outline why they are not the same?

Thanks so much!


r/datacleaning • • Mar 01 '19

Removing near-duplicates from an excel data set

5 Upvotes

I'm trying to clean up a set of data in excel that has names of places repeated incorrectly. For example, I frequently see WP Davidson listed three different ways:

  • WP Davidson (Mobile

  • WP Davidson (Mobile AL)

  • WP Davidson (Mobile, AL)

I currently have a data set of roughly 8700 unique places, but I think it should be closer to 4000-5000 after removing these duplicates. Is there an easy way to do this?