r/datacleaning • u/snazrul • Apr 04 '18
Data Science Interview Guide
r/datacleaning • u/deltawk • Jun 27 '15
r/datacleaning • u/datachili • Dec 16 '15
r/datacleaning • u/raus22 • Nov 26 '15
Video Demo: youtube
More info at: OpenRefine
r/datacleaning • u/Mykguy2 • Jul 14 '20
So last week I found a YouTube video where a guy went through a full set data cleaned and wrangled it and asked the questions he was trying to answer. Let you try to clean and wrangle the data and then did it. It was a great video for learning. I was wondering if there is any other videos that you know of where some take a large set up data and cleans and wrangle and lets you try and wrangle it/clean ahead of time.
Ps I have found many tutorials of little training videos I am looking for large data sets and full working through all the steps as you tackle a real world problem!
r/datacleaning • u/JaneLu0113 • Dec 13 '19
r/datacleaning • u/_Goldengames • Jan 15 '26
Hi everyone 👋 following up on my last post.
I’m continuing work on a desktop app for cleaning and organizing messy datasets locally (everything runs on your PC — no AI, no cloud).
Current capabilities include:
Detecting common data inconsistencies
Fast duplicate identification and removal
Column-level formatting standardization
Exporting cleaned data in multiple formats
I’ve added an Excel data preview and recorded a short clip showing the current flow. More improvements are in progress.
As before, I’d appreciate feedback from people who deal with real-world datasets — especially anything that would make this more practical in daily workflows.
Thanks.
r/datacleaning • u/elbogotazo • Jun 25 '19
I've been tasked with coming up with an automated way of processing a large number of scanned documents and extracting key data items from these docs.
The majority of these are scanned PDFs of varying quality and wildly varying layouts. The data elements im looking to extract are somewhat standardized. Some examples to illustrate : I need to extract client name and that might be recorded in the document as "Client : client X", "client name: client x", "CName: client X". Similarly, to extract invoice date I would look for "invoice date : mmddyyy", "treatment date : dd-MM-yy", "incall date - ddmmyyyy" etc..etc..
I've implemented a solution in R that :
Converts a scanned pdf to PNG
Uses Tesseract to run OCR
Uses Regex to extract key data items from the extracted text (6 to 15 items per document, depending on the document type)
Each document type will have a slightly different way the data needs to be extracted. I have created functions to extract individual items e.g. getClientName(), getInvoiceDate() and then combine these into a list, so that for each document I get the extracted items.
The above works, for most of the simple docs. I can't help feel that regex is a bit unwieldy and might not generalize to all cases - this is supposed to be a process that will used across my organization on a daily basis. My aim is to expose this extraction service as an API so that users in my organization can send pdf, images or text and my API returns key data in JSON.
This is a very specific use case, but I'm hoping there are others out there that have dealt with similar scenarios. Are there any tools or approaches that might work here? Any other things to be mindful of?
r/datacleaning • u/nonkeymn • May 09 '17
r/datacleaning • u/smortaz • Jan 12 '16
r/datacleaning • u/PerceptionFresh9631 • Oct 29 '25
Your go-to approach for cleaning or transforming data in-flight during syncs - do you run transformations inside your integration layer, or push everything into a staging database first?
r/datacleaning • u/That_Aardvark_2948 • Sep 02 '25
Working with data daily and curious about everyone's pain points. When you get a CSV with:
Duplicate rows scattered throughout
Phone numbers in 5 different formats
Names like "john SMITH", "Mary jones", "BOB Wilson"
Emails with extra spaces
How long does it usually take to clean? What's your current process?
Asking because I'm exploring solutions to this problem 🤔
r/datacleaning • u/Hairy_Border_7568 • Dec 23 '25
I noticed something uncomfortable about how I handle missing values.
I almost never look at them.
I just:
drop rows
fill with mean / mode
move on and hope nothing breaks
So I built a tiny UI experiment that forces me to see the damage before I “fix” anything.
What it does:
upload a CSV
shows missing data column by column
visually screams when a column looks risky
lets me try a fill and instantly see before vs after
No rules. No schemas. No “AI knows best”.
Just: “Here’s what your data actually looks like — are you sure?”
It made me realize something:
Curious how others do this honestly:
Do you inspect first?
Or do you auto-fix and trust yourself?
I put the UI + code here if you want to see what I mean:
https://github.com/Abhinay571/missing-data-detector/commit/728054ff5026acdb61c3577075ff1b6ed4546333
r/datacleaning • u/youre_so_enbious • Feb 29 '24
Hi,
Just wondering what requirements or checklist items people would suggest for a definition of Clean Data ready to be used in machine learning? Akin to "tidy data", but for modelling. I.e.
There should be no string fields. All data should be either in a numeric form, or as a categorical data type etc
I know this will likely be opinionated, hence wanting to "crowd source" it 😃
Feel free to disagree with any statements, as I imagine there will be differences
r/datacleaning • u/AdGreat4483 • Jul 01 '23
If you're embarking on the odyssey of studying Python data analysis, commence by acquiring a mastery of the rudiments of Python programming.
Once you've attained a level of proficiency with Python, plunge into the depths of indispensable libraries such as NumPy for numerical computation and Pandas for data manipulation. Engage in practical exercises utilizing authentic datasets to accrue experiential knowledge, and refine your prowess in data visualization employing Matplotlib and Seaborn.
Delve into the realm of statistical analysis using the comprehensive tools provided by SciPy, and contemplate augmenting your skill set with other pertinent libraries such as scikit-learn for machine learning. Engross yourself in online communities, undertake ambitious projects, and perpetually pursue learning and diligent practice to ascend to a zenith of expertise in Python data analysis—a gratifying pursuit that unveils the portals to unearthing invaluable insights from data.
To get you started, I will highly recommend you look at these articles.
Exploratory Data Analysis and visualization practical example:
https://link.medium.com/FYuBpTyvCAb
Data cleaning with python (a practical example)
https://link.medium.com/GBsdtEFvCAb
How to make data Visualization in python
https://link.medium.com/6rWH2nKvCAb
Python data cleaning made easy
https://link.medium.com/6rWH2nKvCAb
Sales Statistical analysis with python
https://link.medium.com/ZGx7NDRvCAb
https://link.medium.com/OidaOBUvCAb
Python Web App Development: Unleashing the Power of Simplicity and Flexibility
Enhancing Your Web Application with Python’s Data Analysis Tools
The Ultimate Python 3 Guide: Everything You Need to Know
https://medium.com/@mondoa/enhancing-a-comprehensive-python-3-tutorial-b8102f0cfcc4
r/datacleaning • u/robertdempsey • Nov 18 '15
In January, my friend Travis and I are doing a talk at Data Wranglers DC entitled ‘Black Hat Data Wrangling’ where we’ll be discussing different ways of obfuscating data, and how to make that data accessible.
We have a growing list of online and offline methods including:
No API
Javascript overlays
Rendering to images
Printing it all out, scanning it, faxing it and then scanning it again (yes, this has happened)
Snail mail a hard drive (people still do this too)
What methods of data obfuscation have you used or encountered? We’d love to add them to our talk.
Thanks for your help!
r/datacleaning • u/[deleted] • Jan 31 '15
r/datacleaning • u/[deleted] • Aug 19 '14
r/datacleaning • u/DoorDesigner7589 • May 28 '23
It can extract exact values (e.g. names, prices, dates), as well as provide ChatGPT-like semantic answers (e.g. text summary). Just describe the entities with a simple format:
description: a free text description of what you want to extract.
type: string / float / integer / string.
variable name: a descriptive variable name.
(optional) valid values: limit the output to a set of specific possible values.
Very impressive, it worked great on my data which consists of product descriptions and specs.
I like the interactive demo (https://www.textraction.ai/). The service is accessible also as an API for any commercial purpose via the RapidAPI platform: https://rapidapi.com/textractionai/api/ai-textraction
r/datacleaning • u/kuwala-io • May 30 '22
r/datacleaning • u/cchaituc • Oct 11 '21
To all the people working with data, Apart from the general issues like
- missing values, incorrect formats, trailing spaces, text case, etc
what are some issues you usually face while cleaning data in your organization
r/datacleaning • u/crossvalidator • Sep 18 '20
Hi All,
I have always been frustrated with data cleaning and the trivial errors I end up fixing each time. That's why, I am thinking of developing a library of functions that can come in handy when cleaning data for ML
Looking to understand what kind of data cleaning steps you repeat often in your work. I am looking into building functions for cleaning textual data, numerical data, date/time data, bash scripts that clean files.
Do any libraries already exist for this? I am used to writing functions from scratch for any specific cleaning I had to do eg correct spelling mistakes, filtering outliers, remove erroneous values.
Any help is appreciated. Thanks.
r/datacleaning • u/Mandypandie • Jun 16 '20
Hi all! I’m currently researching data cleaning and trying to find good information on how it’s done, as there is not much literature/ guidelines from what I know. However, it seems people often say that data wrangling and data cleaning are the same thing, but I was warned against this and told not to bunch them together.
I know that they are different but it’s hard to find something that really lays out why. Can someone please explain the difference between them and outline why they are not the same?
Thanks so much!
r/datacleaning • u/[deleted] • Mar 01 '19
I'm trying to clean up a set of data in excel that has names of places repeated incorrectly. For example, I frequently see WP Davidson listed three different ways:
WP Davidson (Mobile
WP Davidson (Mobile AL)
WP Davidson (Mobile, AL)
I currently have a data set of roughly 8700 unique places, but I think it should be closer to 4000-5000 after removing these duplicates. Is there an easy way to do this?