r/datacleaning • • Apr 11 '17

Anyone here interested in IoT data cleaning?

Thumbnail
brightwolf.com
0 Upvotes

r/datacleaning • • Jun 12 '23

Need help on cleaning this data!!

Post image
0 Upvotes

As in the picture, there are multiple records with same headers, i want to create data which has column headers and it's values below them. I am unable to find a way out. Please help!!!


r/datacleaning • • Oct 27 '24

Need a mentor

0 Upvotes

Hi guys! Urgent need a mentor who can give me tasks from Data cleaning to visualization. I never studied data analytics formely, just studied from YouTube. Need help, I am counting on this reddit community.


r/datacleaning • • Jan 18 '18

Iterating over Pandas dataframe using zip and df.apply()

0 Upvotes

I'm trying to iterate over a df to calculate values for a new column, but it's taking too long. Here is the code (it's been simplified for brevity):

def calculate(row):
    values = []
    weights = []
    continued = False

    df_a = df[((df.winner_id == row['winner_id']) | (df.loser_id == row['winner_id']))].loc[row['index'] + 1:]
    if len(df_a) < 30:
        df.drop(row['index'], inplace = True)    
        continued = True
    #if we dropped the row, we don't want to calculate it's value
    if continued == False:
        for match in zip(df_a['winner_id'],df_a['tourney_date'],df_a['winner_rank'],df_a['loser_rank'],
                         df_a['winner_serve_pts_pct']):
                weight = time_discount(yrs_between(match[1],row['tourney_date']))
                #calculate individual values and weights
                values.append(match[4] * weight * opp_weight(match[3]))
                weights.append(weight)
    #return calculated value
    return sum(values)/sum(weights)


df['new'] = df.apply(calculate, axis = 1)

My dataframe is not too large (60,000 by 35), but it's taking about 40 minutes for my code to run (and I need to do this for 10 different variables). I originally used iterrows(), but people suggested that I use zip() and apply - but it's still taking very long. Any help will be greatly appreciated. Thank you


r/datacleaning • • Sep 09 '18

Join r/MachinesLearn!

2 Upvotes

With the permission from moderators, let me invite you to join the new AI subreddit: r/MachinesLearn.

The community is oriented on practitioners in the AI field, so tutorials, reviews, and news on practically useful machine learning algorithms, tools, frameworks, libraries and datasets are welcome.

Join us!

(Thanks to mods for allowing this post.)


r/datacleaning • • Jun 12 '23

Need help on cleaning this data!!

Post image
0 Upvotes

As in the picture, there are multiple records with same headers, i want to create data which has column headers and it's values below them. I am unable to find a way out. Please help!!!


r/datacleaning • • Jun 12 '17

How can we erase our privacy with protection?

Thumbnail
youtube.com
0 Upvotes

r/datacleaning • • Feb 10 '17

How to Clean Your Data Quickly in 5 Steps

Thumbnail
datasciencecentral.com
2 Upvotes

r/datacleaning • • Dec 05 '17

7 Rules for Spreadsheets and Data Preparation for Analysis and Machine Learning

Thumbnail
jabustyerman.com
2 Upvotes

r/datacleaning • • Dec 24 '24

Expert Data Cleaning Services | Boost Your Data Quality and Accuracy!

0 Upvotes

Is your data messy and incomplete? Let me help you clean it up and transform it into reliable, accurate insights! As a certified Data Analytics expert, I specialize in data cleaning using advanced tools like Python, Excel, and Power BI.

I can help you:

  • Remove duplicates and errors

  • Fill missing values

  • Standardize data formats

  • Clean and organize large datasets for analysis

With my Data Cleaning services, you’ll get high-quality data ready for analysis, helping you make smarter business decisions. Get in touch now for a free consultation or quote!

Contact - [email protected]

#DataCleaning #DataAnalytics #Excel #PowerBI #Python #DataTransformation #CleanData #DataInsights #BigData #BusinessIntelligence #DataScience #DataAnalysis #Freelancer #AI #DataExperts #MachineLearning,CleanUpData


r/datacleaning • • Sep 02 '20

Data Cleaning In R Programming Language

Thumbnail
youtu.be
3 Upvotes

r/datacleaning • • Aug 09 '25

Quick thoughts on this data cleaning application?

0 Upvotes

Hey everyone! I'm working on a project to combine an AI chatbot with comprehensive automated data cleaning. I was curious to get some feedback on this approach?

  • What are your thoughts on the design?

  • Do you think that there should be more emphasis on chatbot capabilities?

  • Other tools that do this way better (besides humans lol)


r/datacleaning • • Aug 09 '25

Quick thoughts on this data cleaning application?

0 Upvotes

Hey everyone! I'm working on a project to combine an AI chatbot with comprehensive automated data cleaning. I was curious to get some feedback on this approach?

  • What are your thoughts on the design?

  • Do you think that there should be more emphasis on chatbot capabilities?

  • Other tools that do this way better (besides humans lol)


r/datacleaning • • Apr 06 '24

What does it imply when the total cost is negative, the unit selling price is positive and the order is 0? I am trying to clean data in Excel.

0 Upvotes

ORDER QUANTITY | UNIT SELLING PRICE| TOTAL COST

0 | 151.47 | -86.9076

0 | 690.89 | -1002.1401

0 | 822.75 | -978.8337

I am trying to clean a dataset and wanted to understand if it makes sense or if I should delete it from the table. There are about 28% of total entries with such data. It won't make sense to delete 28% either. Please drop your suggestions and understanding.


r/datacleaning • • Nov 18 '24

ow can i implement a new lemmatizing function from scratch

0 Upvotes

hello good people i am a student at computer science engineering and i have homework at data retrieval field

using Python and i am not that much with this kind of programming language

but the main thing i want to say is how I should implement a steeming function from scratch without using nltk library because my doctor wants us to build it in the homework could anyone tell me where should i start and what I should do i searched everywhere in the google and with no benefits everything talks about the function in the nltk library

what should i do?

thanks for any help

sorry for my bad English


r/datacleaning • • Jun 01 '17

Urjanet Data Guru Series Part 2: A Guide to Data Mapping and Tagging

Thumbnail
urjanet.com
0 Upvotes

r/datacleaning • • Dec 12 '21

Cleaning my 'Dates' Data on my excel dataset.

0 Upvotes

Hey Guys, I have a dataset with about 2,101 different dates. They're in a table with other things like price and locations but, a lot of the dates in the data set do not follow the date format I am using (MM/DD/YYYY), some use DD/MM/YYYY or something else. How would I tackle this?


r/datacleaning • • Jan 03 '23

What is the American number format?

0 Upvotes

Hello, i’m trying to dataclean some phone numbers, whereas i do understand the EU format, I have no clue about the US format

001-377-014-0631x83215

469-229-6851x300

001-117-566-5683

Here are couple examples of the data i have, I know the country code is +1 but what is the xNNN that follows some of these numbers, it could be the way they wrote it but there's a lot of similar ones so i dont think its human error


r/datacleaning • • Mar 07 '20

Data Cleaning for missing values

0 Upvotes

Hi, I have a dataset with time variable year, month, day, form individual column, and I have some green houses gases column follow by these columns. There are some missing values for each of the green houses column. What is the best way to fill these missing values without affect the accuracy of the whole dataset? Please comment below. Thank you


r/datacleaning • • Jul 26 '22

MLOps Community (recorded) session on new open source data prep tool

0 Upvotes

Quickly move your notebooks from research to production with no extra work!
https://www.youtube.com/watch?v=6Iyt9Wip3C4

Link to tool: https://github.com/mage-ai/mage-ai


r/datacleaning • • Oct 18 '17

What if I don't clean my data 100% properly?

0 Upvotes

Seriously... no matter how hard we clean... some bad examples are going to get through!!

How can I take that into account when looking at my results?

Is it better to have HUGE sets with some errors or small sets with none?


r/datacleaning • • Jun 04 '25

Do you also waste hours cleaning Excel files and building dashboards manually?

Post image
0 Upvotes

I’ve been working on a side project and I’d love feedback from people who work with data regularly.

Every time I get a client file (Excel or CSV), I end up spending hours on the same stuff: removing duplicates, fixing phone numbers, standardizing columns, applying simple filters… then trying to extract KPIs or build charts manually.

I’m testing an idea for a tool where you upload your file, describe what you want (in plain English), and it cleans the data or builds a dashboard for you automatically using GPT.

Examples:

– “Remove rows where email contains ‘test’”

– “Format phone numbers to international format”

– “Show a bar chart of revenue by region”

My questions:

– Would this save you time?

– Would you trust GPT with these kinds of tasks?

– What feature would be a must-have for you?

If this sounds familiar, I’d love to hear your take. I’m not selling anything – just genuinely trying to see if this is worth building further.


r/datacleaning • • Jun 27 '15

16 Free Data Science Books [x-post from /r/machinelearning]

Thumbnail
wzchen.com
11 Upvotes

r/datacleaning • • Dec 23 '25

I stopped fixing missing values. I started watching them.

6 Upvotes

I noticed something uncomfortable about how I handle missing values.

I almost never look at them.
I just:

  • drop rows

  • fill with mean / mode

  • move on and hope nothing breaks

So I built a tiny UI experiment that forces me to see the damage before I “fix” anything.

What it does:

  • upload a CSV

  • shows missing data column by column

  • visually screams when a column looks risky

  • lets me try a fill and instantly see before vs after

No rules. No schemas. No “AI knows best”.
Just: “Here’s what your data actually looks like — are you sure?”

It made me realize something:

Curious how others do this honestly:

  • Do you inspect first?

  • Or do you auto-fix and trust yourself?

I put the UI + code here if you want to see what I mean:

https://github.com/Abhinay571/missing-data-detector/commit/728054ff5026acdb61c3577075ff1b6ed4546333


r/datacleaning • • Nov 07 '25

Are you struggling with slow, manual, and error-prone data cleaning processes?

0 Upvotes

Many teams still depend on manual scripts, spreadsheets, or legacy ETL tools to prepare their data. The problem is that as datasets grow larger and more complex, these traditional methods start to break down. Teams face endless hours of cleaning, inconsistent validation rules, and even security risks when data moves between tools or departments.

This slows down analysis, increases costs, and makes “data readiness” one of the biggest bottlenecks in analytics and machine learning pipelines.

So, what’s the solution?

AI-driven Cleaning Automation can take over repetitive cleaning tasks automatically detecting anomalies, validating data, and standardizing formats across multiple sources. When paired with automated workflows, these tools can improve accuracy, reduce human effort, and free up teams to focus on actual insights rather than endless cleanup.