What is ‘Dirty Data’? And how can we fix it?

What is ‘Dirty Data’? And how can we fix it?

It’s the stuff CRM Manager nightmares are made of.

RMIT Online logo

3 min read | Updated on 20 June 2023

Here's a statistic that should keep you up at night: only 16 per cent of business executives are confident in the accuracy of the data that informs their decisions. Imagine that. We’re living in a data-driven age – there are around 2.5 quintillion bytes of data created every single day – and yet most professionals can’t say for sure whether the numbers, on which they might be basing multi-million dollar decisions, actually mean anything.  

Welcome to the world of Dirty Data. 

What is ‘Dirty Data’?

Dirty Data is simply customer or business information that is corrupt, missing, duplicated or inaccurate. It happens every day in companies all over the world. Every time an Account Manager incorrectly duplicates a customer record, or someone misspells a crucial address, every time a CRM gets cluttered with spam emails, or a date format is followed inconsistently, Dirty Data is the result. They’re numbers without meaning. Information without purpose.  

You might think, what’s the big deal? Any errors are probably small, and they’re unlikely to impact overall business efficiency, right? But Dirty Data is estimated to cost the US economy over $3 trillion dollars every year.  

Experian reports that companies around the world, on average, believe that 26 per cent of their data is inaccurate or corrupt. And since data informs pretty much every facet of industry decision making – from why groceries aren’t selling to the order of your Netflix feed and how people move through Disneyland – 26 per cent data corruption pretty much represents 26 per cent missed opportunity. Opportunity for customer engagement, interaction or simple revenue.

What causes Dirty Data?

It probably comes as no surprise that human error accounts for more than 60 per cent of all Dirty Data. Combine a human brain (or even worse, multiple teams of human brains) with manual input fields, and problems are bound to occur. 

The rest is a combination of inaccurate records and poor data strategy, which eventually circle back to human error in any case: there’s no point having a fully automated CRM or overarching data plan if their integrity relies on manual entry.  

Often Dirty Data is the result of departmental miscommunication: different teams inputting related data into separate siloes, without any cooperation or internal data logic. It’s the classic left-hand-right-hand scenario. Unfortunately, because of this kind of internal bureaucracy, Dirty Data can go unchecked or unnoticed for years. Over 57 per cent of companies only discover Dirty Data when it’s reported by the customer, which everyone can agree is probably the worst way to discover a fundamental business problem. 

Cleaning up Dirty Data

‘Data Cleaning’ refers to the (often painstaking) process of going through data records and correcting mistakes. Pruning all the sloppy irregularities and misspellings and duplicate entries until you’re left with crisp, clean, useful data. Ideally this happens before the data is transferred to a target database or data warehouse.

Data cleansing can be a manual process, but that’s often incredibly labour intensive, hard to scale for enterprise, and (like data input) relies on human accuracy and consistency. So we’re right back where we started. These days there are dozens of automated apps and tools to help developers and data scientists speed things up.   

Of course, data cleansing comes with its own challenges. If you employ an automated script or cleaning program, you need to make sure it can correct data mismatches, order columns correctly, check format (like date or currency information), revise or update your data schema, and even enrich existing data with supplementary information. There’s a helpful guide if you’re looking for more hands-on information.

Skills for the future

Knowledge of data best practice, data integrity and even data cleansing are going to be incredibly valuable professional assets in years to come. As companies scale up their data collection – and according to recent Accenture studies, 83 per cent of enterprise executive have already harnessed Big Data to gain a competitive edge – the demand for skilled data scientists and analysts will continue to grow.  

That’s why, when RMIT designed its online data shortcourses with Udacity, we included units on data integrity and data cleansing. Whether you want to become a data scientist or data analyst, an understanding of Dirty Data won’t just be helpful – it’ll be essential.  

This article was originally published on 20 February 2020

20 June 2023

Share

Related courses and degrees

No courses match your current filters

Try adjusting them or reset all filters to see more options

Business Analyst

Future skills short course
  • next start date 12 October 2026
  • fees AU$1,900 incl. GST
  • learning mode Online
  • duration 6 weeks

Business Analytics

Future skills bundle
  • next start date 12 October 2026
  • fees AU$3,040 incl. GST
  • learning mode Online
  • duration 12 weeks

Business Analytics and Visualisation

Future skills short course
  • next start date 12 October 2026
  • fees AU$1,900 incl. GST
  • learning mode Online
  • duration 6 weeks

Business Analytics with SQL and Python

Future skills short course
  • next start date 12 October 2026
  • fees AU$1,900 incl. GST
  • learning mode Online
  • duration 6 weeks

Data Storytelling

Future skills short course
  • next start date 2 November 2026
  • fees AU$1,900 incl. GST
  • learning mode Online
  • duration 6 weeks

Online Graduate Certificate in Business Analytics

Online postgraduate study
  • fees AU$17,760 (2026 total)
  • learning mode Online
  • duration Part-time 8 months

Online Graduate Certificate in Data Science

Online postgraduate study
  • fees AU$17,760 (2026 total)
  • learning mode Online
  • duration 8 months intensive part-time

Online Master of Business Analytics and AI Strategy

Online postgraduate study
  • fees AU$35,520 (2026 annual)
  • learning mode Online
  • duration 12 courses (2 years part-time)*, 16 courses (2.7 years part-time)

Related articles

rmito-blog-data1-refimage-1220x732.jpg

Data-driven decision-making forms the core of modern business analytics
Icon / Small / CalendarCreated with Sketch. 09 Jul 2026

The great American composer and economist W. Edwards Deming had a good quote: “In God we trust. All others must bring data.”

Two men looking at a laptop screen.

Why choose a career in DevOps in 2022?
Icon / Small / CalendarCreated with Sketch. 27 Sep 2023

DevOps is not a role, person or title. It’s a way of thinking and building that can be applied across disciplines

Woman's arm hovering over a display screen showing various data.

Data storytelling. What is it, and why does everyone want it?
Icon / Small / CalendarCreated with Sketch. 27 Sep 2023

Forbes has named Data Storytelling as one of the most important tech skills for 2023

Birds eye view of a lit up city area at night time.

What is ‘Dirty Data’? And how can we fix it?
Icon / Small / CalendarCreated with Sketch. 20 Jun 2023

It’s the stuff CRM Manager nightmares are made of.

View all articles

Connect with us

Be part of the conversation and keep in touch with us to find out about study options, campus life and upcoming events on our social media channels.

aboriginal flag float-starttorres strait flag float-start

Acknowledgement of Country

RMIT University acknowledges the people of the Woi wurrung and Boon wurrung language groups of the eastern Kulin Nation on whose unceded lands we conduct the business of the University. RMIT University respectfully acknowledges their Ancestors and Elders, past and present. RMIT also acknowledges the Traditional Custodians and their Ancestors of the lands and waters across Australia where we conduct our business - Artwork 'Sentient' by Hollie Johnson, Gunaikurnai and Monero Ngarigo.

Learn more about our commitment to Aboriginal and Torres Strait Islander peoples
aboriginal flag float-starttorres strait flag float-start

Acknowledgement of Country

RMIT University acknowledges the people of the Woi wurrung and Boon wurrung language groups of the eastern Kulin Nation on whose unceded lands we conduct the business of the University. RMIT University respectfully acknowledges their Ancestors and Elders, past and present. RMIT also acknowledges the Traditional Custodians and their Ancestors of the lands and waters across Australia where we conduct our business.

More information