Data hygiene is the practice of ensuring that all structured or unstructured data inside of databases or file shares is “clean,” meaning it’s accurate, up to date, and error-free. Data hygiene is also referred to as “data cleanliness” and “data quality.”
In general, poor data quality comes from:
- Data duplication (also known as data redundancy): When records inside databases are repeated.
- Data incompleteness: When not all of the required data for a record is there.
- Data inconsistency: When the same data exists in different formats in multiple tables, leading to different files containing different information about the same object or person.
- Data inaccuracy: When data values stored for a certain object are incorrect.
Why Is Data Hygiene Important?
Data hygiene drives security, productivity, regulatory and compliance adherence, and efficiency. It does this by ensuring your applications and business processes are only using data that’s clean, correct, and relevant—and that includes removing sensitive personal data that’s no longer needed. Without good data practices, you’ll be following clues and bread crumbs to dead ends and bad decisions.
Here are some examples of issues that poor-quality data can create in organizations.
Sales and Marketing
A study by DiscoverOrg found that sales and marketing departments lose approximately 550 hours and as much as $32,000 per sales rep from using bad data.
In marketing, bad data can lead to overspending. It can also annoy or even drive away prospects if they receive the same content more than once due to data duplication (i.e., duplicate records with the same name spelled a little differently within the same database).
In online sales, poor data hygiene could lead you to try to sell the wrong product to the wrong client if you’re lacking data about your products and target audiences.
Finance
In financial reporting, bad data can give you different answers to the same question due to data inconsistency, leading to inaccurate and misleading financial reports. These reports could potentially give you either a false sense of financial security or an alarming sense of financial insecurity.
Supply Chain
Bad data can also wreak havoc on supply chains because it makes it very hard to automate processes if those process decisions are based on unreliable location information.
Overall Corporate Goals
On the corporate level, data quality issues can significantly impact your ability to meet your long-term goals. They can cause:
- A negative impact on your ability to pivot and react quickly to new market trends and conditions.
- Higher difficulty meeting compliance requirements of major privacy and data protection regulations such as GDPR, HIPAA, and CCPA.
- Difficulties in exploiting predictive analytics on corporate data, resulting in higher-risk decisions for both short- and long-term objectives.
The Challenges of Maintaining Good Data Hygiene
As important as good data hygiene is, many companies struggle to maintain the quality of their data. According to one study published by the Harvard Business Review, on average, 47% of newly created data records have at least one critical (e.g., work-impacting) error and only 3% of data quality scores were rated “acceptable” using the loosest-possible standard.
Various factors can make it challenging to optimize your data hygiene. These include:
- Increasing variety of data sources: Companies used to use only data generated from their own business systems, such as sales or inventory data. Now, data sources vary widely and can include data sets from the internet, IoT devices, scientific and experimental data, and more. The more data sources you have, the harder it is to ensure that data hasn’t been altered or tampered with in some way. Any time you add another system to your data processing engine, you add chances for that data to lose value by becoming tainted or lost because different data sources produce different data types. Unstructured data—or information that isn’t arranged according to a pre-set data model or schema—now accounts for an estimated 80% of all global data.
- Increasing volumes of data: The age of big data is unquestionably here and big data has only become bigger data. Since 1970, the amount of data has doubled every three years. The more data there is, the harder it is to collect, clean, integrate, and achieve a reasonably high quality of data within a certain time frame. If most of this data is unstructured, processing times will increase even more because this unstructured data needs to be turned into structured or semi-structured data, further deteriorating the quality of the data processing.
- Increasing velocity of data: “Real-time” data has become a big buzzword over the last five years. That’s because the more data generated, the faster you have to process it or you risk your systems getting backed up. In that sense, data is like a liquid flowing into a pipe—the faster it comes, the more danger there is of the pipe breaking, and the only way to deal with the increasing volume is to make the pipe bigger. For data, making the pipe bigger means processing it faster to meet the speed at which it’s coming in. But actual real-time processing is still a relatively new field and capability, which means there’s still a lot of “noise” in the form of unused or irrelevant data being used. As a result, decisions made based on that data will tend to be sub-optimal at best and erroneous at worst.
- Lack of clear data quality standards: Product quality standards have been around since 1987 when the International Organization for Standardization (ISO) published ISO 9000. In contrast, official data quality standards have only been around since 2011 (from ISO 8000), which means they’re still maturing and still relatively new. According to a 2015 study published in the Data Science Journal, “Currently, comprehensive analysis and research of quality standards and quality assessment methods for big data are lacking.”
Data Hygiene Best Practices
Although data quality standards are still maturing, there are certain established data hygiene best practices you can adopt right now to ensure your data quality is—and stays—high.
Best practices include:
Auditing
Data auditing is key to maintaining good data hygiene and typically the first step in any data cleansing process. Before taking any action, you need to assess the quality of your data and establish a realistic baseline of your company’s data hygiene. A typical data audit involves taking a close look at your IT infrastructure and processes to see where your data lives, how it’s used, and how often it’s updated.
Compliance
It’s critical to define policies regarding what data is collected and why, especially if the data comes from consumers. This includes solidifying data retention and removal policies. Retention schedules dictate how long data is stored on a system before being purged. Hygiene means knowing what data you’re storing, why, where, and when it needs to be purged. Learn more about data compliance best practices.
Governance
Data governance is the collection of processes, roles, policies, standards, and metrics that ensure the effective and efficient use of information in enabling an organization to achieve its goals. Data governance defines who can take what action, upon what data, in what situations, and using which methods. Good data governance is essential for ensuring high data quality across an organization.
Automation
Finally, good data hygiene comes from automating your data quality-related processes. This primarily means automatically updating your data as frequently as possible to ensure it’s always up to date and correct. Data cleansing systems can sift through masses of data and use algorithms to detect anomalies and identify outliers resulting from human error. They can also scrub your databases for duplicate records.
What Makes for High-Quality Data?
There are several attributes that comprise data quality. High-quality data is:
- Timely: It’s created, maintained, and available immediately and as required.
- Concise: It contains no extraneous information.
- Consistent: There are no conflicts in information within or between systems.
- Accurate: It’s correct, precise, and up to date.
- Complete: All possible data that is required is present.
- Conformant: It’s stored in an appropriate and standardized format.
- Valid: It’s authentic and from known, authoritative sources.
If your data meets all of these criteria, you, your systems, and your applications will be working with the best possible information to drive better customer service, better customer experience, and better business outcomes.
Get Best-in-Class Data Reduction and Deduplication with Pure Storage®
Data deduplication, also known as dedupe, is the process of eliminating duplicate copies of data within a storage volume or across the entire storage system (cross-volume dedupe). It uses pattern recognition to identify redundant data and replace them with references to a single saved copy. With Purity Reduce, Pure Storage uses five different data-reduction technologies to save space in all-flash arrays. Learn more here.
