5 Types of Costly Data Waste and How to Avoid Them

Do you know someone who bought a lot of fancy exercise equipment but doesn’t use it? Turns out, exercise equipment doesn’t provide many benefits when it goes unused.

The same principle applies to getting value from data. Organizations may acquire a lot of data, but they aren’t getting much value from it. This is a widespread issue that cuts across different sectors. It is estimated that nearly 75% of the data that enterprises collect remains unused, and thus, the value is not realized. So, what is the problem?

In the fitness example, the problem is typically not the exercise equipment; it’s an issue with the user’s habits.  Similarly, getting value from data often is not a problem with the data itself. Rather, problems arise from limitations imposed by data infrastructure and data practices that block effective and efficient use. In other words, poor choices in data infrastructure and data habits can lead to data waste. 

What is data waste, and why does it happen?

Fundamentally, data waste means missing an opportunity to get value from data or paying too much to acquire, store, and use data. In large-scale systems, data waste comes in many forms. Some are surprising, most are expensive, and almost all are avoidable. 

To avoid unnecessary data waste in your organization, first you must recognize it. The following describes five common ways that waste occurs:

• Data is used and then thrown away

A common data habit that results in missed opportunity is assuming data has no further value once it’s been used for the particular purpose. Data is ingested, processed, transformed (perhaps for a specific report or to be stored in a traditional database), and then the raw or partially processed data is discarded. It isn’t practical to save all your data, but it is important to realize data may be valuable for other projects. You lose that add-on value when you throw data away. 

This type of data waste results in missing out on the second project advantage. For example, AI and machine learning projects offer great potential value, but they are speculative. Lowering the entry cost by re-using data and infrastructure already in place for other projects makes trying many different approaches feasible. That, in turn, makes it more likely to find the ones that pay off. Fortunately, learning-based projects typically use data collected for other purposes. 

It’s also important to go back to raw data to ask new questions and train new models, particularly as the world is constantly changing. Features that you didn’t think were valuable at first may later be just what you need. You’ve lost that opportunity if the data has been thrown away.  

• You have data but don’t use it

Why does valuable data so often go unused? One reason is people don’t know where it is or even possibly that it exists at all. Lack of annotation with the right metadata is a contributing factor. Another is poor communication between projects or business units.

An even larger issue is that people may not know how to see value in data. Recognizing what data can tell you is an acquired skill for people beyond just data scientists. New approaches are being developed to understand and use unstructured data, for instance. But to get the benefits data has to offer, you must learn to use it, just like you need to know how to use exercise equipment before it can do you any good. 

Another factor that keeps people from fully using and re-using data is data infrastructure requiring specialized tools. This limitation makes it inconvenient for data to be used by different types of applications or different analytics and AI tools. Increasingly, people look for ways to unify their data layer and have flexible access in order to build a data-first environment.

• You have data but not where it is needed

Data in the wrong place is about the same as data that does not exist. And “wrong place” can mean more than one thing. It may be that data is held by a different business unit, making it difficult to identify or challenging to get the permissions and access needed to share that data. Once again, there is a cost for not using data because it is somewhere other than you’d like it to be. 

Another way data is in the wrong place is in a more literal sense: geolocation. For large systems, major data motion from edge to data center or between data centers that are located in different cities or countries is challenging, especially if you do not have data infrastructure designed to do move data  automatically. Coding data motion into applications is not an adequate alternative except in the simplest of cases. To avoid data waste, you must have a way to efficiently move data to where it is needed. Otherwise, hand-coding of data motion can lead to additional problems, including unwanted duplication.

• Your system involves unwanted duplication

Having unnecessary duplication of large data sets is clearly a waste of the resources used to store and access data, but it involves waste in other ways as well. Duplication of data also entails duplication of effort, which is an additional cost. And the problem is not just a matter of too many copies of data. Approximately duplicated data sets may introduce uncertainty about data quality. Near duplicates immediately raise the question of which is authoritative and why there are differences, and that leads to mistrust about data quality. 

Hand-coded data motion by many different users creates its own problems, as this is hard to do accurately at scale. Resultant data sets can introduce unintentional variation in data even where a verbatim copy is intended. 

Another related problem is the creation of data silos in large systems. Unwillingness to share data often points to the lack of a uniform data layer with flexibility in data access. Siloed data not only results in avoidable costs, but it also limits the understanding and insights data scientists and analysts can draw from the data. Siloing and poor data discovery capabilities are wasteful through opportunity cost plus the cost of redundant storage and duplicated effort.

A special example of data waste through unnecessary duplication occurs when an enterprise buys data that could have been obtained for free. This waste happens because people may not know what data options are available.

• Disconnect between data producers and data consumers

One problem with connecting data producers and data consumers is that those who produce data or even those responsible for data ingestion often do not know how it will be used. That disconnect makes it harder for those who need data to know where to find it or to know what the data actually consists of when they do find it. Data producers are challenged with annotating data appropriately without knowing the ways it will be used. This disconnect between data producers and data consumers leads to a classic type of data waste in the sense of missed opportunity or unnecessary effort and expense required to track down data.  

Reducing data waste

How can you address the issues listed above in order to reduce data waste? You need to develop a comprehensive data strategy that includes a unifying data infrastructure engineered to support flexible data access, data sharing, and efficient data motion. HPE Ezmeral Data Fabric is a software-defined and hardware agnostic data technology used to store, manage, and move data at scale across an enterprise — from edge to data center, on premises, or in the cloud. As such, it serves as a unifying data layer that supports a wide range of applications and tools, thus inviting the re-use of data. In addition, data fabric handles data motion automatically at a platform level. 

Other solutions come in the form of better use of metadata to aid in data discovery and understanding, along with new data initiatives to better connect data producers with data consumers. One new initiative is the Agstack Foundation, an open-source digital infrastructure for agriculture. Another example is Dataspaces, a new service platform that helps data producers and data consumers integrate diverse data sets, enhance data discovery, and access and improve data governance and trust.

These solutions can help you reduce costly data waste and take better advantage of the value data offers. Making better use of your exercise equipment, however, is still up to you.

To find out more about data infrastructure that can help you reduce data waste, read this technical paper.

____________________________________

About Ellen Friedman

Ellen Friedman is a principal technologist at HPE focused on large-scale data analytics and machine learning. Ellen worked at MapR Technologies for seven years prior to her current role at HPE, where she was a committer for the Apache Drill and Apache Mahout open source projects. She is a co-author of multiple books published by O’Reilly Media, including AI & Analytics in Production, Machine Learning Logistics, and the Practical Machine Learning series.