6 barriers to becoming a data-driven company

One of Randy Sykes’ previous employers spent eight years building a data warehouse without success.

“That’s because we tried to apply standard system development techniques without making sure that the business was with you in lockstep,” he says.

Today, Sykes is IT director of data services at Hastings Mutual Insurance Co. This time, he took a different approach to consolidating the organization’s data.

Ten years ago, the company decided to bring everything together into a data warehouse. At the time, reports took 45 days to produce and business users didn’t have the information they needed to make business decisions.

First, data would be collected in a landing area via nightly batch imports from legacy systems. It would then move into a staging area, where business rules would be applied to consolidate and reconcile data from different systems. This required a deep understanding of how the company operates and what the data means. But this time, the project was successful because there were subject matter experts on the team.

“We had a couple of business folks who’d been with the company a long time and had a lot of knowledge of the organization,” he says. “You actually have a cross-functional team to be successful.”

For example, different insurance policy systems might have different terms, and different coverage areas and risks. In order to consolidate all this information, the data team needs to have a good understanding of the business language and the rules needed to transform the raw data into a universal format.

“That’s the biggest challenge that companies run into,” he says. “They try to get the data and technically put it together and forget the business story behind the information. A lot of times, these types of projects fail.”

Today, a report that used to take 45 days can be turned around in 24 hours, he says. Then, as databases continue to get modernized and become event-driven, the information will become available in real time.

No short-term business benefits

Once Hastings started getting data together, the data project began producing value for the company, within a year, even though the data warehouse project, which began in 2014, wasn’t delivered until 2017.

That’s because the landing and staging areas were already providing value in terms of gathering and processing the data.

Data projects have to deliver business value all throughout the process, Sykes says. “Nobody is going to wait forever.”

A similar “quick win” helped lead to the success of a major data project for Denise Allec, principal consultant at NTT Americas, back when she was the director of corporate IT at a major corporation.

A six-week proof-of-concept project showed that the project had value, she says, and helped overcome challenges such as business units’ unwillingness to give up their silos of data.

“Giving up ownership of data represents a loss of control to many,” she says. “Information is power.”

This kind of data hoarding isn’t limited to senior executives, though.

“Employees tend not to trust others’ data,” she says.

They want to validate and scrub their own sources, and massage and create their own reporting tools that work for their unique needs.

“We’ve all seen the numerous duplicative databases that exist throughout a company and the challenges that arise from such a situation,” she says.

Choosing data projects that don’t have immediate benefits is a major roadblock to successful data initiatives, confirms Sanjay Srivastava, chief digital strategist at Genpact.

 “Until you do this, it’s all a theoretical discussion.”

The flip side is choosing projects that don’t have any ability to scale—another major barrier.

Without the ability to scale, a data project won’t have meaningful long-term impact, instead using up resources for a small or idiosyncratic use case.

“The key is how you deliver business value in chunks, in a time frame that keeps people’s attention, and that is scalable,” he says.

Putting the business users first means giving people the data they need in the form they need it. Sometimes, that means Excel spreadsheets. At Hastings, for example, staff would historically copy-and-paste data into Excel in order to work with it.

“Everybody uses Excel,” says Hastings’ Sykes. “Now we say, ‘Why don’t we just give you the data so you don’t have to copy-and-paste it anymore.’”

But the company has also been creating dashboards. Today, about a quarter of the company’s 420 employees are using the dashboards as well as outside agencies.

“They can now help agents cross-sell our products,” he says. “We didn’t have that before.”

But providing people with the serf-serve analytics tools they need is a challenge. “We’re still behind the eight ball a little bit,” he says. But with 200 business-focused dashboards already in place, the process is well under way.

Another organization that recently began the process of democratizing access to data is the Dayton Children’s Hospital in Dayton, Ohio.

“We weren’t doing that well five years ago,” says CIO J.D. Whitlock. “There were still a lot of spreadsheets. Now we’re using the Microsoft data stack, like a lot of people are doing. So as long as someone knows a little bit about how to use PowerBI, we’re serving up the appropriate data, in the appropriate format, with appropriate security.”

In addition, data analysts have also been decentralized, so people don’t have to go to a single team with their data questions.

“Say you want to know how many of procedures X doctor Y did last year,” says Whitlock. “It’s a relatively simple query. But if you don’t give people the tools to do that themselves, then you’ve got a thousand requests.”

Putting self-serve data tools in place has helped the company move toward being a data-driven organization, he says. “With the caveat that it’s always a journey and you never declare victory.”

Not including end users in your development process

Ignoring user needs is nearly always a recipe for disaster. For example, Nick Kramer recently worked with a national restaurant services company, with offices in 46 US states and up to $500 million in annual revenues. Kramer is the leader of applied solutions at SSA & Company, a global consulting firm. The restaurant services company was growing rapidly but service levels were dropping.

“Everybody was pointing fingers at each other,” he says. “But the CIO had no dashboards or reports—just anecdotes and opinions.”

One of the problems was that the central installation system was widely ignored. Employees updated records, but after the fact. The system had been imposed on them and was hard to use.

“People in the order department, in sales, legal, and on the installation side—every office had their own spreadsheets they ran their schedules on,” Kramer says. “None of the communication was happening and the data wasn’t flowing. So you had to go office by office to find out who was doing what and how well, and which delays were unsolvable and which ones could be addressed.”

The solution was to get close to the business users, to understand how the data was used.

Joshua Swartz, partner at Kearney, had a similar experience recently when he was working on a consulting project with a US food company with several billion in annual revenues.

The company wanted to enable production managers to make better decisions about what to produce based on real data.

“For example, there’s a production line in a certain production site and it can make either tortilla chips or pita bread,” says Swartz. “If there’s a switchover, you have to stop and clean and change the ingredients.”

But, say, the old way was to do four hours on tortillas and four hours on pita bread, and the data showed that you should do two hours on tortilla chips—and then tomorrow it may be the opposite. And since food products are perishable, getting production wrong means that some product would have to be thrown away. But when the company first designed its solution, the production workers weren’t involved, says Swartz. “They were too busy producing food and didn’t have time to stop and attend meetings.”

This wasn’t expected to be a problem because the company’s culture was hierarchical. “When the CEO says something and pounds their fist on the table, everyone has to follow suit,” he says.

But the new system was used for only a couple of weeks in the pilot site and then the employees found that the system didn’t really work for them and went back to doing things the old way. Also, it didn’t help that the company’s data czar was located a couple of layers down in the company’s technology organization, rather than closer to top management or to the business units.

Fixing the problem required bringing the actual employees to the design suite, even though it required adding capacity to the production lines to free up workers.

“Food companies with very thin margins weren’t comfortable making that investment,” Swartz says. But when they became part of the process, they were able to contribute to the solution, and today a third to a half of the facilities are using the new technology.

Swartz also recommends that the chief data officer be located closer to the company’s most valuable data.

“If data is a strategic asset of the business, I would place the CDO closer to the part of the business that has ownership of the data,” he says. “If the organization is focused on using data for operational efficiency, then under the COO might be the right place.”

A sales-driven company might want to put the CDO under the sales officer, however, and a product company, under the marketing officer, he says. One consumer packaged goods company he worked with actually had the CDO report directly to the CEO.

“If you think of data as a technology problem, you’re going to keep running into challenges of how much value you are actually getting from data and analytics,” says Swartz.

The responsible use of data is important for the success of data initiatives, and nowhere more so than in finance.

“Trust is of utmost importance in the banking sector,” says Sameer Gupta, chief analytics officer at DBS Bank. “It’s crucial to use data and models responsibly, and ethical considerations must be upheld while using data.”

Data use should be purposeful, he says, respectful, and explainable, and should never come as a surprise. “Data use should be expected by individuals and corporates,” he says.

By focusing on trust, he adds, the bank has been able to deploy AI and data use cases across the enterprise—260 at the last count—ranging from customer-facing businesses like consumer and small and medium enterprise banking, to support functions like compliance, marketing, and HR.

“In 2022, the revenue uplift from our AI and machine learning initiatives was about SGD 150 million [US $112 million], more than double that from the previous year,” he says. “We aspire to achieve SGD 1 billion in the next five years.”

Earning trust takes time and commitment. Becoming a data-driven company is all but impossible without it. But once trust is gained, it begins a virtuous cycle. According to a CapGemini change management study released in January, in organizations with strong data analytics, employees are 18% more likely to trust the company. And when those companies need to evolve further, the probability of successful change is 23 to 27% higher than at other organizations.

“Many people, including data experts, think most issues while transitioning toward becoming a data-driven company are technology-related,” says Eugenio Zuccarelli, a data scientist at a global retailer and former AI research scientist at MIT.

But the real barriers are personal, he says, as people have to learn to understand the value of making data-based decisions.

“While doing research at MIT, I often saw experts and leaders of organizations struggle with their transition toward becoming a more data-driven organization,” he says. “The main issues were usually cultural, such as a belief that technology would have overtaken their decision-making, rather than empowering them, and a general tendency to take decisions based on experience and gut feelings.”

People need to understand that their expertise is still vital, he adds, and that the data is there to provide additional input. 

Companies need to stop thinking about becoming a data-driven company as a technology problem.

“All our clients are talking about becoming more data driven, and none of them know what it means,” says Donncha Carroll, partner in the revenue growth practice and head of the data science team at Lotis Blue Consulting. They focus on their technology capabilities, he says, not what people will be able to do with the data they get.

“They don’t put the user of the solution in the frame,” he says. “Lots of data analytics teams provide data dashboards that provide information that is neither useful nor actionable. And it dies on the vine.”