6 modern data stack trends to look for in 2021

Data has become the name of the game for almost every enterprise as companies and organizations look for more ways to stay ahead of the curve and identify past mistakes. 

TechRepublic spoke with industry leaders and experts who discussed six topics and ideas that will emerge in 2021 related to the modern data stack.

Debanjan Saha, vice president of data analytics at Google Cloud, recalled, “I was a network engineer when the big internet revolution happened and I feel really lucky to be in the midst of this cloud and data revolution.”

Saha added, “I think the next year and next five years are going to see digital transformation driven by the data revolution.” He suggested the average life of a Fortune 500 or S&P 500 company is getting shorter because businesses are evolving and updating faster than ever. “The only way to adapt and change is to use data and look around the corner using predictive models and AI/ML. These factors differentiate the winners and losers in the new digital economy.”

SEE: Cloud data storage policy (TechRepublic Premium)

Forrester vice president and principal analyst Michele Goetz said data management is emerging from an analytics first strategy to an outcomes based strategy, meaning data is now dynamic, stream-oriented, and orchestrating through processes and machine learning models. 

“Forrester sees the data stack extending beyond the data fabric into data networks. Data management will be centralized and hyper-local to create in-moment intelligence and experiences,” Goetz said. “Solution architectures increasingly rely on messaging, gateways, APIs, and microservices across an ecosystem and collections of capabilities. Swarm intelligence and smart city use cases are examples of this. Network architectures become tomorrow’s data architecture.”

Democratization of the data stack

George Fraser, CEO of data integration company Fivetran, said the recent wave of simplification in data management was a very important trend that would continue into 2021. 

He explained that just five years ago companies interested in better managing large amounts of data like Netflix had to do a ton of heavy lifting, hiring legions of data engineers and investing millions in impressive open source technology.

These days, much of that can be accomplished through subscriptions to one of the high-end data warehouses.

“Data management is getting easier. Technology often goes the other way, it gets more complicated. But we’re seeing a winnowing and I think that’s significant and a good thing. The cost of the fundamental components has come way down but it’s also true that people are doing more with data,” he said. 

“A few analysts can accomplish what five years ago Netflix would have had to invest $10 million in, which is cool. It’s making it accessible to companies with less sophistication and companies that aren’t on the coasts or hiring the fanciest teams with the best LinkedIn profiles. More like mere mortals can do this stuff, which is a good thing.”

Companies move to the cloud and embrace multicloud

Donal Tobin, CEO at data integration company Xplenty, said more and more large companies are showing interest in the cloud, especially for things like analytics. 

“What we are seeing is multicloud is definitely of interest as well. Having the ability to spin up your platform within any of the big cloud players is becoming more and more of a requirement,” Tobin said. “Customers want that and they don’t like the idea of being tied into any one solution with that or one platform.” 

The industrywide shift to cloud was being made, EnterpriseDB CTO Marc Linster explained, because cloud allows for rapid flexing of capacity and cuts down on lengthy provisioning cycles as well as upfront license investments. The ability to flex capacity to align with needs is key to quick responsiveness as part of a digital transformation strategy. 

Saha, from Google Cloud, said startups and digital native companies were the first to adopt Cloud but as we move into 2021, more traditional enterprises are adopting cloud platforms.

“People hate managing their data centers because they are expensive,” he said. “They actually feel safer that clouds have a lot more investment into security, privacy, and data governance.

The coronavirus pandemic has unfortunately given enterprises in every industry no choice but to embrace digital systems and cloud platforms in order to meet demand. 

“It has essentially accelerated this digital transformation challenge, and I think going forward, whatever hesitation people had, we are well past that at this point,” Saha said.  

Colin Zima, chief analytics officer at the Google-backed data analytics company Looker, said many customers are asking that companies be able to work with lots of different types of data sets and cloud providers.

But beyond regional regulations, most companies were simply interested in keeping their data in multiple places.

“You can’t just put everything in one place because you need backups and you need redundancy. So even at Looker, before we were acquired by Google, we had backups at Amazon. We have backups at Google, other backups at Microsoft and that kind of interconnectivity is becoming normal,” Zima said. 

Andy Maguire, senior machine learning engineer for Netdata, said that what was really needed is a federated data lake that can span multiple clouds. 

“For example, if you had your clickstream data in something like Google BigQuery, and your core production apps and logs in Amazon Web Services (AWS), then it could be too expensive in terms of network ingress and egress to centralize all the data in either Google Cloud Platform (GCP) or AWS or on some vendor’s platform. Instead, we need to learn to live with the complexity of multicloud and how best to navigate it,” he said. 

Exploring predictive value of data

Enterprises without a background in data are increasingly realizing how useful it can be for predictive actions. Joe Maguire, senior research director at Gartner, said that increasingly in 2020 and surely in 2021, enterprises without internal AI/ML skillsets will benefit from AI/ML features embedded in vendor products. 

“Aligning the data, data science, and ML pipelines alongside the application deployment process is fundamental to the continuous delivery and continuous integration of periodically enhanced ML models within AI-based solutions. This requires leveraging DataOps, MLOps and Platform Ops for AI to scale the AI architecture. Hence, AI orchestration platforms to operationalize AI are emerging,” he said.

Google Cloud’s Saha said that it is not just about looking at your data for the last quarter or last week and trying to figure out what happened in the past. 

“It is about looking at the event stream that is coming in and taking action in real time” he said. “Having real-time analytics is going to be really important. People are very interested in looking around the corner and predicting what is going to happen. If you can get value from data faster than others and you create real differentiating value. That’s why people are so interested in predictive analytics and predictive models.”

Increased use of artificial intelligence and machine learning

Ali Siddiqui, chief product officer at BMC Software, noted that a key element of the future of the modern data stack will be the inclusion of AI/ML-driven intelligent and predictive analytics capabilities leveraging a broad range of both historical and real-time data.  

In the IT Operations Management space, this involves analyzing data including metrics, events, logs, topology, incidents and changes, and requires platforms that are open and can integrate data from a myriad of tools and technologies, he said, adding that the stack will also need to support hybrid customers with data from on-premise data center infrastructure and applications as well as multiple cloud assets.  

“As businesses evolve into autonomous digital enterprises, it becomes about more than simply additional insight from the data analysis, but increasingly about actionability–and being able to take automated actions where possible,” Siddiqui said.

“With a year of unpredictability behind us, enterprises will have to expect the unexpected when it comes to making technology stacks infallible and proactive. We’ll see demand for AIOps continue to grow, as it can address and anticipate these unexpected scenarios using AI, ML, and predictive analytics.”

The infusion of AI and machine learning has become a part of everything, Saha said of Google Cloud. The tools are used widely for managing infrastructure by doing auto scaling, auto healing, auto optimization and more. 

“The way people, ordinary business users, are using AI/ML to do extraordinary things, is going to change the way businesses operate in future,” he said, adding that Google is looking for ways to further democratize AI and machine learning so that those without a data background can have access to it through simple spreadsheets. 

“Next year there will also be more augmented analytics, where you are going to see more and more AI and machine learning being integrated into people’s natural business workflows. The modern BI is about creating a data API on top of your data assets and then integrating your dashboard and your work flows into your business applications.”

Zima said the interest in AI and machine learning was already showing promise and would be even more widespread in 2021. 

“We’re starting to see people actually build these data products for internal usage. We work with a couple streaming services and one of them built a data product and I sort of use that generally because it’s effectively a dashboard underneath the sheets, but you actually browse through the titles, like you would in a streaming service. So it’s got icons of all the logos of the products and it’s touch-enabled, but you click on it and you drill in to get metrics about it,” Zima said.

“They’re delivering a product-like experience and I see it happening more and more. It’s more expensive to build those sorts of things. But I think increasingly you’re going to see data products that are built for internal usage.”

Concerns about locked-in data

Some experts expressed concerns that in 2021, some of the biggest names in data will begin locking their data out of rival platforms. 

“The big fear that I always have is that people are using more and more of these SaaS tools. The average company has a ton of SaaS tools and all of them have data locked in. So Salesforce has some of your sales data but Slack has chat data and you’ve got all these systems that are holding data sets,” Zima said.

“My biggest fear is that these services will start trying to lock up their data more. Salesforce just bought Slack and they’ve got Tableau. And I’m always scared that that data is not going to be available for other products or services.”

Robson Grieve, CMO at OutSystems, echoed those concerns and said because SaaS apps are one-size-fits all and don’t allow for any kind of differentiation or customization, we may see a wave of forward-looking companies—ones that are adopting more modern application platform approaches—leave this outdated model behind to build their own apps to become truly differentiated and save a ton of money in the process. 

But while the fear is real, other experts said customer concerns about being trapped with certain vendors will force them to keep their data somewhat open. 

“Analytics tools that try to become one-stop shops for all underlying data and cloud data warehousing needs will also meet challenges, as customers will be wary of vendor lock-in,” said Lewis Carr, senior director of product marketing at Actian. “For a modern data stack to work, it needs to be open to all origination sources, analysis and visualization destinations.”

Security needs of the data stack

Almost every industry is having a cybersecurity reckoning and data management is no different. Saha, from Google Cloud, said more and more customers are asking for a unified, end-to-end data governance structure across various different parts of the data late.

Donal Tobin, CEO at data integration company Xplenty, said companies are “essentially creating a ticking time bomb in your data warehouse.” He said customers are seeking out more information on data security, so much so that companies like Xplenty are now highlighting their encryption capabilities as a way to assure customers that their data will be safe. 

“I think that’s one of the biggest changes you see coming is that people are just now, like over the last three three to six months, starting to realize this and take it seriously,” he said. 

Data, Analytics and AI Newsletter

Learn the latest news and best practices about data science, big data analytics, and artificial intelligence.
Delivered Mondays

Sign up today

Also see