A Window Into the Future of Data in Motion and What It Means for Businesses

Modern businesses have vast amounts of data at their fingertips and are acutely aware of how enterprise data strategies positively impact business outcomes. Despite this, only a handful of organisations interact with all stages of the data life cycle process to truly distill information that distinguishes future-ready businesses from the rest. Much potential remains untapped when businesses do not translate their data into actionable insights from the point it is created, eroding the usefulness of data over time. 

One of the ways to accelerate time to insight is by performing analytics on real-time data. This is the focus of “data in motion”.

What is data in motion?

Data in motion is one of three broad labels used to describe data as part of a unified data life cycle. It can be “at rest”, “in use”, or “in motion”. Data in motion consists of three distinct elements: data flow, message streams, and stream processing and analytics. 

Around 2016, we started talking about data in motion within the context of an enterprise data platform. In the past three years or so, data in motion has grown in popularity due to its widespread applications across various industries.  

What are the industry applications of data in motion?

Data in motion has been utilised in 4G applications, however, 5G opens up a world of new possibilities. To support the proliferation of 5G applications, telecommunications providers are using location analytics to make connections more reliable, such as by speeding up resolution of network issues.The scale of 5G deployments coupled with them being virtualised require real-time monitoring, insight, and predictive models to achieve high standards of service at scale. 

At the same time, 5G adoption accelerates the Internet of Things (IoT). Japan and South Korea are expected to see 150 million IoT connections by 2025, which will include the manufacturing and logistics sectors. The revenue enabled by IoT is expected to reach $460 billion by 2026, which equates to an increase of almost 30% CAGR within manufacturing. Predictive maintenance applications enable large-scale manufacturers to collect telemetry data and integrate all IoT functions, and these are powered by models driven by real-time data. 

The financial services industry has had to dedicate more resources to personalisation, fighting fraud, and reducing cloud concentration risk. Real-time access to accurate data on customers that drives machine learning models are crucial to the accuracy of predictions or recommendations they make in real time. Some countries have set guidelines for the financial sector to manage risks by not relying on a single cloud service provider, and this influences the types of native cloud services that organisations consume. Furthermore, leading financial institutions poised for a digital banking future rely on data in motion to incorporate climate risk into all risk models when accounting for ESG factors, a topic that is increasingly taking centre stage in finance and investing discussions.

There are many more use cases that we will share in our upcoming webinar that examines these in the context of trends and future challenges. 

What a platform needs to support data in motion

A platform is only truly able to harness the potential of data in motion when it can integrate data of different types and sources and covers every stage of the data life cycle, from the edge to AI. In addition, with data constantly being transformed, organisations cannot afford to overlook the importance of protecting the integrity of the data and ensuring traceability of its lineage to ensure the quality and dependability of the insights. To accommodate businesses’ evolving needs, data platforms also need to be extensible to easily support new connectors and processors.

Organisations strive toward an equilibrium of three main dimensions: lower costs, faster insights, and better performance or accuracy. The lower the costs required to process data, the less value needs to be extracted from it for the returns on investment to be worth it. As new methods and technology are created to gain insight at a reduced cost, new possibilities and use cases open up. Given that the value of insight decreases over time, the more time that has lapsed between a business event, the less time an organisation has to analyse the data that affects business-critical decisions. Just as important is the dimension of data accuracy or other measures of performance. A combination of reducing delays and reducing the number of errors greatly increases our confidence in data insight and by extension, its value.

How is data in motion relevant to a data-driven organisation?

In an earlier blog post we discussed how strategy and culture were vital components of a data-driven organisation. Utilising data in motion empowers better and quicker decisions at all levels with greater confidence. The availability of real-time product and consumer insights fosters more agility and fuels innovation while optimising operational efficiencies through predictive maintenance capabilities.

How is data in motion expected to develop in the future?

As data in motion grows in significance, we expect to see three main trends: the convergence of batch and streaming analytics within organisations, the increased implementation of hybrid architectures, and the rise of dynamic usage patterns. 

Organisations will need to unify data at rest with data in motion in novel and flexible ways while storing their data in hybrid data environments, both on premises and potentially across multiple public clouds. This will require analytical tools and platforms that simplify operations and efficiently support secure hybrid deployments. In addition, enterprise data platforms need to be elastic and scalable to accommodate dynamic and at times unpredictable workloads. 

The possibilities of data in motion are endless and will be explored in our upcoming webinar with Cloudera APAC Field CTO Daniel Hand, Are You Ready for the Future of Data in Motion? 

Explore how your organisation’s data can convert into better business outcomes by signing up for our webinar here.