Managing Cost and Risks with Cloud Data Management

Cloud is becoming an increasing popular option for organisations to store data to power data analytics and business intelligence functions: data that is growing rapidly in volume and variety. However migrating data to the cloud and managing it in the cloud creates multiple challenges. This has given rise to a new discipline in enterprise IT: cloud data management.

Cloud data management can be defined as “the implementation of platforms and tools, policies and procedures that give organisations control of their business data, both in the cloud and in setups where data is stored or sourced in a combination of on-premises and cloud applications.

This brandpost will explore cloud data management and the challenges it faces: to manage costs, ensure security, apply governance, avoid lock-in, and enable an organisation to respond effectively to environmental, regulatory and competitive pressures.

Finally, it will examine how the Cloudera Cloud Data Platform fulfils the requirements of cloud data management.

COST MANAGEMENT IN THE CLOUD

Good cloud management is essential to control the costs of using cloud. These costs can easily become excessive, often as a result charges for data ingress and egress and applications that are inefficient in a cloud environment and chew up costly cloud resources.

A recent survey of 750 senior enterprise IT professionals in data-intensive businesses found more than a third having cloud budget overruns of between 20 and 40 percent, and one in 12 experienced even greater overruns.  Companies polled ranged in size from 500 to more than 5000 employees.

Forty seven percent of respondents used a private cloud, 21 percent public cloud, and 28 percent a combination of both. They spent from $US500,000 to more than $US10m on big data analytics. Most respondents said the highest priority for their big data cloud initiative was to optimise cloud resources.

There are many aspects of operating in the cloud where the costs may not be estimated accurately and can easily become excessive. They include:

• Managing multiple clouds

The cost of monitoring and managing applications in a hybrid cloud environment that spans public and private cloud and on-premises data centres can be significant.

• Migrating legacy applications

Applications designed for legacy environments can be very inefficient users of cloud resources unless significant re-architecting is undertaken. Either option can incur substantial costs.

• Hidden costs

Migration to the cloud consumes significant human resources and operating in the cloud requires different skills. These costs can be significant and often are not fully factored in.

SECURITY & GOVERNANCE CHALLENGES IN THE CLOUD

If cloud services are to be used efficiently and securely they need good governance: a set of rules covering people, processes and technology for the planning, implementation and operation of cloud facilities. Achieving and maintaining good cloud governance can be challenging.

As hybrid and multi-cloud landscapes have become the norm for many organisations, so have the challenges of keeping data security and governance policies consistent between different deployments. Implementing and synchronising policies between public clouds and data centres can require a tremendous amount of operational effort. These efforts delay delivering access to data and analytics for end users and create potential security risks.

UK based IT Governance Ltd—which helps organisations implement and maintain a cloud governance framework—notes that an organisation’s board is responsible and accountable for the framework of standards, processes and activities associated with its use of cloud computing, and that “Organisations need to satisfy themselves of the security and resilience of their cloud service providers.”

It references the Cloud Security Alliance’s Cloud Controls Matrix, cybersecurity control framework for cloud computing. This comprises 197 control objectives structured in 17 domains covering all key aspects of cloud technology. It “can be used as a tool for the systematic assessment of a cloud implementation and provides guidance on which security controls should be implemented by which actor within the cloud supply chain.”

In short, organisations face considerable and complex challenges to ensure the security of their data in the cloud.

In 2019 the Cloud Security Alliance tapped over 200 experts to come up with a the “Egregious Eleven”: the top threats to cloud computing

1.    data breaches

2.    misconfigurations and inadequate change control

3.    lack of cloud security architecture and strategy

4.    insufficient identity, credential, access and key management

5.    account hijacking

6.    insider threats

7.    insecure interfaces and APIs

8.    weak control plane

9.    metastructure (the protocols and mechanisms that provide the interface between the infrastructure layer and the other layers) and applistructure (the amalgamation of applications and technological infrastructure) failures

10.  limited cloud usage visibility

11.  abuse and nefarious use of cloud services

MANAGING SECURITY AND GOVERNANCE CHALLENGES WITH THE CLOUDERA DATA PLATFORM

The Cloudera Data Platform (CDP) can greatly alleviate the governance challenges of operating cloud services. It is a platform-as-a-service (PaaS) that ensures the security of data and infrastructure in the public cloud. It uses a control plane architecture that divides the infrastructure of the platform between orchestration components on Cloudera’s public cloud infrastructure and data storage and processing resources in the customer’s public cloud accounts.

This provides the agility and elasticity of the cloud, without the burden of learning provider-specific automation and security tools necessary to scale to enterprise levels.

CDP’s public cloud architecture and Cloudera’s Shared Data Experience (SDX ensure deployments are secure by design. CDP’s Platform-as-a-Service (PaaS) architecture implements industry best practices to comprehensively secure deployments while the SDX data context layer ensures consistent data security and governance as data is used.

Together they provide the flexibility needed to make data available to end users without compromise, while taking full advantage of the agility and elasticity of the public cloud.

All public cloud accounts are managed through a single pane of glass, the CDP Control Plane. CDP’s public cloud architecture has been designed to ensure that customer data is not sent to Cloudera’s infrastructure but always remains within the customer’s workload environment.

To perform necessary operations within accounts on different public clouds CDP uses dedicated roles specific to each public cloud service, the method preferred by providers of public cloud services.

The CDP Control Plane provides comprehensive monitoring of the technologies deployed; the infrastructure that resides within each account continuously reports its health status back to the Control Plane.

The full range of CDP facilities come together in the Data Catalogue: a single pane of glass for data stewards and users alike through which to explore all an organisation’s data assets.