Are you on the cloud-native train? Nowadays, it seems everyone is building new stateless cloud-native applications or refactoring their old “jalopies” to become container-based, microservice-architected, cloud-native “Teslas.” However, that isn’t the reality when you speak with most users. Many are still hesitant to run production workloads in the cloud because they are not sure if their data will be protected and managed. Even current cloud users have concerns about the challenges of protecting their stateful data.
Earlier this month, we announced Rubrik Alta 4.2, which extended our enterprise hybrid cloud data management capabilities. Based on customer demand, we’ve made simplifying the protection of cloud workloads a key component of our platform, beginning with Amazon Web Services (AWS) EC2 instance protection.
Protecting EC2 instances centers on efficiently managing EBS snapshots and Amazon Machine Images. We’ll dive into how Rubrik simplifies AWS EC2 protection in a follow-up blog post. But first, we need to level-set our understanding of related AWS technologies and the possibilities and challenges they present. Let’s begin with the different storage types that an EC2 instance uses.
What is an EC2 Instance Store?
An instance store is temporary or ephemeral block storage for EC2 instances. It’s a logical volume carved out from hard drives directly attached to the physical servers that host EC2 instances. While an instance store is dedicated to only only one EC2 instance, the underlying hard disks can be used to create multiple stores for multiple instances simultaneously.
An instance store and the data that is stored on it only persist for the lifetime of its attached instance. The instance store and the data that is saved to it are deleted whenever the following occurs:
- The instance is stopped
- The instance is terminated
- The underlying disk drive for the instance store fails
Unlike Elastic Block Store (EBS), an instance store is not replicated, and there is no native mechanism for making a copy of the data stored in an instance store. For this reason, instance stores are typically not used for root volumes but as storage for temporary or easily reproducible data.
What is Elastic Block Store?
EBS was introduced as a response to the need for persistent storage, larger capacity, and higher performance. EBS is a service for providing persistent block storage to EC2 instances. An EBS volume is a virtual disk, backed by hard drives across multiple storage systems, that is presented to an EC2 instance over a network. These hard drives can be SSD or spinning disks.
While an EBS volume is accessed over a network, it cannot be attached to multiple EC2 instances. A useful analogy is to think of an EBS volume as a USB drive that can be easily attached to a computer and then detached and moved to another computer. However, a USB drive cannot be attached to two computers simultaneously.
Every EBS volume is automatically replicated within the same Availability Zone to provide high availability and is the recommended option to use for all volumes, especially root volumes, when launching an EC2 instance.
What is an EBS Snapshot?
An EC2 instance with attached EBS volumes can be protected by taking periodic point-in-time snapshots of the volumes, especially the root volume. All EBS snapshots are stored in a hidden S3 bucket that is available for use in that Region. Snapshots can be used to recover data that has been deleted or an entire instance to a previous point in time.
While users can initiate and delete snapshots using the AWS console or the EC2 APIs, lifecycle management is not a part of the native AWS service. Here are the main challenges:
- EBS snapshots cannot be scheduled from the AWS console.
- There is no native policy engine for automatically expiring and deleting snapshots based on a schedule or a retention policy.
Typical workarounds for both of these challenges include cron jobs that kick off a script to delete snapshots or using a Lambda function to delete snapshots based on a CloudWatch metric.
What is an Amazon Machine Image (AMI)?
An EBS snapshot can be leveraged to create an Amazon Machine Image (AMI). An AMI captures and encapsulates all the information required to launch additional instances that are pre-configured based on the configuration of the original snapshotted instance. For example, you can create an AMI based on a running Linux instance that has MySQL software installed. You can then create additional instances using the AMI instead of having to install Linux and MySQL manually on individual instances.
An AMI includes the following:
- A template of the root volume
- Permissions detailing who can launch new instances using the AMI
- Block device mapping that specifies the volumes to attach to the instance when it’s launched
- Depending on how the AMI is created, the EC2ProductBillingCode verifies the subscription/licensing status of certain guest operating systems
Besides being used for automation creation of new EC2 instances, an AMI can also be leveraged to replicate instances in different Regions for development, migration, or disaster recovery.
How can EC2 Protection be Improved?
While Amazon has created powerful primitives, such as EBS snapshots and AMI, for protecting user data, managing these primitives can be a pain for customers. In particular, we hear often about the following challenges when attempting to protect AWS resources:
- Lack of automated lifecycle management tools: Taking snapshots, creating AMIs, and deleting them after a set time are currently manual processes or managed via homegrown tools and scripts. This presents an operational burden that customers typically do not want to have to shoulder.
- Recovery from failure or data corruption is time consuming: The process to recover using a snapshot can be time consuming and error prone. This is particularly the case for file-level recovery since customers cannot see where in their snapshot history the specific restore point for a file may exist.
- Another infrastructure brings another management tool: While there are some tools aimed at easing EBS snapshot management, they are typically very different from a customer’s existing tools for managing on-premises environments. This adds additional complexity to an already overly-complex process.
With these customer challenges in mind, Rubrik designed the Alta 4.2 release to simplify the protection of AWS EC2 instances. Find out in in the next post how we are able to accomplish this goal.