Search for a command to run...
Encrypt EBS volumes of Amazon EC2 instances efficiently with a script that automates the process using the AWS SDK for Python (Boto3).
No comments yet. Be the first to comment.
Learn how to deploy a self-hosted AI chat platform on AWS using LibreChat, MongoDB Atlas, Amazon ECS, and Amazon Bedrock, provisioned via Terraform.
Explore AWS Control Tower proactive controls, why they don’t work with Terraform, and a proof of concept that attempts to make them work. Introduction As a Terraform advocate and an AWS consultant who
I recently passed the Terraform Authoring and Operations Professional Exam. Here are my practical tips to help you prepare more effectively.
Learn how to monitor data ingestion jobs for Amazon Bedrock Knowledge Bases and enable notifications using CloudWatch Logs.
Learn how to enable logging for Amazon Bedrock Knowledge Bases, which uses the log delivery feature of Amazon CloudWatch Logs, with Terraform.
Ensuring compliance with stringent security requirements often leads to unexpected challenges - here’s one I recently tackled. The account in question has hundreds of EC2 instances with EBS volumes that are encrypted with the KMS AWS managed key aws/ebs. Due to more stringent security compliance requirements, the encryption key must be rotated every 90 days. The default one-year rotation period of the AWS managed key no longer suffices, thus all EC2 instance volumes must be re-encrypted with a custom managed key (CMK) that provides more control.
With a looming deadline, it was simply not feasible to manually re-encrypt all EBS volumes. Thus I set out to find a tool or script that can automate this daunting task before resorting to developing my own (even if it’s AI-generated). Luckily I found a GitHub repository with a script that meets 90% of my needs, and made perfect after some enhancements. I’d like to share my experience and the resulting script in this blog post in case it benefits any fellow builders facing the same problem. Let’s first set the stage by examining the encryption workflow.
To encrypt or re-encrypt an EBS volume that is attached to an EC2 instance, it is unfortunately not as simple as setting a KMS key ID on the volume. The process is a bit roundabout and involves the following steps:
Shut down the EC2 instance.
Create a snapshot of the volume.
Create a new volume from the previously created snapshot, while enabling encryption with the new KMS key. Ensure that you select the same availability zone as the original volume and apply any volume settings.
Detach the original volume from the EC2 instance while making note of the device name to which the volume is attached.
Attach the new volume to the EC2 instance with the same device name as above.
Repeat step 2 to step 6 for any additional volumes that the EC2 instance has.
Start the EC2 instance and verify that it is running properly.
Delete the original volumes and the snapshots taken during this process as appropriate.
There are other considerations and best practices for Amazon EBS encryption with auto scaling groups, spot instances, and snapshot sharing, however they are not relevant for the basic scenario for this blog post.
As you can see, the workflow is quite involved and thus makes a great candidate for automation.
The need to encrypt or re-encrypt EBS volumes is not uncommon, so I figure that someone would have developed tools and scripts for it. Indeed, a quick Google search yielded three possible options:
dwbelliston/aws_volume_encryption - a Python-based script developed by Dustin Belliston that orchestrates encryption of EBS volumes of an EC2 instance.
jbrt/ec2cryptomatic - a Go-based tool developed by Julien B. that is very similar to the Python solution above, but with a few more quality-of-life features.
aws-samples/aws-system-manager-automation-unencrypted-to-encrypted-resources - an AWS solution that automatically remediates unencrypted EBS and RDS resources using AWS Config and SSM Automation.
Given Boto3 and Python are part of my preferred toolset, I decided to leverage the aws_volume_encryption solution as my starting point. The repository has a well-written README file that provides usage instructions and detailed explanation on what each section of the code does. Be sure to check it out so you understand the general architecture and usage of the script.
Although developed years ago, the original script remains fully functional, proving its reliability. That said, I have identified a few small enhancements that could improve the usability of the script:
Defer to Boto3’s default credential search mechanism instead of adding redundant options to the script.
Create an encrypted volume directly from an encrypted snapshot.
Add KMS key ID validation and skip encryption of volumes that are already encrypted with the provided KMS key. The KMS key ID can be any of the four supported formats by the AWS API.
Add option to preserve the original volumes and add metadata tags (prefixed by VolumeEncryptionMetadata:) to them in case volume changes need to be reverted.
Improve console logging with timestamps and more details.
These enhancements allow me to test and benchmark the script more effectively, and they provide me an extra level of assurance when working on production workloads.
To test the script, create a Windows EC2 instance with an unencrypted root volume and a data volume that is encrypted using the AWS managed key:
Once the EC2 instance is running, connect to Windows, initialize the second volume as D: drive, and add a text file to help with validation later.
Then create a KMS CMK that will be used to encrypt the EBS volumes of the EC2 instance.
Lastly, clone the GitHub repository or download the code zip file, then follow the README file to set up the prerequisites including Python 3.x and the AWS CLI. Since the script defers to the typical credentials lookup sequence, you may use any of the supported methods. Typically, you either configure a profile with the AWS CLI and refer to it using the AWS_PROFILE environment variable, or you use the more verbose environment variables including AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. In any case, ensure that you provide the target region in the profile or using the AWS_DEFAULT_REGION environment variable.
Run the volume_encryption.py script while providing the following arguments:
-i with the ID of the target EC2 instance
-k with the KMS CMK ID or alias
-p to preserve the original volume, which can be inspected and deleted after validating the EC2 instance with newly encrypted volumes
Here is the command that is specific to my resources:
python volume_encryption.py -i i-095dc6901ff37f71d -k 73d13a4a-b0b0-4ced-b82b-d86a78c89df0 -p
How long the script takes largely depends on the volume sizes and the encryption state of the volumes. For my test instance with a 30 GB encrypted volume and a 8 GB encrypted volume, it took a bit over 6 minutes to complete. If I were to run the script again using another key, it would take more than 20 minutes to complete, presumably because re-encryption takes longer. In any case, the console logs include timestamps that indicate how long each step takes.
$ python volume_encryption.py -i i-095dc6901ff37f71d -k 73d13a4a-b0b0-4ced-b82b-d86a78c89df0 -p
[2025-01-19T13:51:38.781614-05:00] Checking instance i-095dc6901ff37f71d
[2025-01-19T13:51:39.624892-05:00] Stopping instance i-095dc6901ff37f71d
[2025-01-19T13:52:55.908023-05:00] Create snapshot of volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:53:57.048132-05:00] Create encrypted volume from snapshot snap-0fb3ecd0eaf583f82
[2025-01-19T13:53:57.684619-05:00] Detach volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:53:58.048124-05:00] Attach volume vol-0ddb7f75257290d19 (xvdb)
[2025-01-19T13:54:13.966946-05:00] Create snapshot of volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:14.751392-05:00] Create encrypted volume from snapshot snap-0dac8c77e6e777d1a
[2025-01-19T13:55:15.271831-05:00] Detach volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:15.615831-05:00] Attach volume vol-01140d9f4c666a763 (/dev/sda1)
[2025-01-19T13:55:31.975440-05:00] Start instance i-095dc6901ff37f71d
[2025-01-19T13:55:48.251798-05:00] Clean up resources
[2025-01-19T13:55:48.252797-05:00] Delete snapshot snap-0fb3ecd0eaf583f82
[2025-01-19T13:55:48.438101-05:00] Skipping deletion of original volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:55:48.438101-05:00] Delete snapshot snap-0dac8c77e6e777d1a
[2025-01-19T13:55:48.614087-05:00] Skipping deletion of original volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:48.615090-05:00] Encryption finished
$
Once the script completes, verify that the EBS volumes are encrypted with the new KMS key.
You should also see that the original volumes still exist and have some metadata tags added by the script for traceability. Note the original volume IDs from the console logs of the script (for example, vol-054b9017ef1f8f25c is the ID of the original root volume).
Lastly, log in to the EC2 instance and ensure that Windows is working as intended and D:\hello.txt still exists.
However, you will notice that the EC2 instance is seemingly slower than usual. This is because volumes that are restored from snapshots are not fully initialized or pre-warmed. This means that only when a block within the volume is accessed that it will be loaded from the snapshot that’s stored in S3 behind the scenes, thus increasing the I/O latency. While this may not be a huge issue with common use cases, for use cases with high disk I/O needs (such as running a database), you may need to manually initialize the disks.
With this improved script, you can (re-)encrypt the EBS volumes of any EC2 instance with ease. If you are encrypting volumes for many instances, you can also write another script that reads a CSV file containing EC2 instance information and runs volume_encryption.py on multiple instances in parallel. AI tools like ChatGPT, Amazon Q Developer, or GitHub Copilot can easily create one for you, as I did for my own needs. I will leave this as an exercise for the audience.
As they say, prevention is better than cure. If your organization’s security policies require that EBS volumes be encrypted, consider using the Amazon EBS encryption by default feature to automatically encrypt any new EBS volumes.
This demonstrates how automation and generative AI empower DevOps engineers to tackle complex challenges efficiently. I hope you find this blog post informative and the script useful should you run into similar situations. If you like this article, please check out my other blog posts for more helpful and intriguing content on AWS and DevOps. Thank you for reading and have a great one!