Search for a command to run...
Discover how to utilize Amazon S3 Replication for efficient migration of an S3 bucket to a different AWS account (well, minus the bucket name).
No comments yet. Be the first to comment.
Learn how to deploy a self-hosted AI chat platform on AWS using LibreChat, MongoDB Atlas, Amazon ECS, and Amazon Bedrock, provisioned via Terraform.
Explore AWS Control Tower proactive controls, why they don’t work with Terraform, and a proof of concept that attempts to make them work. Introduction As a Terraform advocate and an AWS consultant who
I recently passed the Terraform Authoring and Operations Professional Exam. Here are my practical tips to help you prepare more effectively.
Learn how to monitor data ingestion jobs for Amazon Bedrock Knowledge Bases and enable notifications using CloudWatch Logs.
Learn how to enable logging for Amazon Bedrock Knowledge Bases, which uses the log delivery feature of Amazon CloudWatch Logs, with Terraform.
Recently I've had the opportunity to assist a client transition from a single-account landing zone to a multi-account landing zone on AWS. As non-production workloads are moved to new accounts for better governance, some S3 buckets need to be moved. The client requires that the object versions and metadata (such as timestamps) be preserved since the context is important to the end users.
While there are multiple S3 data replication approaches, the only option that preserves last-modified timestamps is using Amazon S3 Replication. This feature takes care of the continuous replication of new objects from one S3 bucket to another. For one-time replication of existing objects, it can be achieved using S3 Batch Replication. One neat thing is that when you create a replication rule for the source S3 bucket, the wizard will provide you with an option to also create a batch job to replicate existing objects. Let's look at how to set this up with an example.
For our scenario, we have an S3 bucket named replication-test-source-c7bwn2l4j in account A which we want to replicate to a new S3 bucket named replication-test-dest-fn7fcqjism in account B. Both buckets already have bucket versioning enabled, as it is a prerequisite for replication. Assuming the default, both buckets have server-side encryption with Amazon S3 managed keys (SSE-S3) enabled. Follow the steps below to configure the replication rule and batch replication job together:
Log in to account A.
In the S3 console, create a new bucket (for example, replication-report-m0htap9nqp) that will store the replication report generated by the batch replication job. This will help you troubleshoot in case the batch job fails. You don't need to enable bucket versioning on this S3 bucket.
Open the source bucket replication-test-source-c7bwn2l4j in the S3 console. Go the to Management tab and click Create replication rule.
On the Create replication rule page, enter the following information (leave all other settings as default) and click Save:
Replication rule name: Replicate to replication-test-dest-fn7fcqjism
Choose a rule scope: Apply to all objects in the bucket
Destination bucket: Specify a bucket in another account
Account ID: The ID of the AWS account where the destination bucket resides
Bucket name: replication-test-dest-fn7fcqjism
Change object ownership to destination bucket owner: Checked
IAM role: Create new role
Delete marker replication: Checked
When you are prompted whether you want to replicate existing objects, select Yes, replicate existing objects and click Submit.
On the Create Batch Operations job page, let's park it for now but don't close it, as we need to complete the setup here afterward.
In a new browser session, log in to account B where the destination S3 bucket resides.
Open the source bucket replication-test-dest-fn7fcqjism in the S3 console. Go to the Permissions tab and click Edit in the Bucket policy section.
Copy and paste the following policy JSON, replace the bucket name (###DEST_BUCKET_NAME###) and the source replication IAM role (###CRR_IAM_ROLE_ARN###), and click Save changes. You can find the source replication IAM role, which was created in step 4 above, in the IAM console in account A. The role would have a name with a standard prefix, which in our case is s3crr_role_for_replication-test-source-c7bwn2l4jg. Refer to official instructions from AWS for details.
{
"Version":"2012-10-17",
"Id":"PolicyForDestinationBucket",
"Statement":[
{
"Sid":"Permissions on objects",
"Effect":"Allow",
"Principal":{
"AWS":"###CRR_IAM_ROLE_ARN###"
},
"Action":[
"s3:ReplicateDelete",
"s3:ReplicateObject"
],
"Resource":"arn:aws:s3:::###DEST_BUCKET_NAME###/*"
},
{
"Sid":"Permissions on bucket",
"Effect":"Allow",
"Principal":{
"AWS":"###CRR_IAM_ROLE_ARN###"
},
"Action": [
"s3:List*",
"s3:GetBucketVersioning",
"s3:PutBucketVersioning"
],
"Resource":"arn:aws:s3:::###DEST_BUCKET_NAME###"
}
]
}
Go back to the original browser session which we left off at step 6. On the Create batch replication job page, enter the following information (leave all other settings as default) and click Save:
replication-report-m0htap9nqp which was created in step 2After being redirected to the Batch Operations page, wait for the batch replication job to complete. It may take from minutes to hours depending on how many existing objects there are in the source S3 bucket and their sizes. Ensure that % Complete is 100% and Total failed (rate) is 0 (0%).
Optionally, upload a new file/object to replication-test-source-c7bwn2l4j in account A. Then wait a few minutes and check whether the file is replicated to replication-test-dest-fn7fcqjism in account B.
We have now successfully fully replicated the source S3 bucket replication-test-source-c7bwn2l4j in account A to the destination bucket replication-test-dest-fn7fcqjism in account B. Since the purpose is to migrate the bucket, delete the source bucket and ensure that any consumers (end-users, applications, services, etc.) update their references to point to the new (destination) S3 bucket instead.
It might be obvious, but S3 bucket names must be unique across all AWS accounts in all the AWS Regions within a partition as per the bucket naming rules. Consequently, you won't be able to reuse the bucket name. If you insist on reusing the bucket name, you could try to first replicate the original bucket to a temporary bucket, then delete the original bucket. After a few hours, the original bucket name might become available for use again. You can then create the destination bucket in the target account using the original name and complete the replication/migration. Be warned that there is NO guarantee that the original name would be made available again by AWS!
Following this method, I was able to migrate all my client's S3 buckets successfully. I also used the same procedures (without the cross-account bucket policy) to enable cross-region backup for a production S3 bucket. I hope you find this blog post useful should you ever find yourself in need of migrating S3 buckets.