Introduction

Recently I've had the opportunity to assist a client transition from a single-account landing zone to a multi-account landing zone on AWS. As non-production workloads are moved to new accounts for better governance, some S3 buckets need to be moved. The client requires that the object versions and metadata (such as timestamps) be preserved since the context is important to the end users.

While there are multiple S3 data replication approaches, the only option that preserves last-modified timestamps is using Amazon S3 Replication. This feature takes care of the continuous replication of new objects from one S3 bucket to another. For one-time replication of existing objects, it can be achieved using S3 Batch Replication. One neat thing is that when you create a replication rule for the source S3 bucket, the wizard will provide you with an option to also create a batch job to replicate existing objects. Let's look at how to set this up with an example.

Procedures

For our scenario, we have an S3 bucket named replication-test-source-c7bwn2l4j in account A which we want to replicate to a new S3 bucket named replication-test-dest-fn7fcqjism in account B. Both buckets already have bucket versioning enabled, as it is a prerequisite for replication. Assuming the default, both buckets have server-side encryption with Amazon S3 managed keys (SSE-S3) enabled. Follow the steps below to configure the replication rule and batch replication job together:

1. Log in to account A.

2. In the S3 console, create a new bucket (for example, replication-report-m0htap9nqp) that will store the replication report generated by the batch replication job. This will help you troubleshoot in case the batch job fails. You don't need to enable bucket versioning on this S3 bucket.

3. Open the source bucket replication-test-source-c7bwn2l4j in the S3 console. Go the to Management tab and click Create replication rule.

4. On the Create replication rule page, enter the following information (leave all other settings as default) and click Save:

   • Replication rule name: Replicate to replication-test-dest-fn7fcqjism

   • Choose a rule scope: Apply to all objects in the bucket

   • Destination bucket: Specify a bucket in another account

   • Account ID: The ID of the AWS account where the destination bucket resides

   • Bucket name: replication-test-dest-fn7fcqjism

   • Change object ownership to destination bucket owner: Checked

   • IAM role: Create new role

   • Delete marker replication: Checked

5. When you are prompted whether you want to replicate existing objects, select Yes, replicate existing objects and click Submit.

   

6. On the Create Batch Operations job page, let's park it for now but don't close it, as we need to complete the setup here afterward.

7. In a new browser session, log in to account B where the destination S3 bucket resides.

8. Open the source bucket replication-test-dest-fn7fcqjism in the S3 console. Go to the Permissions tab and click Edit in the Bucket policy section.

9. Copy and paste the following policy JSON, replace the bucket name (###DEST_BUCKET_NAME###) and the source replication IAM role (###CRR_IAM_ROLE_ARN###), and click Save changes. You can find the source replication IAM role, which was created in step 4 above, in the IAM console in account A. The role would have a name with a standard prefix, which in our case is s3crr_role_for_replication-test-source-c7bwn2l4jg. Refer to official instructions from AWS for details.

    {
       "Version":"2012-10-17",
       "Id":"PolicyForDestinationBucket",
       "Statement":[
          {
             "Sid":"Permissions on objects",
             "Effect":"Allow",
             "Principal":{
                "AWS":"###CRR_IAM_ROLE_ARN###"
             },
             "Action":[
                "s3:ReplicateDelete",
                "s3:ReplicateObject"
             ],
             "Resource":"arn:aws:s3:::###DEST_BUCKET_NAME###/*"
          },
          {
             "Sid":"Permissions on bucket",
             "Effect":"Allow",
             "Principal":{
                "AWS":"###CRR_IAM_ROLE_ARN###"
             },
             "Action": [
                "s3:List*",
                "s3:GetBucketVersioning",
                "s3:PutBucketVersioning"
             ],
             "Resource":"arn:aws:s3:::###DEST_BUCKET_NAME###"
          }
       ]
    }
   

   

10. Go back to the original browser session which we left off at step 6. On the Create batch replication job page, enter the following information (leave all other settings as default) and click Save:

    • Path to completion report destination: Browse for replication-report-m0htap9nqp which was created in step 2

11. After being redirected to the Batch Operations page, wait for the batch replication job to complete. It may take from minutes to hours depending on how many existing objects there are in the source S3 bucket and their sizes. Ensure that % Complete is 100% and Total failed (rate) is 0 (0%).

    

12. Optionally, upload a new file/object to replication-test-source-c7bwn2l4j in account A. Then wait a few minutes and check whether the file is replicated to replication-test-dest-fn7fcqjism in account B.

We have now successfully fully replicated the source S3 bucket replication-test-source-c7bwn2l4j in account A to the destination bucket replication-test-dest-fn7fcqjism in account B. Since the purpose is to migrate the bucket, delete the source bucket and ensure that any consumers (end-users, applications, services, etc.) update their references to point to the new (destination) S3 bucket instead.

Additional thoughts and summary

It might be obvious, but S3 bucket names must be unique across all AWS accounts in all the AWS Regions within a partition as per the bucket naming rules. Consequently, you won't be able to reuse the bucket name. If you insist on reusing the bucket name, you could try to first replicate the original bucket to a temporary bucket, then delete the original bucket. After a few hours, the original bucket name might become available for use again. You can then create the destination bucket in the target account using the original name and complete the replication/migration. Be warned that there is NO guarantee that the original name would be made available again by AWS!

Following this method, I was able to migrate all my client's S3 buckets successfully. I also used the same procedures (without the cross-account bucket policy) to enable cross-region backup for a production S3 bucket. I hope you find this blog post useful should you ever find yourself in need of migrating S3 buckets.