# Migrating an S3 Bucket to Another AWS Account With S3 Replication

## Introduction

Recently I've had the opportunity to assist a client transition from a single-account landing zone to a multi-account landing zone on AWS. As non-production workloads are moved to new accounts for better governance, some S3 buckets need to be moved. The client requires that the object versions and metadata (such as timestamps) be preserved since the context is important to the end users.

While there are [multiple S3 data replication approaches](https://aws.amazon.com/blogs/storage/considering-four-different-replication-options-for-data-in-amazon-s3/), the only option that preserves last-modified timestamps is using [Amazon S3 Replication](https://aws.amazon.com/s3/features/replication/). This feature takes care of the continuous replication of *new objects* from one S3 bucket to another. For one-time replication of *existing objects*, it can be achieved using [S3 Batch Replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html). One neat thing is that when you create a replication rule for the source S3 bucket, the wizard will provide you with an option to also create a batch job to replicate existing objects. Let's look at how to set this up with an example.

## Procedures

For our scenario, we have an S3 bucket named `replication-test-source-c7bwn2l4j` in account `A` which we want to replicate to a new S3 bucket named `replication-test-dest-fn7fcqjism` in account `B`. Both buckets already have [bucket versioning enabled](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html), as it is a prerequisite for replication. Assuming the default, both buckets have [server-side encryption with Amazon S3 managed keys (SSE-S3) enabled](https://docs.aws.amazon.com/AmazonS3/latest/userguide/default-bucket-encryption.html). Follow the steps below to configure the replication rule and batch replication job together:

1. Log in to account `A`.
    
2. In the S3 console, create a new bucket (for example, `replication-report-m0htap9nqp`) that will store the replication report generated by the batch replication job. This will help you troubleshoot in case the batch job fails. You don't need to enable bucket versioning on this S3 bucket.
    
3. Open the source bucket `replication-test-source-c7bwn2l4j` in the S3 console. Go the to **Management** tab and click **Create replication rule**.
    
4. On the **Create replication rule** page, enter the following information (leave all other settings as default) and click **Save**:
    
    * **Replication rule name:** `Replicate to replication-test-dest-fn7fcqjism`
        
    * **Choose a rule scope:** `Apply to all objects in the bucket`
        
    * **Destination bucket:** `Specify a bucket in another account`
        
    * **Account ID:** *The ID of the AWS account where the destination bucket resides*
        
    * **Bucket name:** `replication-test-dest-fn7fcqjism`
        
    * **Change object ownership to destination bucket owner:** *Checked*
        
    * **IAM role:** `Create new role`
        
    * **Delete marker replication:** *Checked*
        
5. When you are prompted whether you want to replicate existing objects, select **Yes, replicate existing objects** and click **Submit**.
    
    ![Replicate existing objects](https://cdn.hashnode.com/res/hashnode/image/upload/v1685171314968/8a954e1d-dd97-4018-a82e-ef58708bdff2.png align="center")
    
6. On the **Create Batch Operations job** page, let's park it for now but don't close it, as we need to complete the setup here afterward.
    
7. In a new browser session, log in to account `B` where the destination S3 bucket resides.
    
8. Open the source bucket `replication-test-dest-fn7fcqjism` in the S3 console. Go to the **Permissions** tab and click **Edit** in the **Bucket policy** section.
    
9. Copy and paste the following policy JSON, replace the bucket name (`###DEST_BUCKET_NAME###`) and the source replication IAM role (`###CRR_IAM_ROLE_ARN###`), and click **Save changes**. You can find the source replication IAM role, which was created in *step 4* above, in the IAM console in account `A`. The role would have a name with a standard prefix, which in our case is `s3crr_role_for_replication-test-source-c7bwn2l4jg`. Refer to [official instructions from AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/setting-repl-config-perm-overview.html#setting-repl-config-crossacct) for details.
    
    ```json
    {
       "Version":"2012-10-17",
       "Id":"PolicyForDestinationBucket",
       "Statement":[
          {
             "Sid":"Permissions on objects",
             "Effect":"Allow",
             "Principal":{
                "AWS":"###CRR_IAM_ROLE_ARN###"
             },
             "Action":[
                "s3:ReplicateDelete",
                "s3:ReplicateObject"
             ],
             "Resource":"arn:aws:s3:::###DEST_BUCKET_NAME###/*"
          },
          {
             "Sid":"Permissions on bucket",
             "Effect":"Allow",
             "Principal":{
                "AWS":"###CRR_IAM_ROLE_ARN###"
             },
             "Action": [
                "s3:List*",
                "s3:GetBucketVersioning",
                "s3:PutBucketVersioning"
             ],
             "Resource":"arn:aws:s3:::###DEST_BUCKET_NAME###"
          }
       ]
    }
    ```
    
    ![Bucket policy for destination bucket to allow cross-account replication from source bucket](https://cdn.hashnode.com/res/hashnode/image/upload/v1685173550414/45d0f0e4-2dc9-4286-8d34-34648c6205b7.png align="center")
    
10. Go back to the original browser session which we left off at *step 6*. On the **Create batch replication job** page, enter the following information (leave all other settings as default) and click **Save**:
    
    * **Path to completion report destination:** *Browse for* `replication-report-m0htap9nqp` *which was created in step 2*
        
11. After being redirected to the **Batch Operations** page, wait for the batch replication job to complete. It may take from minutes to hours depending on how many existing objects there are in the source S3 bucket and their sizes. Ensure that **% Complete** is 100% and **Total failed (rate)** is 0 (0%).
    
    ![Batch replication job status](https://cdn.hashnode.com/res/hashnode/image/upload/v1685174169291/60d75617-4706-4232-b51f-563b7ddad656.png align="center")
    
12. Optionally, upload a new file/object to `replication-test-source-c7bwn2l4j` in account `A`. Then wait a few minutes and check whether the file is replicated to `replication-test-dest-fn7fcqjism` in account `B`.
    

We have now successfully fully replicated the source S3 bucket `replication-test-source-c7bwn2l4j` in account `A` to the destination bucket `replication-test-dest-fn7fcqjism` in account `B`. Since the purpose is to migrate the bucket, delete the source bucket and ensure that any consumers (end-users, applications, services, etc.) update their references to point to the new (destination) S3 bucket instead.

## Additional thoughts and summary

It might be obvious, but S3 bucket names must be unique across all AWS accounts in all the AWS Regions within a partition as per the [bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html). Consequently, you won't be able to reuse the bucket name. If you insist on reusing the bucket name, you *could* try to first replicate the original bucket to a temporary bucket, then delete the original bucket. After a few hours, the original bucket name *might* become available for use again. You can then create the destination bucket in the target account using the original name and complete the replication/migration. Be warned that there is NO guarantee that the original name would be made available again by AWS!

Following this method, I was able to migrate all my client's S3 buckets successfully. I also used the same procedures (without the cross-account bucket policy) to enable cross-region backup for a production S3 bucket. I hope you find this blog post useful should you ever find yourself in need of migrating S3 buckets.