Generative AI has fundamentally shifted how we approach work, from writing and coding to research and problem-solving. For the past year, I used ChatGPT Business almost daily at work to improve my writing and research. However, I noticed limitations like fabrication and confirmation bias, so I wanted to explore how other non-OpenAI models perform. Additionally, my organization is consolidating on Microsoft 365 Copilot, which doesn't match ChatGPT's capabilities for my needs. This led me to search for a self-hosted, ChatGPT-like platform with flexibility in model choices.

I also needed it to be web-based for team members to access. As an AWS advocate, I wanted to leverage a diverse set of foundational models that Amazon Bedrock has to offer, and to host the platform using primarily AWS services. Based on my research, the three main options are LibreChat, Open WebUI, and AnythingLLM. Given that LibreChat is more feature-rich, customizable, and seemingly easier to deploy, I decided to give it a try and share my experience.

Without further ado, let's walk through the solution architecture and how it addresses my requirements.

Architecture Overview

The main design principle of the solution is to be cost-effective initially while allowing flexibility to scale in the future. Minimizing cost also means reducing operational overhead, not just service charges.

While cramming all components into an Amazon Lightsail instance is the cheapest option, it would need to be re-architected to scale horizontally. Deploying to an Amazon EC2 instance provides more flexibility, but it requires manual LibreChat installation and VM management. Ultimately, I decided to take a more modern approach and adopt a componentized architecture depicted in the following diagram:

This architecture uses the following technologies:

• MongoDB Atlas - The LibreChat database runs on MongoDB Atlas using the Free (M0) cluster tier. It's technically free, runs in AWS, and is sufficient as a starter database engine as recommended by LibreChat.

• Amazon ECS with AWS Fargate - The LibreChat application runs as a container with 512 (0.5) vCPU and 1 GB memory using 64-bit ARM architecture, which is sufficient when not enabling too many LibreChat features. Secrets are stored in AWS Systems Manager (SSM) Parameter Store (free), and configurations are stored in an Amazon S3 bucket to avoid additional shared storage services.

• Application Load Balancer (ALB) - The public-facing endpoint. Although there is a running cost, its simpler TLS setup and support for scaling and AWS WAF integration makes it worthwhile.

• fck-nat - Provides NAT gateway functionality using a pair of EC2 t4g.nano instances instead of AWS-managed NAT Gateways. This significantly reduces NAT gateway data transfer charges, making it a cost-effective option for modest traffic volumes.

The monthly cost of this architecture should be about $50 USD in us-east-1 including moderate Bedrock model use. To prevent surprise bills, set up a budget in AWS Budgets and create a cost monitor in AWS Cost Anomaly Detection.

Now that we have a good understanding of the architecture, let's go through the LibreChat concepts and prerequisites before we look at the Terraform configuration.

Deploying a MongoDB Atlas Database

Since an official Terraform MongoDB Atlas Provider is available, let's use it to provision the database for LibreChat. If you have not already done so, sign up for a new account using the Get Started button on the MongoDB website.

Once you've completed sign-up, log in to the MongoDB Atlas Console. MongoDB Atlas automatically creates an organization and a sample project named Project 0. Since we'll use Terraform to create a new project, feel free to delete this sample project. You may also edit the organization name in Organizational Settings as needed.

Next, create an API key at the organization level for the Terraform provider. In the MongoDB Atlas Console, go to the organization level view and select Identity & Access > Applications in the left menu. On the Application page, select the Service Accounts tab and click Create service account:

On the Create Service Account page, enter a name (for example, "terraform") and a description (for example, "Service account for Terraform"), keep the client secret expiration as recommended, select the Organization Project Creator permission, and click Create. Copy both the client ID and secret from the next page for use with Terraform:

We're now ready to write the Terraform configuration to:

1. Create a new project

2. Create a new Free (M0) cluster with AWS as the backing provider

3. Create a database user for LibreChat

4. Add the NAT Gateway's public IP to the project IP Access List for security

To avoid hardcoding credentials, set the service account credentials as environment variables before running Terraform using MONGODB_ATLAS_CLIENT_ID and MONGODB_ATLAS_CLIENT_SECRET as per the provider configuration.

The following Terraform configuration provisions these MongoDB Atlas resources. Note that some attributes are provided as variables for flexibility, and the IP access list CIDR block refers to the NAT service elastic IPs (since egress traffic from LibreChat container goes through the fck-nat instances or NAT gateways):

resource "mongodbatlas_project" "librechat" {
  org_id = var.atlas_org_id
  name   = var.atlas_project_name
}

resource "mongodbatlas_advanced_cluster" "librechat" {
  project_id   = mongodbatlas_project.librechat.id
  name         = var.atlas_cluster_name
  cluster_type = "REPLICASET"

  replication_specs = [
    {
      region_configs = [
        {
          electable_specs = {
            instance_size = "M0"
          }
          provider_name         = "TENANT"
          backing_provider_name = "AWS"
          region_name           = var.atlas_region_name
          priority              = 7
        }
      ]
    }
  ]
}

resource "mongodbatlas_project_ip_access_list" "librechat" {
  for_each   = var.use_fck_nat ? aws_eip.fck_nat : aws_nat_gateway.zonal
  project_id = mongodbatlas_project.librechat.id
  cidr_block = "${each.value.public_ip}/32"
  comment    = "ECS egress CIDR for LibreChat"
}

resource "random_password" "atlas_db" {
  length = 16
}

resource "mongodbatlas_database_user" "librechat" {
  project_id         = mongodbatlas_project.librechat.id
  username           = var.atlas_db_username
  password           = random_password.atlas_db.result
  auth_database_name = "admin"

  roles {
    role_name     = "readWriteAnyDatabase"
    database_name = "admin"
  }
}

Although using an IAM role to authenticate the Atlas database user would be more secure, it unfortunately doesn't work with the off-the-shelf LibreChat Docker image because it lacks the aws4 module required by the Mongoose library for AWS authentication. To avoid rebuilding a new container image, we'll stick with password authentication for now.

Strategies for Managing LibreChat Configuration

LibreChat uses two main sources of configuration: environment variables and LibreChat YAML. Environment variables are typically provided via a .env file created from the example in LibreChat's GitHub repository. Most LibreChat configuration is done using environment variables, and the LibreChat YAML file references these variables using the ${} notation for values such as API keys.

Additionally, the LibreChat YAML file (librechat.yaml) is typically placed in the application folder or mounted as an override in a containerized environment. However, it can also be provided in other locations by specifying the configuration path using the CONFIG_PATH environment variable.

Running LibreChat as a container introduces some challenges:

1. Building custom images is inefficient - While it's possible to build a LibreChat container image with .env and librechat.yaml baked in, rebuilding the image for every configuration change is inefficient. It's ideal to use the official LibreChat image from Docker Hub.

2. Managing many environment variables is difficult - Setting environment variables directly without using .env is hard to manage due to the sheer number of variables to configure, even when omitting those irrelevant to your use case.

3. Security concerns - Providing security-sensitive information as plain-text environment variables is not a security best practice.

ECS provides features to address these concerns:

1. Environment variable files - ECS supports passing environment variables via an environment variable file stored in S3. Since LibreChat already uses this format, it's a perfect fit.

2. Secrets management - AWS Secrets Manager or AWS Systems Manager (SSM) Parameter Store can securely pass sensitive data to containers as environment variables, avoiding hardcoded credentials.

To keep costs low, we will define all sensitive data as SSM Parameter Store parameters of type SecureString, while keeping all other configuration in a .env file provided to the container via ECS.

Providing the LibreChat YAML file to the container is more complicated. While the standard approach uses persistent storage options like Amazon EFS, that's cost-prohibitive for managing a single file. A better approach is using a sidecar container to write the file to a task-level shared volume before the main container starts.

For this, I implemented a sidecar container using busybox to decode a base64-encoded librechat.yaml (provided as an environment variable) and write it to /config/librechat.yaml in the task-level shared volume. The LibreChat container references this path using the CONFIG_PATH environment variable. The resulting container definitions (defined in the ECS task definition) are shown below:

  container_definitions = jsonencode([
    {
      name      = "init-librechat-config"
      image     = "public.ecr.aws/docker/library/busybox:1.36"
      essential = false

      command = [
        "sh",
        "-lc",
        "mkdir -p /config && printf '%s' \"$LIBRECHAT_YAML_B64\" | base64 -d > /config/librechat.yaml"
      ]

      environment = [
        {
          name  = "LIBRECHAT_YAML_B64"
          value = filebase64("\({path.module}/librechat/\){var.librechat_version}/librechat.yaml")
        }
      ]

      mountPoints = [
        {
          sourceVolume  = "librechat-config"
          containerPath = "/config"
          readOnly      = false
        }
      ]

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.librechat.name
          "awslogs-region"        = var.region
          "awslogs-stream-prefix" = "librechat"
        }
      }
    },
    {
      name      = "librechat"
      image     = "librechat/librechat:v0.8.4"
      essential = true

      dependsOn = [
        {
          containerName = "init-librechat-config"
          condition     = "SUCCESS"
        }
      ]

      portMappings = [
        {
          containerPort = 3080
          protocol      = "tcp"
        }
      ]

      secrets = [
        {
          name      = "CREDS_KEY"
          valueFrom = aws_ssm_parameter.librechat_creds_key.arn
        },
        {
          name      = "CREDS_IV"
          valueFrom = aws_ssm_parameter.librechat_creds_iv.arn
        },
        {
          name      = "JWT_SECRET"
          valueFrom = aws_ssm_parameter.librechat_jwt_secret.arn
        },
        {
          name      = "JWT_REFRESH_SECRET"
          valueFrom = aws_ssm_parameter.librechat_jwt_refresh_secret.arn
        },
        {
          name      = "MEILI_MASTER_KEY"
          valueFrom = aws_ssm_parameter.librechat_meili_master_key.arn
        },
        {
          name      = "MONGO_URI"
          valueFrom = aws_ssm_parameter.librechat_mongo_uri.arn
        }
      ]

      environment = [
        {
          name  = "CONFIG_PATH"
          value = "/config/librechat.yaml"
        }
      ]

      environmentFiles = [
        {
          value = aws_s3_object.librechat_dot_env.arn
          type  = "s3"
        }
      ]

      mountPoints = [
        {
          sourceVolume  = "librechat-config"
          containerPath = "/config"
          readOnly      = true
        }
      ]

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.librechat.name
          "awslogs-region"        = var.region
          "awslogs-stream-prefix" = "librechat"
        }
      }
    }
  ])

Now that we have a strategy for managing LibreChat configuration, let's define the minimal set of environment variables and LibreChat YAML as a starting point.

Preparing the Starter Environment File

LibreChat provides starter files .env.example and librechat.example.yaml that we'll use as the basis for our configuration. Let's start with the environment file.

Environment File (.env)

Copy the .env.example file to a local folder and comment out any sensitive values that will be provided as SSM Parameter Store environment variables specified below. Although individually defined environment variables take precedence over variables in environment files per the Amazon ECS Developer Guide, commenting them out avoids confusion.

In addition to MONGO_URI (the MongoDB connection string), LibreChat recommends adjusting any "secret" values from their default value for added security:

• CREDS_IV - 16-byte Initialization Vector (IV) (32 characters in hex) for securely storing credentials

• CREDS_KEY - 32-byte key (64 characters in hex) for securely storing credentials

• JWT_SECRET - 32-byte key (64 characters in hex) as the JWT secret key

• JWT_REFRESH_SECRET - 32-byte key (64 characters in hex) as the JWT refresh secret key

• MEILI_MASTER_KEY - 16-byte key (32 characters in hex) as the MeiliSearch master key (required only if message and conversation search is enabled)

LibreChat provides a Credentials Generator to generate cryptographically secure random values for these secrets. Store them as SSM Parameter Store SecureString parameters or generate and store them directly using Terraform.

Next, adjust these environment variables for proper and secure operation:

• HOST - Set to 0.0.0.0 to listen on all network interfaces, allowing ALB access

• CONSOLE_JSON - Set to true to write logs to CloudWatch in JSON format for easier querying

• ALLOW_REGISTRATION - Set to false to disable self-registration (we will create users with the create user script instead)

• SEARCH - Set to false to disable message and conversation search (we're only demonstrating a minimal setup)

For the AI provider, we'll enable AWS Bedrock. Set ENDPOINTS to bedrock (the .env.example enables all pre-configured endpoints except Bedrock). Then configure the Bedrock endpoint by setting the following environment variables:

• BEDROCK_AWS_DEFAULT_REGION - Set to your Bedrock region (e.g., us-east-1)

• BEDROCK_AWS_MODELS - Set to us.amazon.nova-2-lite-v1:0 to use Amazon Nova Lite as a starting point (referring to the US Nova Lite system-defined inference profile)

• OPENAI_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to OpenAI APIs even if the endpoint is disabled via ENDPOINTS.

• ANTHROPIC_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to Anthropic APIs even if the endpoint is disabled via ENDPOINTS.

• GOOGLE_KEY - Comment out to ensure that LibreChat does not try to make any calls to Google APIs even if the endpoint is disabled via ENDPOINTS.

• ASSISTANTS_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to OpenAI Assistants APIs even if the endpoint is disabled via ENDPOINTS.

Note that we'll be using the ECS task IAM role to allow LibreChat to seamlessly call Bedrock APIs, so we don't need to set BEDROCK_AWS_ACCESS_KEY_ID nor BEDROCK_AWS_SECRET_ACCESS_KEY.

LibreChat YAML File (librechat.yaml)

Copy the librechat.example.yaml file to a local folder. For this minimal setup, we won't need to define custom endpoints or configure advanced settings. However, to prevent custom endpoints like Groq and Mistral AI from appearing in the UI, comment out the custom key in the endpoints block and set it to an empty array [].

With all environment variables set in the environment file or SSM Parameter Store, and the librechat.yaml file prepared, we're ready to tie everything together with Terraform.

Building the Terraform Configuration

Since this blog post is getting quite long, let's focus on the key design elements of the Terraform configuration. You can find the complete Terraform configuration and source code in the 1_ecs_basic directory in this GitHub repository. Here are the descriptions for each Terraform configuration file:

• atlas.tf - Defines the MongoDB Atlas resources, as explained in the earlier section of this blog post.

• vpc.tf - Defines the VPC infrastructure for the solution. Key design elements include:

  • The VPC design follows a three-tier architecture across two AZs. Although the database subnets are currently not in use, they can be utilized for AWS cache and database services in the future.

  • There is built-in support for either fck-nat (default) or NAT Gateways, depending on your preference.

• s3.tf - Defines the S3 bucket that hosts the LibreChat files and the S3 object for the .env file. Key design elements include:

  • A ready-to-use .env file is included in the librechat/v0.8.4 folder. You can edit this file if using the same version, or upload a new .env file when you upgrade LibreChat (be sure to change the librechat_version variable).

  • This S3 bucket may be used in the future as the LibreChat file storage backend, hence the .env file is placed in the config subfolder for better separation.

• ecs.tf - Defines all ECS and related resources to run LibreChat as an ECS service. Key design elements include:

  • A ready-to-use librechat.yaml file is included in the librechat/v0.8.4 folder. You can edit this file if using the same version, or upload a new librechat.yaml file when you upgrade LibreChat (be sure to change the librechat_version variable).

  • The LibreChat credentials are generated by first creating a random password using the random_password ephemeral resource, then defining an aws_ssm_parameter resource with the write-only value storing the hash of the password (SHA1 or SHA256). You can replace this logic if you prefer to manually store the generated credentials in SSM Parameter Store first.

  • The IAM policy for the ECS task role, aws_iam_role_policy.ecs_task_librechat, contains permissions to invoke Bedrock models and manage marketplace subscription of third-party models.

  • Service auto-scaling is defined for future use, but for now, it is scaled to 1 for cost control.

  • The LibreChat container runs on TCP port 3080 by default.

  • CloudWatch logs are streamed to the /ecs/librechat log group.

• alb.tf - Defines the ALB resources as the public-facing endpoint for LibreChat. Key design elements include:

  • A HTTPS listener is defined with HTTP redirect to ensure security. Consequently, you must import or create a TLS certificate in AWS Certificate Manager (ACM), then pass the certificate's ARN using the alb_certificate_arn variable.

  • HTTPS also requires a custom host name, so you must pass the DNS name using the librechat_dns_name variable.

  • The target group checks the ECS task health using LibreChat's /health endpoint.

Deploying the Solution

After cloning the GitHub repository, deploy the solution as follows:

1. From the root of the cloned GitHub repository, navigate to 1_ecs_basic.

2. Set the MONGODB_ATLAS_CLIENT_ID and MONGODB_ATLAS_CLIENT_SECRET to the Atlas service account credentials created in the earlier section.

3. Configure your AWS credentials using aws configure for IAM, or aws configure sso for IAM Identity Center. The profile name will be provided as a Terraform variable.

4. Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.

5. Run terraform init and terraform apply.

It is advised to check the ECS service status in the AWS Management Console to ensure that the tasks run successfully. Failed tasks will cause a perpetual restart due to the required capacity of 1, and overlooking this may lead to unexpected cost consequences. Check the task logs in the CloudWatch log group /ecs/librechat for errors if needed.

Once the configuration is applied, create the CNAME record for the ALB DNS name. If you manage your domain/subdomain using a public hosted zone in Amazon Route 53, you can also create an alias record pointing to the ALB. Lastly, go to the custom host name for LibreChat and ensure that the login page loads successfully.

Creating a User and Validating LibreChat

Since self-registration is disabled, we need to use the create user script to create the first user. The easiest way is to use ECS Exec in the ECS console to run the script in the librechat container in the task configuration. Here's a screenshot showing where to find the Connect button to open an interactive session:

Once CloudShell opens and connects to the container's shell, run the following command to start the user creation wizard:

npm run create-user

Enter the user's information as prompted, and a user will be created in LibreChat's database. Here's an example of creating a user for John Doe:

Now you're ready to log in. Open your LibreChat application URL and log in using the credentials you just created. Upon successful login, accept LibreChat's terms of service. You should see the Nova 2 Lite model already selected at the top, since it's the only configured model.

Let's test the setup with a simple prompt:

Tell me a joke about AWS.

If LibreChat responds with a joke, you've successfully completed the setup! Here's the joke I received, which honestly suggests that the Nova model could use some additional training with a better comedy dataset...

Summary

Congratulations, you now have your own LibreChat instance in AWS! You're now ready to start exploring and expanding its capabilities. While this setup gives you a functional chat interface, there's much more you can do to enhance its features. Here are some examples:

• Configure more Bedrock models to unlock diverse capabilities and customization, balancing functionality, performance, and cost

• Enable web search to allow LibreChat to search the internet and retrieve relevant information, enhancing conversations

• Build custom AI assistants using AI Agents and integrate with various built-in and MCP tools to elevate capabilities and user experience

As your LibreChat usage grows, it's imperative to align your architecture with the AWS Well-Architected Framework. Here are some examples to improve security and operational robustness:

• Streamline authentication by integrating with an Identity Provider (IDP)

• Enable caching, session storage, and horizontal scaling in LibreChat using Redis and compute auto-scaling

• Scale file storage with an Amazon S3 storage backend

Given the vast possibilities, I will start a new blog series about LibreChat and how to best use AWS services and best practices to enable these capabilities. If you enjoyed this post, stay tuned for new content at the Avangards Blog. Thanks so much for reading, and I hope you have fun chatting with LibreChat!

Introduction

As a Terraform advocate and an AWS consultant who builds many landing zones, AWS Control Tower has always been one of my favorite AWS services. Beyond its common use cases, such as account provisioning with Account Factory Customization (AFC) and Account Factory for Terraform (AFT), I am always on the lookout for opportunities to bring the two technologies closer together.

During landing zone design workshops, when walking customers through Control Tower controls, I often found myself unable to recommend proactive controls because many organizations prefer using Terraform over CloudFormation for infrastructure as code (IaC). To fully leverage everything Control Tower has to offer, wouldn’t it be nice if proactive controls worked with other IaC tools, including Terraform?

Through research, I learned that proactive controls are implemented as CloudFormation Hooks and can target resources created via the Cloud Control API. Having worked with the Terraform AWS Cloud Control (CC) Provider, I began to wonder whether proactive controls could evaluate Terraform resources created through this provider. This question became the experiment that is the subject of this blog post.

Let’s start with a quick explanation of what proactive controls are.

What Are AWS Control Tower Proactive Controls?

Proactive controls are pre-built compliance rules that evaluate AWS resources before deployment via CloudFormation stack operations, preventing non-compliant resources from being created or updated. AWS Control Tower provides more than 200 controls covering a wide range of AWS services and compliance frameworks.

An example of a proactive control is [CT.EC2.PR.7] Require an Amazon EBS volume resource to be encrypted at rest when defined by means of the AWS::EC2::Instance BlockDeviceMappings property or AWS::EC2::Volume resource type. As the name suggests, this control prevents an EC2 instance from being created or updated if it specifies an unencrypted EBS volume.

For the full list of proactive controls, you can either view them on the Control Catalog page in the AWS Control Tower console or refer to the Proactive control section in the AWS Control Tower Control Reference Guide.

Testing Proactive Controls with Terraform (Unsuccessfully)

Since proactive controls are implemented using CloudFormation Hooks, I initially assumed they would evaluate all Hook targets, particularly resources supported by the Cloud Control API. Because the Terraform AWS Cloud Control (CC) Provider is implemented using the Cloud Control API (as opposed to the standard AWS API used by the original Terraform AWS Provider), I expected proactive controls to apply there as well.

Although it is still uncommon for organizations to fully adopt the Terraform AWS CC Provider, I wanted to determine whether proactive controls could be used with Terraform to enforce compliance.

As a quick test, I used the following Terraform configuration to create an EC2 instance with an unencrypted EBS volume using the Terraform AWS CC Provider, expecting it to fail:

variable "subnet_id" {
    type = string
}

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-x86_64"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

# EC2 instance with unencrypted EBS volume
resource "awscc_ec2_instance" "this" {
  image_id      = data.aws_ami.al2023.id
  instance_type = "t3.micro"
  subnet_id     = var.subnet_id

  block_device_mappings = [
    {
      device_name = "/dev/xvda"
      ebs = {
        volume_size = 20
        volume_type = "gp3"
        encrypted   = false  # Explicitly unencrypted
        delete_on_termination = true
      }
    }
  ]

  tags = [
    {
      key   = "Name"
      value = "unencrypted-vol-test"
    }
  ]
}

However, terraform apply ran successfully and the EC2 instance was created:

$ terraform apply
data.aws_ami.al2023: Reading...
data.aws_ami.al2023: Read complete after 0s [id=ami-0f3caa1cf4417e51b]

Terraform used the selected providers to generate the following execution plan.       
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # awscc_ec2_instance.this will be created
  + resource "awscc_ec2_instance" "this" {
      + additional_info                      = (known after apply)
      + affinity                             = (known after apply)
      + availability_zone                    = (known after apply)
      + block_device_mappings                = [
          + {
              + device_name  = "/dev/xvda"
              + ebs          = {
                  + delete_on_termination = true
                  + encrypted             = false
                  + iops                  = (known after apply)
                  + kms_key_id            = (known after apply)
                  + snapshot_id           = (known after apply)
                  + volume_size           = 20
                  + volume_type           = "gp3"
                }
              + no_device    = (known after apply)
              + virtual_name = (known after apply)
            },
        ]
      + cpu_options                          = (known after apply)
      + credit_specification                 = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + elastic_gpu_specifications           = (known after apply)
      + elastic_inference_accelerators       = (known after apply)
      + enclave_options                      = (known after apply)
      + hibernation_options                  = (known after apply)
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + image_id                             = "ami-0f3caa1cf4417e51b"
      + instance_id                          = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_type                        = "t3.micro"
      + ipv_6_address_count                  = (known after apply)
      + ipv_6_addresses                      = (known after apply)
      + kernel_id                            = (known after apply)
      + key_name                             = (known after apply)
      + launch_template                      = (known after apply)
      + license_specifications               = (known after apply)
      + metadata_options                     = (known after apply)
      + monitoring                           = (known after apply)
      + network_interfaces                   = (known after apply)
      + placement_group_name                 = (known after apply)
      + private_dns_name                     = (known after apply)
      + private_dns_name_options             = (known after apply)
      + private_ip                           = (known after apply)
      + private_ip_address                   = (known after apply)
      + propagate_tags_to_volume_on_creation = (known after apply)
      + public_dns_name                      = (known after apply)
      + public_ip                            = (known after apply)
      + ramdisk_id                           = (known after apply)
      + security_group_ids                   = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = (known after apply)
      + ssm_associations                     = (known after apply)
      + state                                = (known after apply)
      + subnet_id                            = "subnet-0a0bb7e920672c803"
      + tags                                 = [
          + {
              + key   = "Name"
              + value = "unencrypted-vol-test"
            },
        ]
      + tenancy                              = (known after apply)
      + user_data                            = (known after apply)
      + volumes                              = (known after apply)
      + vpc_id                               = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

awscc_ec2_instance.this: Creating...
awscc_ec2_instance.this: Still creating... [00m10s elapsed]
awscc_ec2_instance.this: Creation complete after 16s [id=i-0963bfcf44274c8d9]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$

So how did the EC2 instance manage to be created?

Investigating Why Proactive Controls Don’t Work with Terraform

To investigate the issue, I examined the Hook in the CloudFormation console. Looking at the Hook named AWS::ControlTower::Hook, I noticed that its Targets field was set to None.

This seemed odd, as I expected at least one target to be listed. Upon reviewing the Hook details, I observed that the Hook targets included only CloudFormation resources, not the Cloud Control API:

Assuming the Hook details reflect the actual configuration, this implies that proactive controls validate only CloudFormation resources, not resources provisioned through the Cloud Control API (and therefore not those created via the Terraform AWS CC Provider). To confirm this behavior, I opened an AWS Support case.

The AWS support engineer explained that proactive controls are implemented using a special Hook type called Controls (Managed Hooks), which supports only CloudFormation resources as targets. To extend proactive controls to other targets, each control must be re-implemented as a custom Hook.

Although I submitted a feature request to the AWS Control Tower team to expand proactive controls to additional targets, I decided to proceed with a workaround, even if it required additional effort.

Replicating Proactive Controls to Target the Cloud Control API with Lambda Hooks

The AWS support engineer initially suggested re-implementing each proactive control using Lambda Hooks. While this approach would require significant effort, I was provided with the following Python code used for the CT.EC2.PR.7 control:

import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    '''
    CloudFormation Hook Handler for EBS Encryption Validation
    Validates that EC2 instances have encrypted EBS volumes
    '''

    logger.info(f"Received full event: {json.dumps(event, indent=2)}")

    # Extract request details from Cloud Control API event structure
    request_data = event.get('requestData', {})
    resource_type = request_data.get('targetName')
    target_model = request_data.get('targetModel', {})
    resource_properties = target_model.get('resourceProperties', {}) 

    logger.info(f"Extracted - Type: {resource_type}, Properties keys: {list(resource_properties.keys())}")

    # Only validate EC2 instances
    if not resource_type or resource_type != 'AWS::EC2::Instance':
        logger.info(f"Skipping validation - resource type '{resource_type}' is not EC2 Instance")
        return {
            'hookStatus': 'SUCCESS',
            'message': f'Resource type {resource_type} not applicable for this hook'
        }

    # Validation logic
    validation_result = validate_ebs_encryption(resource_properties)

    if validation_result['compliant']:
        return {
            'hookStatus': 'SUCCESS',
            'message': 'EC2 instance has encrypted EBS volumes'
        }
    else:
        return {
            'hookStatus': 'FAILED',
            'errorCode': 'NonCompliant',
            'message': validation_result['message']
        }

def validate_ebs_encryption(properties):
    '''
    Validates that all EBS volumes are encrypted
    '''

    # Check BlockDeviceMappings
    block_device_mappings = properties.get('BlockDeviceMappings', [])

    if not block_device_mappings:
        return {
            'compliant': False,
            'message': 'No BlockDeviceMappings specified. Ensure the AMI uses encrypted volumes or specify encrypted BlockDeviceMappings.'
        }

    # Validate each block device mapping
    for idx, mapping in enumerate(block_device_mappings):
        ebs = mapping.get('Ebs', {})

        if ebs:
            encrypted = ebs.get('Encrypted', False)

            if not encrypted:
                return {
                    'compliant': False,
                    'message': f'BlockDeviceMapping at index {idx} has an unencrypted EBS volume. Set Encrypted to true.'
                }

    return {
        'compliant': True,
        'message': 'All EBS volumes are encrypted'
    }

After creating the Lambda function with the provided code, I created a Lambda Hook as per screenshots below. The key configuration details were:

• Hook targets should include at least Cloud Control API . Since proactive controls already target CloudFormation resources, including them here is unnecessary.

• Actions should include Create and Update .

• Hook mode should be set to Fail .

• Target resources should include AWS::EC2::Instance and AWS::EC2::Volume . These resource types are specified in the control details within the AWS::ControlTower::Hook configuration.

After the Lambda Hook is created, when I reran terraform apply, and this time it failed as expected due to the Hook:

awscc_ec2_instance.this: Creating...
╷
│ Error: AWS SDK Go Service Operation Incomplete
│
│   with awscc_ec2_instance.this,
│   on main.tf line 21, in resource "awscc_ec2_instance" "this":
│   21: resource "awscc_ec2_instance" "this" {
│
│ Waiting for Cloud Control API service CreateResource operation completion returned: 
│ waiter state transitioned to FAILED. StatusMessage:
│ 149a3eef-eaba-4459-8ea7-1707b1183e11. Hook failures: HookName:
│ Private::Lambda::CTEC2PR7, HookArn:
│ arn:aws:cloudformation:us-east-1:2**********1:type/hook/Private-Lambda-CTEC2PR7/00000001/aws-hooks/AWS-Hooks-LambdaHook/00000001.00000024,
│ HookVersion: 00000025, Time: 2026-02-23T21:06:45Z, HookMessage: BlockDeviceMapping  
│ at index 0 has an unencrypted EBS volume. Set Encrypted to true.
╵

Even Better: Replicating Proactive Controls with Guard Hooks

While researching further, I noticed that the rule specifications for all proactive controls are published under Proactive controls in the AWS Control Tower Controls Reference Guide. This makes replication significantly easier.

According to the documentation, proactive controls are implemented using Guard Hooks powered by AWS CloudFormation Guard, a domain-specific language (DSL) for policy-as-code. For more context and development instructions of Guard Hook, refer to Writing AWS CloudFormation Guard rules in the AWS CloudFormation Guard User Guide.

For our purposes, the CT.EC2.PR.7 control specification already contains everything needed to create the Hook. For instance, the rule specification is as follows:

# ###################################
##       Rule Specification        ##
#####################################
# 
# Rule Identifier:
#   ec2_encrypted_volumes_check
# 
# Description:
#   Checks whether standalone Amazon EC2 EBS volumes and new EC2 EBS volumes created through EC2 instance
#   Block Device Mappings are encrypted at rest.
# 
# Reports on:
#   AWS::EC2::Instance, AWS::EC2::Volume
# 
# Evaluates:
#   CloudFormation, CloudFormation hook
# 
# Rule Parameters:
#   None
# 
# Scenarios:
#   Scenario: 1
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document does not contain any Amazon EC2 volume resources
#      Then: SKIP
#   Scenario: 2
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 instance resource
#       And: 'BlockDeviceMappings' has not been provided or has been provided as an empty list
#      Then: SKIP
#   Scenario: 3
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 instance resource
#       And: 'BlockDeviceMappings' has been provided as a non-empty list
#       And: 'Ebs' has been provided in a 'BlockDeviceMappings' configuration
#       And: 'Encrypted' has not been provided in the 'Ebs' configuration
#      Then: FAIL
#   Scenario: 4
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 instance resource
#       And: 'BlockDeviceMappings' has been provided as a non-empty list
#       And: 'Ebs' has been provided in a 'BlockDeviceMappings' configuration
#       And: 'Encrypted' has been provided in the 'Ebs' configuration and set to bool(false)
#      Then: FAIL
#   Scenario: 5
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 volume resource
#       And: 'Encrypted' on the EC2 volume has not been provided
#      Then: FAIL
#   Scenario: 6
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 volume resource
#       And: 'Encrypted' on the EC2 volume has been provided and is set to bool(false)
#      Then: FAIL
#   Scenario: 7
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 instance resource
#       And: 'BlockDeviceMappings' has been provided as a non-empty list
#       And: 'Ebs' has been provided in a 'BlockDeviceMappings' configuration
#       And: 'Encrypted' has been provided in the 'Ebs' configuration and set to bool(true)
#      Then: PASS
#   Scenario: 8
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 volume resource
#       And: 'Encrypted' on the EC2 volume has been provided and is set to bool(true)
#      Then: PASS

#
# Constants
#
let EC2_VOLUME_TYPE = "AWS::EC2::Volume"
let EC2_INSTANCE_TYPE = "AWS::EC2::Instance"
let INPUT_DOCUMENT = this

#
# Assignments
#
let ec2_volumes = Resources.*[ Type == %EC2_VOLUME_TYPE ]
let ec2_instances = Resources.*[ Type == %EC2_INSTANCE_TYPE ]

#
# Primary Rules
#
rule ec2_encrypted_volumes_check when is_cfn_template(%INPUT_DOCUMENT)
                                      %ec2_volumes not empty {
    check_volume(%ec2_volumes.Properties)
        <<
        [CT.EC2.PR.7]: Require that an Amazon EBS volume attached to an Amazon EC2 instance is encrypted at rest
        [FIX]: Set 'Encryption' to true on EC2 EBS Volumes.
        >>
}

rule ec2_encrypted_volumes_check when is_cfn_hook(%INPUT_DOCUMENT, %EC2_VOLUME_TYPE) {
    check_volume(%INPUT_DOCUMENT.%EC2_VOLUME_TYPE.resourceProperties)
        <<
        [CT.EC2.PR.7]: Require that an Amazon EBS volume attached to an Amazon EC2 instance is encrypted at rest
        [FIX]: Set 'Encryption' to true on EC2 EBS Volumes.
        >>
}

rule ec2_encrypted_volumes_check when is_cfn_template(%INPUT_DOCUMENT)
                                      %ec2_instances not empty {
    check_instance(%ec2_instances.Properties)
        <<
        [CT.EC2.PR.7]: Require that an Amazon EBS volume attached to an Amazon EC2 instance is encrypted at rest
        [FIX]: Set 'Encryption' to true on EC2 EBS Volumes.
        >>
}

rule ec2_encrypted_volumes_check when is_cfn_hook(%INPUT_DOCUMENT, %EC2_INSTANCE_TYPE) {
    check_instance(%INPUT_DOCUMENT.%EC2_INSTANCE_TYPE.resourceProperties)
        <<
        [CT.EC2.PR.7]: Require that an Amazon EBS volume attached to an Amazon EC2 instance is encrypted at rest
        [FIX]: Set 'Encryption' to true on EC2 EBS Volumes.
        >>
}

#
# Parameterized Rules
#

rule check_instance(ec2_instance) {
    %ec2_instance[
        filter_ec2_instance_block_device_mappings(this)
    ] {
        BlockDeviceMappings[
            Ebs exists
            Ebs is_struct
        ] {
            check_volume(Ebs)
        }
    }
}

rule check_volume(ec2_volume) {
    %ec2_volume {
        # Scenario 2
        Encrypted exists
        # Scenarios 3 and 4
        Encrypted == true
    }
}

rule filter_ec2_instance_block_device_mappings(ec2_instance) {
    %ec2_instance {
        BlockDeviceMappings exists
        BlockDeviceMappings is_list
        BlockDeviceMappings not empty
    }
}

#
# Utility Rules
#
rule is_cfn_template(doc) {
    %doc {
        AWSTemplateFormatVersion exists  or
        Resources exists
    }
}

rule is_cfn_hook(doc, RESOURCE_TYPE) {
    %doc.%RESOURCE_TYPE.resourceProperties exists
}

To implement this, I first deleted the previous Lambda Hook and its associated IAM role and policy to avoid redundant checks. Then following the instructions to activate a Guard Hook, I created an S3 bucket and uploaded the Guard rule as CT.EC2.PR.7.guard, and created the Guard Hook as per the screenshots below. The key configuration details were:

• Hook targets should include at least Cloud Control API . Since proactive controls already target CloudFormation resources, including them here is unnecessary.

• Actions should include Create and Update .

• Hook mode should be set to Fail .

• Target resources should include AWS::EC2::Instance and AWS::EC2::Volume . These resource types are specified in the control details within the AWS::ControlTower::Hook configuration.

After the Guard Hook is created, rerunning terraform apply resulted in a failure as expected:

awscc_ec2_instance.this: Creating...
╷
│ Error: AWS SDK Go Service Operation Incomplete
│
│   with awscc_ec2_instance.this,
│   on main.tf line 21, in resource "awscc_ec2_instance" "this":
│   21: resource "awscc_ec2_instance" "this" {
│
│ Waiting for Cloud Control API service CreateResource operation completion returned: 
│ waiter state transitioned to FAILED. StatusMessage:
│ 0708b661-d689-4451-b05f-28c8429f3836. Hook failures: HookName:
│ Private::Guard::CTEC2PR7, HookArn:
│ arn:aws:cloudformation:us-east-1:2**********1:type/hook/Private-Guard-CTEC2PR7/00000002/aws-hooks/AWS-Hooks-GuardHook/00000001.00000071,
│ HookVersion: 00000072, Time: 2026-02-24T02:28:27Z, HookMessage: Template failed     
│ validation, the following rule(s) failed: ec2_encrypted_volumes_check.
╵

This confirms that Guard Hooks provide a clean and scalable way to extend proactive controls to Terraform via the Cloud Control API.

Next Steps

Now that we have a viable approach for replicating proactive controls using Guard Hooks, the next logical step is automation at scale.

I submitted another feature request to the Control Tower team to publish proactive control Guard rules in a GitHub repository, similar to the AWS Guard Rules Registry. After cross-checking the rules in that repository against the proactive control documentation, I found they differ.

As a workaround, I could develop a scraper to extract rule definitions directly from the documentation and publish them into my own GitHub repository.

From there, I could identify an appropriate trigger, such as a CloudTrail event for updates to the AWS::ControlTower::Hook, to invoke a Lambda function that automatically manages Guard Hook replication based on enabled proactive controls.

This could make for an interesting project, and perhaps a future blog post, demonstrating the capabilities of AI coding assistants such as Kiro.

Summary

In this blog post, I explored whether AWS Control Tower proactive controls can apply to resources created with Terraform. After a failed attempt, I looked for a workaround, which ultimately took the form of replicating the CloudFormation Guard rules that power the proactive controls. Hopefully, AWS will eventually implement the feature request to extend proactive controls to cover the Cloud Control API as well. In the meantime, we now have a viable and potentially automatable approach to replicate them.

If you enjoyed this blog post and the topic it covers, be sure to check out the Avangards Blog for more content. Thanks for reading!

Over the past year, I have worked extensively with Terraform and AWS in building turnkey infrastructure solutions and contributing to the Terraform AWS Provider as a HashiCorp Core Contributor. As a formal validation of my Terraform expertise, I was motivated to take the Terraform Authoring and Operations Professional Exam. I recently passed the exam, and it was such a fun and unique experience that I decided to share my study tips with the community in this blog post to raise awareness and spark interest. Let’s first take a look at what this exam is all about.

About the Terraform Authoring and Operations Professional Exam

The Terraform Authoring and Operations Professional certification is not an entry-level exam. It’s intended for engineers who already have hands-on experience using Terraform in production and maintaining infrastructure over time. Passing the exam demonstrates strong skills in writing Terraform modules and managing Terraform operations in an organizational setting. The objectives of the exam include:

1. Manage resource lifecycle - Covers the end-to-end Terraform workflow for creating, updating, destroying, and managing infrastructure state using core CLI commands.

2. Develop and troubleshoot dynamic configuration - Focuses on writing flexible, reusable Terraform configuration using HCL features, functions, variables, and best practices for sensitive data.

3. Develop collaborative Terraform workflows - Addresses how Terraform is used in team and automated environments, including versioning, remote state, automation, and data sharing.

4. Create, maintain, and use Terraform modules - Examines how to design, consume, refactor, and version Terraform modules to enable reuse and maintainability.

5. Configure and use Terraform providers - Covers provider architecture, configuration, authentication, versioning, and troubleshooting provider-related issues.

6. Collaborate on infrastructure as code using HCP Terraform - Focuses on operating Terraform at scale with HCP Terraform, including runs, workspaces, credential management, and governance controls.

The exam is primarily lab-based, with scenarios in which you modify configuration and provision and manage infrastructure in a virtual exam environment. It also includes a small multiple-choice section that validates your knowledge of HCP Terraform and related topics. This is an online, proctored exam that is four hours in length, with an optional 15-minute break. It costs $295 USD plus tax; however, it does include a free retake.

If the content and format of this exam intrigue you enough to take the plunge, here are some practical tips that may help with your preparation.

Tip 1: Use the Official Prep Tutorial to Guide Your Study

The Terraform Professional certification exam prep guide was my go-to resource for studying. Both the learning path and the exam content list provide an exhaustive and structured set of topics to master, including recommended tutorials. Based on these resources, I created my own study plan and a list of things to test in my lab environment. Much of my study involved writing Terraform configuration, running commands, and observing how the system behaves.

While there aren’t many third-party study materials available for the exam, I did come across the Terraform Authoring and Operations Professional Study Guide, which was mentioned in this Reddit thread I found during my research. I read a free chapter of the book and found it to be quite well written, although I personally have enough Terraform and AWS experience that I could do without it.

Tip 2: Practical Experience with Terraform and AWS Is a Must

Unlike the HashiCorp Certified: Terraform Associate Exam and many cloud provider exams, passing the Terraform Authoring and Operations Professional Exam requires extensive hands-on experience with Terraform. Learning from basic tutorials alone will not help much, and it is futile to try studying and memorizing your way to a passing grade for this exam.

Although the exam details do not list AWS experience as a prerequisite, you must know how to deploy resources from core AWS services, such as Amazon EC2, Amazon VPC, and AWS IAM, to complete the lab-based scenarios efficiently. The list of AWS resources to review in the exam content outline is a good indicator of what you need to know and brush up on before the exam.

The required Terraform knowledge is also advanced, as you will need to employ dynamic configuration (think functions, modules, and meta-arguments) and state manipulation to complete the lab-based scenarios. Even in my professional services role with daily interaction with AWS and Terraform, I rarely had a need to work so extensively with states and advanced Terraform features. Fortunately, I explicitly practiced based on cues from Reddit and the study guide, which left me well prepared. Make sure you go through tutorials on these advanced topics and try them out in a sandbox environment.

Additionally, your Terraform CLI experience likely amounts to running terraform init, terraform plan and terraform apply even if you work with Terraform regularly. You should therefore review and experiment with the full list of CLI commands and their various options. I, for one, learned about an experimental option for a particular CLI command that proved helpful during the exam.

Tip 3: Know Your Way Around the Exam Environment

The exam is conducted within a Guacamole-powered Linux virtual desktop environment that is accessed via your web browser. The video walkthrough on the exam orientation page should give you an idea of what the exam environment looks like. The exam is primarily delivered through a local web page in the preinstalled Mozilla Firefox web browser. It provides the exam instructions and lab scenarios, a section for completing the multiple-choice questions, and links to whitelisted resources such as Terraform and provider documentation.

For the lab-based scenarios, the preinstalled Visual Studio Code is your main interface. The IDE has some useful extensions, such as the HashiCorp Terraform extension, preinstalled, giving you access to features like IntelliSense. However, the IDE may not be fully configured with all the quality-of-life features that you’d expect (such as format on save), so you will either have to change the settings yourself or bear with the minor inconveniences. One thing I found to be extremely helpful is the terminal in VS Code. Even though it is not mentioned in the exam instructions, opening the terminal in VS Code provides you with a prompt to navigate to the directory of each lab scenario. This made it more efficient for me to open files and run commands, without having to browse and open files from the desktop. If you are not already using VS Code for Terraform development, be sure to familiarize yourself with it and the Terraform extension before the exam.

One notable annoyance with the exam environment is that if you use your mouse side buttons to navigate between documentation pages in Firefox, the shortcut is instead recognized by your main web browser and leads you out of the exam session. You’d then have to ask the proctor to let you back in. I have also seen other exam takers reporting similar issues with keyboard shortcuts. It was difficult for me to break this habit during the exam, so I accidentally exited the exam environment several times, much to the proctor’s dismay, which he made clear in the chat. After the exam, I reported the issue and was contacted by someone from IBM, so I hope this can be fixed soon.

Otherwise, it wasn’t too difficult to adapt to the copy-and-paste shortcuts, and overall latency was acceptable.

Tip 4: Learn to Navigate Terraform and AWS Documentation Efficiently

During the exam, you can access the Terraform documentation, the Terraform Registry, the AWS provider documentation, and some AWS documentation in Firefox. However, you will not be able to search, so you will need to rely on navigating the documentation.

You should know where to find CLI command references for available options, as well as configuration construct references such as functions and built-in resources. You can navigate to these topics from the main documentation page via Terraform CLI and Configuration Language in the left-hand menu, respectively:

As I am fairly proficient with Terraform and practiced before the exam, I only referred to the documentation a couple of times. Where it helped me most was in validating my answers to multiple-choice questions related to platform and enterprise features that I am not experienced with.

Navigating the AWS provider documentation should be straightforward, but you should know where to find information about provider configuration and in-scope resources for the exam. As I work with AWS almost on a daily basis, I didn’t need to refer to AWS documentation at all. Overall, you should familiarize yourself with navigating the documentation without relying on search.

Tip 5: Manage Your Time Effectively During the Exam

The Terraform Authoring and Operations Professional Exam is 4 hours in duration with an optional 15-minute break, making it the longest IT exam I’ve taken to date. That said, you will likely need the entire allotted time to complete the exam. The majority of your time will be spent on lab-based scenarios that are aligned with the exam objectives by theme. Be sure to review each lab scenario description thoroughly and complete them “to spec”. The code doesn’t have to be pretty - it just needs to work correctly.

It may also be wise to time-box each scenario, perhaps to 45 minutes, so that you at least have an opportunity to attempt every question. I recall getting stuck on a very specific provider-related issue in the second scenario and spending about an hour on it before deciding to park it and move on. I was fortunately able to complete the other scenarios fairly quickly, which afforded me about 30 minutes to figure out the earlier scenario and double-check my answers to the multiple-choice questions. I completed the exam right at the time limit with confidence and received a passing notification an hour or two later.

Summary

The Terraform Authoring and Operations Professional Exam is a challenging, hands-on exam that really tests how well you know Terraform in real-world scenarios. In this blog post, I shared what the exam is like and the study strategies that worked for me - from focusing on hands-on practice and advanced Terraform features to getting comfortable with the exam environment and managing your time effectively.

If you’re thinking about taking the exam, I hope these tips help you prepare with more confidence. And if you found this useful, feel free to check out my other blog posts in the Avangards Blog, where I share more lessons learned from working with Terraform, AWS, and infrastructure as code. Good luck and happy “Terraforming”!

In the earlier blog post Building a Data Ingestion Solution for Amazon Bedrock Knowledge Bases, we developed a data ingestion solution that includes job completion notifications with a status pull mechanism which wasn’t as efficient as it could be. Since then, we examined Knowledge Bases logging which publishes ingestion job log events to CloudWatch Logs, which opens up a new opportunity for a better design with a status push mechanism based on subscription filters. In this blog post, we will examine how to update the original solution with the new design.

Updated Design Overview

The overall design of the updated solution is depicted in the following diagram:

The updated solution works as follows:

1. Logging is configured for the Bedrock Knowledge Base to deliver logs to CloudWatch Logs. A subscription filter is created in the associated log group to filter ingestion job status change events that correspond to an end state and send log events to a Lambda function.

2. A Lambda function, triggered by an EventBridge schedule rule, periodically starts an ingestion (a.k.a. sync) job for each specified knowledge base and data source. Note that the SQS queue is removed as it is no longer necessary.

3. Another Lambda function serves as the destination for the subscription filter. For each event message that is extracted from the log events, the function uses the job ID information to get details about the ingestion job. A notification is sent to one of the two SNS topics depending on whether the job is successful or failed.

Updating the Components

As the SQS queue is not required, the only change to the Lambda function that starts the ingestion job is a minor cleanup. The updated Lambda function code is as follows:

import boto3
import json
from botocore.exceptions import ClientError

bedrock_agent = boto3.client('bedrock-agent')
ssm = boto3.client('ssm')


def lambda_handler(event, context):
    try:
        # Retrieve the JSON config from Parameter Store
        response = ssm.get_parameter(Name='/start-kb-ingestion-jobs/config-json')
        config_json = response['Parameter']['Value']
        config = json.loads(config_json)

        for record in config:
            knowledge_base_id = record.get('knowledge_base_id')
            for data_source_id in record.get('data_source_ids'):
                # Start the ingestion job
                print(f'Starting ingestion job for data source {data_source_id} of knowledge base {knowledge_base_id}')
                response = bedrock_agent.start_ingestion_job(
                    knowledgeBaseId=knowledge_base_id,
                    dataSourceId=data_source_id
                )
        return {
            'statusCode': 200,
            'body': 'Success'
        }
    except ClientError as e:
        return {
            'statusCode': 500,
            'body': f'Client error: {str(e)}'
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': f'Unexpected error: {str(e)}'
        }

Meanwhile, updating the component that checks ingestion job statuses is slightly more complex. First, we need to update the check-kb-job-statuses Lambda function to be a subscription filter target. As described in the Log group-level subscription filters page of the CloudWatch Logs user guide, the log data received by the function is compressed, Base64-encoded, and batched. I was able to easily find this StackOverflow question which has the exact code we need in the first answer.

Next, we need to know what a relevant log event looks like. The examples of knowledge base logs in the AWS documentation provide the general format for an ingestion job event, however it is preferable to look at an actual log event. Here’s one that captures a job completion event for a successful job:

{
    "event_timestamp": 1740895462316,
    "event": {
        "ingestion_job_id": "W0V45LVZY6",
        "data_source_id": "ATUWOVZJOD",
        "ingestion_job_status": "COMPLETE",
        "knowledge_base_arn": "arn:aws:bedrock:us-east-1:<redacted>:knowledge-base/R1K1UIZKKQ",
        "resource_statistics": {
            "number_of_resources_updated": 366,
            "number_of_resources_ingested": 0,
            "number_of_resources_deleted": 0,
            "number_of_resources_with_metadata_updated": 0,
            "number_of_resources_failed": 15
        }
    },
    "event_version": "1.0",
    "event_type": "StartIngestionJob.StatusChanged",
    "level": "INFO"
}

The Lambda function must extract the following details from the log event:

1. The knowledge base ID, which can be extracted from the value of the event.knowledge_base_arn, specifically after the /.

2. The data source ID, which is the value of the event.data_source_id field.

3. The ingestion job ID, which is the value of the event.ingestion_job_id field.

Although we are able to extract all of the information required for notification, the log event does not contain the verbose content ingestion failures which we get from the response of the GetIngestionJob API action. Although this approach is slightly less efficient, we will still call the API for completeness. The resulting Lambda function should look like this:

import base64
import boto3
import gzip
import json
from botocore.exceptions import ClientError

bedrock_agent = boto3.client('bedrock-agent')
ssm = boto3.client('ssm')
sns = boto3.client('sns')


def get_ssm_parameter(name):
    response = ssm.get_parameter(Name=name, WithDecryption=True)
    return response['Parameter']['Value']


def get_ingestion_job(knowledge_base_id, data_source_id, ingestion_job_id):
    response = bedrock_agent.get_ingestion_job(
        knowledgeBaseId=knowledge_base_id,
        dataSourceId=data_source_id,
        ingestionJobId=ingestion_job_id
    )
    return response['ingestionJob']


def lambda_handler(event, context):
    try:
        success_sns_topic_arn = get_ssm_parameter('/check-kb-ingestion-job-statuses/success-sns-topic-arn')
        failure_sns_topic_arn = get_ssm_parameter('/check-kb-ingestion-job-statuses/failure-sns-topic-arn')

        encoded_zipped_data = event['awslogs']['data']
        zipped_data = base64.b64decode(encoded_zipped_data)
        data = json.loads(gzip.decompress(zipped_data))
        log_events = data['logEvents']
        for log_event in log_events:
            message = json.loads(log_event['message'])
            knowledge_base_arn = message['event']['knowledge_base_arn']
            knowledge_base_id = knowledge_base_arn.split('/')[-1]
            data_source_id = message['event']['data_source_id']
            ingestion_job_id = message['event']['ingestion_job_id']

            print(
                f'Checking ingestion job status for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id}')
            ingestion_job = get_ingestion_job(knowledge_base_id, data_source_id, ingestion_job_id)
            print(
                f'Ingestion job summary: \n\n{json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)}')
            job_status = ingestion_job['status']
            if job_status == 'COMPLETE':
                sns.publish(
                    TopicArn=success_sns_topic_arn,
                    Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} Completed',
                    Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                )
            elif job_status == 'FAILED':
                sns.publish(
                    TopicArn=failure_sns_topic_arn,
                    Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} FAILED',
                    Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                )
            elif job_status == 'STOPPED':
                sns.publish(
                    TopicArn=failure_sns_topic_arn,
                    Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} STOPPED',
                    Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                )
        return {
            'statusCode': 200,
            'body': 'Success'
        }
    except ClientError as e:
        return {
            'statusCode': 500,
            'body': f'Client error: {str(e)}'
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': f'Unexpected error: {str(e)}'
        }

Lastly, we need to create a subscription filter in the log group that acts as the log delivery destination of the knowledge base. Since we are only interested in log events for ingestion job completion, we need to define an appropriate subscription filter pattern. There are two fields which we need for this purpose:

1. The event_type field with the value StartIngestionJob.StatusChanged.

2. The event.ingestion_job_status field with the value matching one of COMPLETE, FAILED, CRAWLING_COMPLETED, as described in the data ingestion job log example.

Based on some testing, a CRAWLING_COMPLETED event does not indicate a full completion of an ingestion job. A COMPLETE (and presumably FAILED) event is always sent upon job completion. So we can use COMPLETE and FAILED for the filter. Furthermore, stopping a job does not generate an event and there is no status value for it. This seems like a miss on AWS’ part, so I’ll open an AWS support case for it. For now, we will still add STOPPED to the filter for the sake of completeness.

Referring to subscription filter pattern for JSON log events, we can define our compound expression that checks for the event type and ingestion job status as follows:

{$.event_type = "StartIngestionJob.StatusChanged" && ($.event.ingestion_job_status = "COMPLETE" || $.event.ingestion_job_status = "FAILED" || $.event.ingestion_job_status = "STOPPED")}

We can first test the pattern in the AWS Management Console's subscription filter creation dialog without creating the filter. Later, we will implement it using Terraform. Here is a screenshot of what the dialog looks like:

In this example, the subscription filter is created in the log group as evident by the standard naming pattern. Knowledge base logs are written to the log stream bedrock/knowledgebaselogs, so we need to select that. Using the Test pattern button, we can see one filtered entry in the test results among 50 log events. The log events were generated from a single ingestion job, and the other events are either resource change events or non-related status change events.

Updating the Terraform Configuration

The following changes are required to the original solution’s Terraform configuration to support the new design:

• Remove the SQS queue, the associated IAM permissions, and the SSM parameters.

• Update the Lambda permission for the check-kb-intgestion-job-statuses to allow trigger from CloudWatch logs via the log group to which the Bedrock knowledge base writes its application logs.

Lastly, we need a new resource for the subscription filter as follows:

resource "aws_cloudwatch_log_subscription_filter" "check_kb_ingestion_job_statuses" {
  name            = "check-kb-ingestion-job-statuses"
  log_group_name  = var.kb_app_log_group_name
  filter_pattern  = "{$.event_type = \"StartIngestionJob.StatusChanged\" && ($.event.ingestion_job_status = \"COMPLETE\" || $.event.ingestion_job_status = \"FAILED\" || $.event.ingestion_job_status = \"STOPPED\")}"
  destination_arn = aws_lambda_function.check_kb_ingestion_job_statuses.arn
  depends_on      = [aws_lambda_permission.check_kb_ingestion_job_statuses]
}

Note that the log group name is provided as a variable. The log group name should follow the default format provided by AWS, which is /aws/vendedlogs/bedrock/knowledge-base/APPLICATION_LOGS/<KB_ID>, where <KB_ID> is the Bedrock knowledge base ID.

Deploying and Testing the Solution

To deploy and test the solution, you need a knowledge base with at least one data source that has content to ingest either in an S3 bucket or a crawlable website. You can set this up in the Bedrock console using the vector database quick start options. Alternatively, deploy a sample knowledge base using the Terraform configuration from my blog post How To Manage an Amazon Bedrock Knowledge Base Using Terraform. This configuration is also available in the same GitHub repository under the 2_knowledge_base directory.

Additionally, you must also change the knowledge base’s logging configuration to deliver application logs to CloudWatch Logs. You can enable it either manually following the AWS documentation or using the Terraform configuration from my previous blog post Enabling Logging for Amazon Bedrock Knowledge Bases using Terraform. This configuration is also available in the same GitHub repository under the 4_kb_logging directory.

With the prerequisites in place, deploy the solution as follows:

1. From the root of the cloned GitHub repository, navigate to 5_kb_data_ingestion_via_logs.

2. Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.

   • By default, the start-kb-ingestion-jobs Lambda function runs daily at 0:00 UTC.

3. Configure your AWS credentials.

4. Run terraform init and terraform apply -var-file terraform.tfvars.

Once deployed, test the solution by adding an email subscription to the SNS topics check-kb-ingestion-job-statuses-success and check-kb-ingestion-job-statuses-failure for your e-mail address so that you can receive email notifications. Confirm your subscriptions using the link in the verification emails.

Next, manually invoke the start-kb-ingestion-jobs Lambda function in the Lambda console.

As the ingestion jobs run and complete, logs are written to CloudWatch Logs and pass through the subscription filter. The status change events should be filtered and sent to the Lambda function for notification, ultimately leading to the emails you’ll receive. Here’s an example:

Once you've verified the solution works, remove the SNS subscription and replace it with those that better fit your needs. If you don’t plan to keep the knowledge base, delete it along with the vector store (for example, the OSS index) to avoid unnecessary costs.

Summary

In this blog post, we improved the original Bedrock Knowledge Bases data ingestion solution with push-based notification using CloudWatch features. This is likely more efficient than a scheduled pull-based mechanism and allows us to leverage a Lambda subscription filter.

That being said, the ideal solution would be an EventBridge rule to react to native ingestion job events from Bedrock. The Bedrock service unfortunately does not publish such events today, but I’ve made a feature request via an AWS support case. Hopefully this will be supported soon and we can evolve our data ingestion solution further.

I hope you find this blog post helpful and engaging. Please feel free to check out my other blog posts in the Avangards Blog. Take care and happy learning!

In the recent blog post Building a Data Ingestion Solution for Amazon Bedrock Knowledge Bases, we created a data ingestion solution that includes job completion notifications with a status pull mechanism. Not satisfied with how frequently the Lambda function must run to check job statuses, I looked into whether a push mechanism is available.

From my research, I found that Bedrock Knowledge Bases supports observability logs and that it logs events related to content ingestion. With support for log delivery to CloudWatch Logs, it unlocks the possibility of using a subscription filter to push ingestion job completion log events. Consequently, I dedicated this blog post to reviewing this feature and determining how to enable it efficiently using Terraform.

With this context, let’s first look at how CloudWatch log delivery works in general and how it applies to Bedrock Knowledge Bases.

Creating a Delivery Source for the Knowledge Base

Bedrock Knowledge Bases is one of the AWS services that uses the log delivery feature in CloudWatch Logs to write vended logs. This is a framework that provides a standard interface to configure logging, which typically involves a delivery source, a delivery destination, and a delivery that enables logging by linking the two.

As per Monitor knowledge bases using CloudWatch Logs, Bedrock Knowledge Bases currently only support application logs. Thus, we can create the delivery source in Terraform using the aws_cloudwatch_log_delivery_source resource as follows:

resource "aws_cloudwatch_log_delivery_source" "kb_logs" {
  count        = var.enable_kb_log_delivery_cloudwatch_logs || var.enable_kb_log_delivery_s3 || var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name         = "bedrock-kb-${var.kb_id}"
  log_type     = "APPLICATION_LOGS"
  resource_arn = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:knowledge-base/${var.kb_id}"
}

Notice that there is a condition to create the resource only if one of the log delivery options is enabled using variables, which will also be used in the destination-specific configurations that are explained in subsequent sections. This makes the configuration more generic and pluggable to your Terraform configuration that manages your Bedrock Agents and Knowledge Bases.

Sending Logs to CloudWatch Logs

To send logs to CloudWatch Logs, we need to create a log group and configure it as destination for delivery. Creating a log group is simple enough using the aws_cloudwatch_log_group resource. The log group name should follow the default format provided by AWS, which is /aws/vendedlogs/bedrock/knowledge-base/APPLICATION_LOGS/<KB_ID>, where <KB_ID> is the Bedrock knowledge base ID.

Using the log group for log delivery requires a log group resource policy. As per the AWS documentation, CloudWatch Logs can automatically add the appropriate policy if the log group does not have a resource policy, and the user setting up the logging has the appropriate permissions. Although, for the sake of completeness, we should manually create the resource policy as described in the aforementioned documentation using the aws_cloudwatch_log_resource_policy resource.

Lastly, we need to create a delivery destination for it using the aws_cloudwatch_log_delivery_destination resource, and then establish the delivery from the source (i.e., the knowledge base) to the destination (i.e., the log group) using the aws_cloudwatch_log_delivery resource. The resulting Terraform configuration should look like the following:

resource "aws_cloudwatch_log_group" "kb_logs" {
  count = var.enable_kb_log_delivery_cloudwatch_logs ? 1 : 0
  name  = "/aws/vendedlogs/bedrock/knowledge-base/APPLICATION_LOGS/${var.kb_id}"
}

resource "aws_cloudwatch_log_resource_policy" "kb_logs" {
  count       = var.enable_kb_log_delivery_cloudwatch_logs ? 1 : 0
  policy_name = "bedrock-kb-${var.kb_id}-policy"
  policy_document = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AWSLogDeliveryWrite20150319"
        Effect = "Allow"
        Principal = {
          Service = ["delivery.logs.amazonaws.com"]
        }
        Action = [
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = ["${aws_cloudwatch_log_group.kb_logs[0].arn}:log-stream:*"]
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = ["${local.account_id}"]
          },
          ArnLike = {
            "aws:SourceArn" = ["arn:${local.partition}:logs:${local.region}:${local.account_id}:*"]
          }
        }
      }
    ]
  })
}

resource "aws_cloudwatch_log_delivery_destination" "kb_logs_cloudwatch_logs" {
  count = var.enable_kb_log_delivery_cloudwatch_logs ? 1 : 0
  name  = "bedrock-kb-${var.kb_id}-cloudwatch-logs"
  delivery_destination_configuration {
    destination_resource_arn = aws_cloudwatch_log_group.kb_logs[0].arn
  }
  depends_on = [aws_cloudwatch_log_resource_policy.kb_logs]
}

resource "aws_cloudwatch_log_delivery" "kb_logs_cloudwatch_logs" {
  count                    = var.enable_kb_log_delivery_cloudwatch_logs ? 1 : 0
  delivery_destination_arn = aws_cloudwatch_log_delivery_destination.kb_logs_cloudwatch_logs[0].arn
  delivery_source_name     = aws_cloudwatch_log_delivery_source.kb_logs[0].name
}

Sending Logs to S3

The process to enable S3 as a delivery destination follows a similar pattern. The first step is to create the S3 bucket using the aws_s3_bucket resource with a bucket policy that provides the appropriate permissions for log delivery as described in the AWS documentation. Note that if you are using SSE-KMS for server-side encryption, you’ll also add the appropriate permissions to the key policy for the CMK. For completeness, we also choose not to rely on CloudWatch Logs to set the bucket policy and instead use the aws_s3_bucket_policy resource to manage it.

We also need to create a delivery destination for it using the aws_cloudwatch_log_delivery_destination resource, then establish the delivery from the source (i.e. the knowledge base) and the destination (i.e. the S3 bucket) using the aws_cloudwatch_log_delivery resource. Note that updating multiple aws_cloudwatch_logs_delivery resources in parallel will cause concurrency issues, so we must ensure that they are created sequentially using depends_on meta-argument. In this case, the delivery resource for S3 depends on that of CloudWatch Logs.

The resulting Terraform configuration should look like the following:

resource "aws_s3_bucket" "kb_logs_s3" {
  count         = var.enable_kb_log_delivery_s3 ? 1 : 0
  bucket        = "bedrock-kb-logs-${lower(var.kb_id)}-${local.region_short}-${local.account_id}"
  force_destroy = true
}

resource "aws_s3_bucket_policy" "kb_logs_s3" {
  count  = var.enable_kb_log_delivery_s3 ? 1 : 0
  bucket = aws_s3_bucket.kb_logs_s3[0].id
  policy = jsonencode({
    Version = "2012-10-17"
    Id      = "AWSLogDeliveryWrite20150319"
    "Statement" : [
      {
        Sid    = "AWSLogDeliveryWrite171157658"
        Effect = "Allow"
        Principal = {
          Service = "delivery.logs.amazonaws.com"
        }
        Action   = "s3:PutObject"
        Resource = "${aws_s3_bucket.kb_logs_s3[0].arn}/AWSLogs/${local.account_id}/bedrock/knowledgebases/*"
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = "${local.account_id}"
            "s3:x-amz-acl"      = "bucket-owner-full-control"
          }
          ArnLike = {
            "aws:SourceArn" = "${aws_cloudwatch_log_delivery_source.kb_logs[0].arn}"
          }
        }
      }
    ]
  })
}

resource "aws_cloudwatch_log_delivery_destination" "kb_logs_s3" {
  count = var.enable_kb_log_delivery_s3 ? 1 : 0
  name  = "bedrock-kb-${var.kb_id}-s3"
  delivery_destination_configuration {
    destination_resource_arn = aws_s3_bucket.kb_logs_s3[0].arn
  }
  depends_on = [aws_s3_bucket_policy.kb_logs_s3[0]]
}

resource "aws_cloudwatch_log_delivery" "kb_logs_s3" {
  count                    = var.enable_kb_log_delivery_s3 ? 1 : 0
  delivery_destination_arn = aws_cloudwatch_log_delivery_destination.kb_logs_s3[0].arn
  delivery_source_name     = aws_cloudwatch_log_delivery_source.kb_logs[0].name
  depends_on               = [aws_cloudwatch_log_delivery.kb_logs_cloudwatch_logs]
}

Sending Logs to Data Firehose

Sending logs to Data Firehose is slightly more involved because of the Firehose delivery stream’s configuration. Since this blog post does not focus on the downstream destination at the Firehose level, we will use an S3 bucket with basic configuration. To set up a Firehose delivery stream, we first need to create an IAM role that the delivery stream uses to send data to its destination (that is, the S3 bucket). Controlling access with Amazon Data Firehose provides IAM policy examples for different configuration, including one for an S3 destination. To create the Firehose delivery stream, we use the aws_kinesis_firehose_delivery_stream resource. One thing to know is that the Firehose delivery stream needs to have the tag LogDeliveryEnabled set to true, which the service-linked role that CloudWatch Logs creates uses to write to Firehose delivery streams.

We also need to create a delivery destination for it using the aws_cloudwatch_log_delivery_destination resource, then establish the delivery from the source (i.e. the knowledge base) and the destination (i.e. the S3 bucket) using the aws_cloudwatch_log_delivery resource. To ensure that the delivery resources are created sequentially so to avoid concurrent modification issues, this delivery resource depends on that of S3.

The resulting Terraform configuration should look like the following:

resource "aws_s3_bucket" "kb_logs_data_firehose" {
  count         = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  bucket        = "bedrock-kb-logs-data-firehose-${lower(var.kb_id)}-${local.region_short}-${local.account_id}"
  force_destroy = true
}

resource "aws_iam_role" "kb_logs_data_firehose" {
  count = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name  = "S3RoleForDataFirehose-bedrock-kb-logs-${var.kb_id}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "firehose.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "sts:ExternalId" = "${local.account_id}"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "kb_logs_data_firehose" {
  count = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name  = "S3PolicyForDataFirehose-bedrock-kb-logs-${var.kb_id}"
  role  = aws_iam_role.kb_logs_data_firehose[0].name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "s3:AbortMultipartUpload",
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
          "s3:ListBucketMultipartUploads",
          "s3:PutObject"
        ]
        Effect = "Allow"
        Resource = [
          aws_s3_bucket.kb_logs_data_firehose[0].arn,
          "${aws_s3_bucket.kb_logs_data_firehose[0].arn}/*"
        ]
      }
    ]
  })
}

resource "aws_kinesis_firehose_delivery_stream" "kb_logs" {
  count       = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name        = "bedrock-kb-logs-${var.kb_id}"
  destination = "extended_s3"
  extended_s3_configuration {
    role_arn   = aws_iam_role.kb_logs_data_firehose[0].arn
    bucket_arn = aws_s3_bucket.kb_logs_data_firehose[0].arn
  }
  tags = {
    "LogDeliveryEnabled" = "true"
  }
  depends_on = [aws_iam_role_policy.kb_logs_data_firehose]
}

resource "aws_cloudwatch_log_delivery_destination" "kb_logs_data_firehose" {
  count = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name  = "bedrock-kb-${var.kb_id}-data-firehose"
  delivery_destination_configuration {
    destination_resource_arn = aws_kinesis_firehose_delivery_stream.kb_logs[0].arn
  }
}

resource "aws_cloudwatch_log_delivery" "kb_logs_data_firehose" {
  count                    = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  delivery_destination_arn = aws_cloudwatch_log_delivery_destination.kb_logs_data_firehose[0].arn
  delivery_source_name     = aws_cloudwatch_log_delivery_source.kb_logs[0].name
  depends_on               = [aws_cloudwatch_log_delivery.kb_logs_s3]
}

Testing the Configuration

To deploy and test the configuration, you need a knowledge base with at least one data source that has content to ingest either in an S3 bucket or a crawlable website. You can set this up in the Bedrock console using the vector database quick start options. Alternatively, deploy a sample knowledge base using the Terraform configuration from my blog post How To Manage an Amazon Bedrock Knowledge Base Using Terraform. This configuration is also available in the same GitHub repository under the 2_knowledge_base directory.

With the prerequisites in place, deploy the solution as follows:

1. From the root of the cloned GitHub repository, navigate to 4_kb_logging.

2. Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.

   • All log delivery destinations are enabled is enabled in terraform.tfvars.example. However, only delivery to CloudWatch Logs is enabled by default in the variable definition.

3. Configure your AWS credentials.

4. Run terraform init and terraform apply -var-file terraform.tfvars.

Once the configuration is applied, you can open the target knowledge base in the Amazon Bedrock Console and click Edit in the Knowledge Base overview section to review the logging configuration:

Assuming all three log destinations are enabled, it should look something like:

For good measure, we can perform a task with the knowledge base that generates application logs and see if logs are being delivered. At the time of writing, Bedrock Knowledge Bases only generate logs from ingestion job events. As such, we can trigger a sync of a data source in the knowledge base. The log group should have logs similar to the following:

Next, the S3 bucket should have logs similar to the following:

Lastly, the destination of the Firehose delivery stream, which in our case is another S3 bucket, should have logs similar to the following:

If you don’t need the resources after testing, be sure to delete them to avoid unexpected costs.

Summary

In this blog post, we examined how logging works for Amazon Bedrock Knowledge Bases, which uses the log delivery feature in CloudWatch Logs. We created and tested Terraform configuration that demonstrates knowledge base log delivery to all three supported destinations - CloudWatch Logs, S3, and Data Firehose. You can also repurpose the Terraform configuration for other AWS services that use the log delivery mechanism with minimal changes, should you have a need.

At this point, we have the know-how to write ingestion logs to CloudWatch Logs, so we can update the data ingestion solution I previously wrote about to improve how ingestion job notifications are triggered. Please stay tuned for my next blog post on this topic. Thanks for reading, as always, and be sure to check out the Avangards Blog for more AWS and Terraform content.

2025 started off busy, and only recently have I had the chance to catch up on all the new Amazon Bedrock features that were launched during re:Invent 2024. These new capabilities make it easier than ever to build a comprehensive RAG solution using a low-code approach. As I explored these new features, I realized that most, if not all, are functional features. I feel that there isn’t a lot of guidance on the operational aspects of Bedrock services, so I decided to write more about this topic.

A key component of any RAG system is the data ingestion pipeline. Amazon Bedrock Knowledge Bases has built-in data ingestion that does the heavy lifting, and synchronizations can be triggered on demand. From an operational perspective, the end-to-end process should ideally be automated and aligned with either the update cadence of the data source or a designated maintenance window. This is an ideal use case for a Lambda-based automation solution, for which I have built a basic version. In this blog post, we’ll walk through its design and implementation.

Design Overview

The overall design of the solution is depicted in the following diagram:

The solution works as follows:

1. A Lambda function, triggered by an EventBridge schedule rule, periodically starts an ingestion (a.k.a. sync) job for each specified knowledge base and data source. The function also sends a message with job ID information to an SQS queue.

2. Another Lambda function, triggered by an EventBridge schedule rule, frequently fetches messages from the SQS queue. For each message, the function uses the job ID information to get details about the ingestion job. The message is removed from the queue if the job has completed, and a notification is sent to one of the two SNS topics depending on whether the job is successful or failed.

3. (Optional) Additional downstream tasks can be performed by subscribing to the SNS topics.

With the high-level architecture in mind, let’s now dive into the detailed design of each major component.

Component Design: Starting Ingestion Jobs

Amazon Bedrock Knowledge Bases simplifies ingestion for data sources it natively supports such as Amazon S3 and Web Crawler. Its default parsing logic covers most cases, while its default chunking logic allows the selection of different strategies that could improve data retrieval quality.

For Amazon S3, while it is possible to start ingesting data as the S3 bucket is updated (using S3 event notifications for instance), it is not recommended when the knowledge base is in use especially during high usage periods. For Web Crawler, detecting website updates - especially one you don't own - is often difficult. Instead, a more reliable approach is to schedule ingestion during a maintenance window (for example, after midnight). This, as we know, can easily be implemented using an EventBridge rule that runs on a schedule.

When using default settings, you simply need to synchronize the data source when the data source is updated. This is done programmatically via the StartIngestionJob action in the Agents for Amazon Bedrock API. This action is available as the start_ingestion_job method in Boto3, which is used in our Python-based Lambda function.

Since ingestion is asynchronous, you must check job statuses separately. In our event-based setup, each job’s details (knowledge base ID, data source ID, ingestion job ID) is passed to the other Lambda function that checks ingestion job statuses. To facilitate communication of the decoupled components, we can use an SNS topic, SQS queue, or DynamoDB table. Since long-running jobs require multiple status checks, an SQS standard queue seems like a better fit.

Lastly, we need to configure which knowledge bases, data sources, and SQS queues the Lambda function should manage. AWS Systems Manager Parameter Store is an obvious choice. To structure the knowledge base and data source information, we’ll use a JSON list that contains an object with the knowledge base ID and the list of data source IDs, for example:

[
  {
    "knowledge_base_id" = "YO4R9AYHQZ",
    "data_source_ids"   = ["5IHZ5YAIBY"]
  }
]

Now that we've outlined the design, let's look at the implementation of the Lambda function:

import boto3
import json
from botocore.exceptions import ClientError

bedrock_agent = boto3.client('bedrock-agent')
sqs = boto3.client('sqs')
ssm = boto3.client('ssm')


def lambda_handler(event, context):
    try:
        # Retrieve the JSON config from Parameter Store
        response = ssm.get_parameter(Name='/start-kb-ingestion-jobs/config-json')
        config_json = response['Parameter']['Value']
        config = json.loads(config_json)

        for record in config:
            knowledge_base_id = record.get('knowledge_base_id')
            for data_source_id in record.get('data_source_ids'):
                # Start the ingestion job
                print(f'Starting ingestion job for data source {data_source_id} of knowledge base {knowledge_base_id}')
                response = bedrock_agent.start_ingestion_job(
                    knowledgeBaseId=knowledge_base_id,
                    dataSourceId=data_source_id
                )
                ingestion_job_id = response['ingestionJob']['ingestionJobId']

                # Send a message to the SQS queue
                response = ssm.get_parameter(Name='/start-kb-ingestion-jobs/sqs-queue-url')
                sqs_queue_url = response['Parameter']['Value']
                message = {
                    'knowledge_base_id': knowledge_base_id,
                    'data_source_id': data_source_id,
                    'ingestion_job_id': ingestion_job_id
                }
                sqs.send_message(
                    QueueUrl=sqs_queue_url,
                    MessageBody=json.dumps(message)
                )
        return {
            'statusCode': 200,
            'body': 'Success'
        }
    except ClientError as e:
        return {
            'statusCode': 500,
            'body': f'Client error: {str(e)}'
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': f'Unexpected error: {str(e)}'
        }

Component Design: Checking Ingestion Job Statuses

The second piece of the puzzle is checking the status of any ingestion jobs initiated by the solution and sending notifications when a job completes, fails, or is canceled. Notifications should also include any warnings to highlight issues ingesting specific documents that could impact the quality of the RAG solution.

To retrieve ingestion job details, the GetIngestionJob action in the Agents for Amazon Bedrock API can be used. The Boto3 equivalent would be the get_ingestion_job method. The required parameters are included in each SQS message sent by the Lambda function that starts ingestion jobs as described earlier. The approach is to poll and process SQS messages on a schedule (for example, every five minutes).

SQS polling can be tricky, especially if you’re unfamiliar with it. Short and long polling determine how long a ReceiveMessage API request (or its Boto3 equivalent, the receive_message method) waits for at least one message to show up in the queue. Here, long polling is not ideal since the best practice is to minimize Lambda function runtime when idle. It is also unnecessary given the Lambda function is run on a schedule anyway. Additionally, SQS’s distributed nature affects how many messages the ReceiveMessage API returns. Even with a higher MaxNumberOfMessages number, receiving multiple messages aren’t guaranteed in each call. Consequently, you must call the API until it returns empty result.

For notifications, the publish-subscribe pattern fits the scenario well. We’ll use two Amazon SNS topics: one for successful job completions and another for failures and cancellations. Administrators can handle subscriptions downstream as needed, whether via email or additional Lambda processing.

Finally, SSM Parameter Store will store configuration details, including the SQS queue URL and SNS topic ARNs, ensuring consistency. The resulting Lambda function could look like this:

import boto3
import json
from botocore.exceptions import ClientError

bedrock_agent = boto3.client('bedrock-agent')
ssm = boto3.client('ssm')
sns = boto3.client('sns')
sqs = boto3.client('sqs')


def get_ssm_parameter(name):
    response = ssm.get_parameter(Name=name, WithDecryption=True)
    return response['Parameter']['Value']


def get_ingestion_job(knowledge_base_id, data_source_id, ingestion_job_id):
    response = bedrock_agent.get_ingestion_job(
        knowledgeBaseId=knowledge_base_id,
        dataSourceId=data_source_id,
        ingestionJobId=ingestion_job_id
    )
    return response['ingestionJob']


def lambda_handler(event, context):
    try:
        sqs_queue_url = get_ssm_parameter('/check-kb-ingestion-job-statuses/sqs-queue-url')
        success_sns_topic_arn = get_ssm_parameter('/check-kb-ingestion-job-statuses/success-sns-topic-arn')
        failure_sns_topic_arn = get_ssm_parameter('/check-kb-ingestion-job-statuses/failure-sns-topic-arn')

        response = sqs.receive_message(
            QueueUrl=sqs_queue_url,
            MaxNumberOfMessages=10
        )
        while 'Messages' in response:
            messages = response['Messages']
            for message in messages:
                body = json.loads(message['Body'])
                knowledge_base_id = body['knowledge_base_id']
                data_source_id = body['data_source_id']
                ingestion_job_id = body['ingestion_job_id']

                print(
                    f'Checking ingestion job status for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id}')
                ingestion_job = get_ingestion_job(knowledge_base_id, data_source_id, ingestion_job_id)
                print(
                    f'Ingestion job summary: \n\n{json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)}')
                job_status = ingestion_job['status']
                if job_status == 'COMPLETE':
                    sns.publish(
                        TopicArn=success_sns_topic_arn,
                        Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} Completed',
                        Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                    )
                elif job_status == 'FAILED':
                    sns.publish(
                        TopicArn=failure_sns_topic_arn,
                        Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} FAILED',
                        Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                    )
                elif job_status == 'STOPPED':
                    sns.publish(
                        TopicArn=failure_sns_topic_arn,
                        Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} STOPPED',
                        Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                    )

                if job_status in ['COMPLETE', 'FAILED', 'STOPPED']:
                    sqs.delete_message(
                        QueueUrl=sqs_queue_url,
                        ReceiptHandle=message['ReceiptHandle']
                    )
            response = sqs.receive_message(
                QueueUrl=sqs_queue_url,
                MaxNumberOfMessages=10
            )
        return {
            'statusCode': 200,
            'body': 'Success'
        }
    except ClientError as e:
        return {
            'statusCode': 500,
            'body': f'Client error: {str(e)}'
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': f'Unexpected error: {str(e)}'
        }

Terraform Configuration Design

No solution is complete without an easy way to deploy it. Terraform is an obvious choice given my advocacy for it, however any IaC solutions or even frameworks such as AWS SAM would do. Since the event-based automation architecture is quite typical, I won’t go into too much details on how the Terraform configuration is developed. But here is a snippet related to the first Lambda function:

# Data sources and local values omitted for brevity

resource "aws_sqs_queue" "check_kb_ingestion_job_statuses" {
  name = "check-kb-ingestion-job-statuses"
}

resource "aws_ssm_parameter" "start_kb_ingestion_jobs_config_json" {
  name  = "/start-kb-ingestion-jobs/config-json"
  type  = "String"
  value = jsonencode(var.start_kb_ingestion_jobs_config_json)
}

resource "aws_ssm_parameter" "start_kb_ingestion_jobs_sqs_queue_url" {
  name  = "/start-kb-ingestion-jobs/sqs-queue-url"
  type  = "String"
  value = aws_sqs_queue.check_kb_ingestion_job_statuses.id
}

resource "aws_iam_role" "lambda_start_kb_ingestion_jobs" {
  name = "FunctionExecutionRoleForLambda-start-kb-ingestion-jobs"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = "${local.account_id}"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_start_kb_ingestion_jobs_lambda_basic_execution" {
  role       = aws_iam_role.lambda_start_kb_ingestion_jobs.name
  policy_arn = data.aws_iam_policy.lambda_basic_execution.arn
}

resource "aws_iam_role_policy" "lambda_start_kb_ingestion_jobs" {
  name = "FunctionExecutionRolePolicyForLambda-start-kb-ingestion-jobs"
  role = aws_iam_role.lambda_start_kb_ingestion_jobs.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ssm:GetParameter",
          "ssm:GetParameters",
          "ssm:GetParametersByPath"
        ]
        Effect   = "Allow"
        Resource = "arn:${local.partition}:ssm:${local.region}:${local.account_id}:parameter/*"
      },
      {
        Action   = "sqs:SendMessage"
        Effect   = "Allow"
        Resource = "arn:${local.partition}:sqs:${local.region}:${local.account_id}:*"
      },
      {
        Action = [
          "bedrock:StartIngestionJob"
        ]
        Effect   = "Allow"
        Resource = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:knowledge-base/*"
      }
    ]
  })
}

resource "aws_lambda_function" "start_kb_ingestion_jobs" {
  function_name = "start-kb-ingestion-jobs"
  role          = aws_iam_role.lambda_start_kb_ingestion_jobs.arn
  description   = "Lambda function that starts ingestion jobs for Bedrock Knowledge Bases"
  filename      = data.archive_file.start_kb_ingestion_jobs_zip.output_path
  handler       = "index.lambda_handler"
  runtime       = "python3.13"
  architectures = ["arm64"]
  timeout       = 60
  # source_code_hash is required to detect changes to Lambda code/zip
  source_code_hash = data.archive_file.start_kb_ingestion_jobs_zip.output_base64sha256
}

resource "aws_cloudwatch_event_rule" "start_kb_ingestion_jobs" {
  name                = "lambda-start-kb-ingestion-jobs"
  schedule_expression = var.start_kb_ingestion_jobs_schedule
}

resource "aws_cloudwatch_event_target" "start_kb_ingestion_jobs" {
  rule = aws_cloudwatch_event_rule.start_kb_ingestion_jobs.name
  arn  = aws_lambda_function.start_kb_ingestion_jobs.arn
}

resource "aws_lambda_permission" "start_kb_ingestion_jobs" {
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.start_kb_ingestion_jobs.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.start_kb_ingestion_jobs.arn
}

# Remaining resources omitted for brevity

What you will notice is that:

• The IAM and resource policies follow the least privilege principle.

• The SSM parameters uses a simple parameter hierarchy based on the Lambda function name. If deploying multiple copies of the solution, consider using a variable to set the function name and related source names.

• The Lambda function uses the arm64 architecture for ~20% better cost efficiency per GB-second/month vs. x86_64, though the impact in this solution is negligible.

• The default Lambda execution timeout of three seconds is insufficient. Increasing it to 60 seconds provides a good safety net.

Deploying and Testing the Solution

To deploy and test the solution, you need a knowledge base with at least one data source that has content to ingest either in an S3 bucket or a crawlable website. You can set this up in the Bedrock console using the vector database quick start options. Alternatively, deploy a sample knowledge base using the Terraform configuration from my blog post How To Manage an Amazon Bedrock Knowledge Base Using Terraform. This configuration is also available in the same GitHub repository under the 2_knowledge_base directory.

With the prerequisites in place, deploy the solution as follows:

1. From the root of the cloned GitHub repository, navigate to 3_kb_data_ingestion.

2. Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.

   • By default, the schedule for the start-kb-ingestion-jobs Lambda function is daily at 0:00 UTC, while the schedule for the check_kb_ingestion_job_statuses Lambda function is every five minutes.

3. Configure your AWS credentials.

4. Run terraform init and terraform apply -var-file terraform.tfvars.

Once deployed, test the solution by adding an email subscription to the SNS topics check-kb-ingestion-job-statuses-success and check-kb-ingestion-job-statuses-failure for your e-mail address so that you can receive email notifications. Confirm your subscriptions using the link in the verification emails.

Next, manually invoke the start-kb-ingestion-jobs Lambda function in the Lambda console within the five minutes of the scheduled check_kb_ingestion_job_statuses runs.

After the next check_kb_ingestion_job_statuses invocation completes at the scheduled time, check the CloudWatch logs to confirm it ran successfully. You should also receive a an email from SNS with its status. Here is an example:

The message shows several warnings about files that couldn’t be ingested. In my case, I used a S3 bucket that contained tar.gz files, which are unsupported. There was also one file exceeding the 50 MB limit, so it was also skipped. With this information, you can remediate these issues to ensure good data input to the knowledge base.

Once you've verified the solution works, remove the SNS subscription and replace it with those that better fit your needs. If you don’t plan to keep the knowledge base, delete it along with the vector store (for example, the OSS index) to avoid unnecessary costs.

Summary

In this blog post, we examined the development of a Lambda-based data ingestion solution for Bedrock knowledge bases. The design follows a familiar event-based pattern, leveraging AWS APIs to perform the required tasks and Terraform for deployment. That said, while checking ingestion job statues on a schedule works, it is not as efficient as a true event-based system which Amazon Bedrock does not seem to support at the moment. Perhaps I can update the solution once more support for data ingestion job events are added in the future.

Another interesting experience while writing this blog post is that I used GitHub Copilot to develop both the Lambda functions and the Terraform configuration. While it didn’t produce ready-to-use code, it saved me significant time. I’ll share more about this experience in my next blog post.

In the mean time, please check out other helpful content in the Avangards Blog. Thanks for reading!

Ensuring compliance with stringent security requirements often leads to unexpected challenges - here’s one I recently tackled. The account in question has hundreds of EC2 instances with EBS volumes that are encrypted with the KMS AWS managed key aws/ebs. Due to more stringent security compliance requirements, the encryption key must be rotated every 90 days. The default one-year rotation period of the AWS managed key no longer suffices, thus all EC2 instance volumes must be re-encrypted with a custom managed key (CMK) that provides more control.

With a looming deadline, it was simply not feasible to manually re-encrypt all EBS volumes. Thus I set out to find a tool or script that can automate this daunting task before resorting to developing my own (even if it’s AI-generated). Luckily I found a GitHub repository with a script that meets 90% of my needs, and made perfect after some enhancements. I’d like to share my experience and the resulting script in this blog post in case it benefits any fellow builders facing the same problem. Let’s first set the stage by examining the encryption workflow.

The Encryption Workflow

To encrypt or re-encrypt an EBS volume that is attached to an EC2 instance, it is unfortunately not as simple as setting a KMS key ID on the volume. The process is a bit roundabout and involves the following steps:

1. Shut down the EC2 instance.

2. Create a snapshot of the volume.

3. Create a new volume from the previously created snapshot, while enabling encryption with the new KMS key. Ensure that you select the same availability zone as the original volume and apply any volume settings.

4. Detach the original volume from the EC2 instance while making note of the device name to which the volume is attached.

5. Attach the new volume to the EC2 instance with the same device name as above.

6. Repeat step 2 to step 6 for any additional volumes that the EC2 instance has.

7. Start the EC2 instance and verify that it is running properly.

8. Delete the original volumes and the snapshots taken during this process as appropriate.

There are other considerations and best practices for Amazon EBS encryption with auto scaling groups, spot instances, and snapshot sharing, however they are not relevant for the basic scenario for this blog post.

As you can see, the workflow is quite involved and thus makes a great candidate for automation.

Leveraging an Existing Script on GitHub

The need to encrypt or re-encrypt EBS volumes is not uncommon, so I figure that someone would have developed tools and scripts for it. Indeed, a quick Google search yielded three possible options:

1. dwbelliston/aws_volume_encryption - a Python-based script developed by Dustin Belliston that orchestrates encryption of EBS volumes of an EC2 instance.

2. jbrt/ec2cryptomatic - a Go-based tool developed by Julien B. that is very similar to the Python solution above, but with a few more quality-of-life features.

3. aws-samples/aws-system-manager-automation-unencrypted-to-encrypted-resources - an AWS solution that automatically remediates unencrypted EBS and RDS resources using AWS Config and SSM Automation.

Given Boto3 and Python are part of my preferred toolset, I decided to leverage the aws_volume_encryption solution as my starting point. The repository has a well-written README file that provides usage instructions and detailed explanation on what each section of the code does. Be sure to check it out so you understand the general architecture and usage of the script.

Enhancing the Existing Script

Although developed years ago, the original script remains fully functional, proving its reliability. That said, I have identified a few small enhancements that could improve the usability of the script:

1. Defer to Boto3’s default credential search mechanism instead of adding redundant options to the script.

2. Create an encrypted volume directly from an encrypted snapshot.

3. Add KMS key ID validation and skip encryption of volumes that are already encrypted with the provided KMS key. The KMS key ID can be any of the four supported formats by the AWS API.

4. Add option to preserve the original volumes and add metadata tags (prefixed by VolumeEncryptionMetadata:) to them in case volume changes need to be reverted.

5. Improve console logging with timestamps and more details.

These enhancements allow me to test and benchmark the script more effectively, and they provide me an extra level of assurance when working on production workloads.

Running the Script

To test the script, create a Windows EC2 instance with an unencrypted root volume and a data volume that is encrypted using the AWS managed key:

Once the EC2 instance is running, connect to Windows, initialize the second volume as D: drive, and add a text file to help with validation later.

Then create a KMS CMK that will be used to encrypt the EBS volumes of the EC2 instance.

Lastly, clone the GitHub repository or download the code zip file, then follow the README file to set up the prerequisites including Python 3.x and the AWS CLI. Since the script defers to the typical credentials lookup sequence, you may use any of the supported methods. Typically, you either configure a profile with the AWS CLI and refer to it using the AWS_PROFILE environment variable, or you use the more verbose environment variables including AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. In any case, ensure that you provide the target region in the profile or using the AWS_DEFAULT_REGION environment variable.

Run the volume_encryption.py script while providing the following arguments:

• -i with the ID of the target EC2 instance

• -k with the KMS CMK ID or alias

• -p to preserve the original volume, which can be inspected and deleted after validating the EC2 instance with newly encrypted volumes

Here is the command that is specific to my resources:

python volume_encryption.py -i i-095dc6901ff37f71d -k 73d13a4a-b0b0-4ced-b82b-d86a78c89df0 -p

How long the script takes largely depends on the volume sizes and the encryption state of the volumes. For my test instance with a 30 GB encrypted volume and a 8 GB encrypted volume, it took a bit over 6 minutes to complete. If I were to run the script again using another key, it would take more than 20 minutes to complete, presumably because re-encryption takes longer. In any case, the console logs include timestamps that indicate how long each step takes.

$ python volume_encryption.py -i i-095dc6901ff37f71d -k 73d13a4a-b0b0-4ced-b82b-d86a78c89df0 -p
[2025-01-19T13:51:38.781614-05:00] Checking instance i-095dc6901ff37f71d
[2025-01-19T13:51:39.624892-05:00] Stopping instance i-095dc6901ff37f71d
[2025-01-19T13:52:55.908023-05:00] Create snapshot of volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:53:57.048132-05:00] Create encrypted volume from snapshot snap-0fb3ecd0eaf583f82
[2025-01-19T13:53:57.684619-05:00] Detach volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:53:58.048124-05:00] Attach volume vol-0ddb7f75257290d19 (xvdb)
[2025-01-19T13:54:13.966946-05:00] Create snapshot of volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:14.751392-05:00] Create encrypted volume from snapshot snap-0dac8c77e6e777d1a
[2025-01-19T13:55:15.271831-05:00] Detach volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:15.615831-05:00] Attach volume vol-01140d9f4c666a763 (/dev/sda1)
[2025-01-19T13:55:31.975440-05:00] Start instance i-095dc6901ff37f71d
[2025-01-19T13:55:48.251798-05:00] Clean up resources
[2025-01-19T13:55:48.252797-05:00] Delete snapshot snap-0fb3ecd0eaf583f82
[2025-01-19T13:55:48.438101-05:00] Skipping deletion of original volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:55:48.438101-05:00] Delete snapshot snap-0dac8c77e6e777d1a
[2025-01-19T13:55:48.614087-05:00] Skipping deletion of original volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:48.615090-05:00] Encryption finished
$

Once the script completes, verify that the EBS volumes are encrypted with the new KMS key.

You should also see that the original volumes still exist and have some metadata tags added by the script for traceability. Note the original volume IDs from the console logs of the script (for example, vol-054b9017ef1f8f25c is the ID of the original root volume).

Lastly, log in to the EC2 instance and ensure that Windows is working as intended and D:\hello.txt still exists.

However, you will notice that the EC2 instance is seemingly slower than usual. This is because volumes that are restored from snapshots are not fully initialized or pre-warmed. This means that only when a block within the volume is accessed that it will be loaded from the snapshot that’s stored in S3 behind the scenes, thus increasing the I/O latency. While this may not be a huge issue with common use cases, for use cases with high disk I/O needs (such as running a database), you may need to manually initialize the disks.

Summary

With this improved script, you can (re-)encrypt the EBS volumes of any EC2 instance with ease. If you are encrypting volumes for many instances, you can also write another script that reads a CSV file containing EC2 instance information and runs volume_encryption.py on multiple instances in parallel. AI tools like ChatGPT, Amazon Q Developer, or GitHub Copilot can easily create one for you, as I did for my own needs. I will leave this as an exercise for the audience.

As they say, prevention is better than cure. If your organization’s security policies require that EBS volumes be encrypted, consider using the Amazon EBS encryption by default feature to automatically encrypt any new EBS volumes.

This demonstrates how automation and generative AI empower DevOps engineers to tackle complex challenges efficiently. I hope you find this blog post informative and the script useful should you run into similar situations. If you like this article, please check out my other blog posts for more helpful and intriguing content on AWS and DevOps. Thank you for reading and have a great one!

In the blog post Developing a Generic Streamlit UI to Test Amazon Bedrock Agents, I shared the design and source code of a basic yet functional UI for testing Bedrock agents. I’ve since added support for Knowledge Bases for Amazon Bedrock by displaying citations and their details to match the functionality in the Bedrock console.

Recently I’ve started experimenting with Guardrails for Amazon Bedrock, a feature that enables the implementation of safeguards for your generative AI applications based on specific use cases and responsible AI policies. As part of the blog post A Guide to Effective Use of the Terraform AWS Cloud Control Provider, I created a simple history-themed Bedrock agent with a guardrail that filters violent content. As I test a guardrail-enabled agent in the Bedrock console, I see additional traces showing whether guardrails have intervened as they assess the input and output, and if so, what policies were triggered:

With a bit of work, I have added similar support to the generic test UI and I am happy to share the updates in the GitHub repository.

Design overview

With the latest update, guardrail traces are now added to the pre-processing and post-processing trace sections in a manner similar to how they are displayed in the Bedrock console:

The Boto3 invoke_agent method provides a new guardrailTrace trace type that includes trace details used in the guardrail. Distinguishing between pre- and post-guardrail traces took a bit of work, as they need to be tracked as events are streamed in sequence. If a guardrailTrace shows up for the first time (naturally before preProcessingTraces, it would be a pre-guardrail trace. A subsequent guardrailTrace that shows up (naturally after any postProcessingTraces would be a post-guardrail trace. Then they must be displayed under the Pre-Processing and Post-Processing sections as a single JSON object unlike other traces where they are broken down into smaller JSON objects for readability.

Summary

With the improvements to the generic test UI outlined in this post, you should now be able to test any Bedrock agents with an associated guardrail. I will be sure to incorporate support for new Agents for Amazon Bedrock features. If you find this blog post helpful, there is plenty more similar content at the Avangards Blog. Be sure to check them out!

The AWS Cloud Control (CC) Provider gained significant attention in May 2024 when it became generally available, three years after its initial launch. It promises to support new AWS features and services immediately due to its auto-generated nature, which is especially beneficial for the rapidly evolving generative AI services like Amazon Bedrock.

But should you immediately switch to the AWS CC Provider and abandon the classic AWS Provider? Not necessarily. While the CC Provider brings speed and coverage of new services, it’s not without limitations. In this blog post, we’ll break down the strengths and weaknesses of both providers, highlighting when it makes sense to leverage the AWS CC Provider and where the classic AWS Provider still shines. A practical example will demonstrate how both can be best used together.

Understanding the Strengths and Weaknesses of the AWS CC Provider

The main selling point of the AWS CC Provider is that it provides support for new AWS services sooner than the classic AWS Provider. The GA announcement showcases this speed by supporting the Amazon Q Business resources early on. In contrast, the enhancement request that was opened against the AWS Provider in January 2024 is still pending, even though a pull request (PR) has already been submitted by a contributor for some time. It makes sense to leverage the resources from the AWS CC Provider to not delay your IaC automation effort.

Even though the AWS CC Provider covers new AWS services quickly, there’s still a lot of room for improvement when it comes to older services. According to this GitHub issue, as of October 2023, the Cloud Control API supports only 859 resources, many of which are not supported in all AWS regions. According to Resource types that support Cloud Control API, as of July 2024 it supports 1034 resources, which is encouraging to see. However, there is still a list of suppressed resources that are not compatible with how the AWS CC Provider generates resources, bringing the actual supported number of resources to around 1,000. Compared to about 1,400 resources supported by the AWS Provider, that's only about 70% coverage and less if you consider resources that haven't been implemented in the AWS Provider.

While working with the AWS CC Provider, I noticed challenges with both documentation and quality assurance. The quality of descriptions for resources and attributes is somewhat inconsistent. For example, the documentation for the awscc_iam_group resource is quite well written, while the documentation for the awscc_qbusiness_application resource is practically non-existent. Overall, it pales in comparison to the AWS API documentation which I often refer to when contributing to the AWS Provider. I am not sure why the CloudFormation schemas (from which the AWS CC Provider resources and documentation are generated) are so far apart from the AWS API documentation, but I hope AWS can reconcile the two sources at some point.

As for the functional quality of the AWS CC Provider, my experience unfortunately hasn't been great. While adding examples to the Lightsail resources, I ran into two major functional issues and two documentation issues that are caused upstream in the Cloud Control API. This led me to believe that there is insufficient quality assurance with the Cloud Control API, and the generated nature of the AWS CC Provider does not help catch these issues. The situation will hopefully improve over time, but for the time being I would prefer the AWS Provider for mission-critical use. Nevertheless, I must give credit to the AWS CC Provider maintainers in diligently reporting and working with AWS to resolve upstream issues in a timely manner. The turnaround time is much quicker than if I were to open AWS support cases myself.

Also Knowing the Merits and Drawbacks of the Classic AWS Provider

The Terraform AWS Provider has been active for over ten years and has recently surpassed three billion downloads. The tremendous work that HashiCorp, AWS, and the community put into the provider over the years has led to high AWS service coverage. The provider boasts ample acceptance tests which result in a relatively high degree of quality. The user base also proactively reports less prevalent issues that are not caught by automated tests.

Since the AWS Provider code is not generated like the AWS CC Provider, the hand-crafted nature means that development is labor intensive and time consuming. Consequently, lower priority issues and features often take some time to be fixed. Even when a PR is submitted by a contributor, a maintainer from HashiCorp still needs to review, test, and merge it in between their other work such as those related to product roadmaps.

Meanwhile, the need for hands-on development also affords the flexibility to add custom logic to resources and data sources. As previously mentioned, the AWS API often does not fit perfectly into a CRUDL model due to actions that fall outside these operations. For instance, the Agents for Amazon Bedrock API has an AssociateAgentKnowledgeBase action that associates a knowledge base to an agent. Since it is not considered a resource in the AWS CC API, it is not mapped to a resource in the AWS CC Provider. However, an experienced developer for the AWS Provider is able to adapt this action into an "association", resulting in the aws_bedrockagent_agent_knowledge_base_association resource.

As another example, a Bedrock agent must be prepared using the PrepareAgent action after it is updated. Since this action cannot be easily adapted to a resource, the logical approach is to call this API action when an aws_bedrockagent_agent resource is created and updated, leading to the custom prepare_agent argument. Similar logic can be added to other Agents for Bedrock resources that indirectly modifies an agent. Having this type of custom resources and logic is only possible in the AWS Provider today.

Using Both Providers in a Complementary Manner

The good news is that Terraform is designed to work with multiple providers, so you can leverage both the AWS Provider and the AWS CC Provider for what they each excel at.

Let’s look at a use case of adding a guardrail to a Bedrock agent. Currently, the AWS Provider has the aws_bedrock_guardrail resource, but it does not yet have a resource to manage guardrail versions. While the aws_bedrock_agent resource has been around for some time, it does not yet have the configuration to associate a guardrail.

On the other hand, the AWS CC Provider has a awscc_bedrock_guardrail_version resource and the awscc_bedrock_agent resource supports the guardrail_configuration argument for associating a guardrail. Thus we can strategically use the AWS CC Provider for the new features while using the AWS Provider for all other resources.

Here is the Terraform configuration for a simple Bedrock agent that answers questions about world history, but is guarded against providing information on violent historical events like what happened to Julius Caesar:

data "aws_caller_identity" "this" {}
data "aws_partition" "this" {}
data "aws_region" "this" {}
locals {
  account_id = data.aws_caller_identity.this.account_id
  partition  = data.aws_partition.this.partition
  region     = data.aws_region.this.name
}

data "aws_bedrock_foundation_model" "this" {
  model_id = "anthropic.claude-3-haiku-20240307-v1:0"
}

resource "aws_bedrock_guardrail" "this" {
  name                      = "MyGuardrail"
  description               = "My guardrail"
  blocked_input_messaging   = "Sorry, I cannot answer this question."
  blocked_outputs_messaging = "Sorry, I cannot answer this question."
  content_policy_config {
    filters_config {
      input_strength  = "HIGH"
      output_strength = "HIGH"
      type            = "VIOLENCE"
    }
  }
}

resource "awscc_bedrock_guardrail_version" "this" {
  guardrail_identifier = aws_bedrock_guardrail.this.guardrail_id
  lifecycle {
    replace_triggered_by = [aws_bedrock_guardrail.this]
  }
}

resource "aws_iam_role" "bedrock_agent_this" {
  name = "AmazonBedrockExecutionRoleForAgents_MyAgent"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "bedrock.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = local.account_id
          }
          ArnLike = {
            "aws:SourceArn" = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:agent/*"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "bedrock_agent_this" {
  name = "AmazonBedrockAgentBedrockFoundationModelPolicy_MyAgent"
  role = aws_iam_role.bedrock_agent_this.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "InvokeFoundationModel"
        Action   = "bedrock:InvokeModel"
        Effect   = "Allow"
        Resource = data.aws_bedrock_foundation_model.this.model_arn
      },
      {
        Sid      = "ApplyGuardrail"
        Action   = "bedrock:ApplyGuardrail"
        Effect   = "Allow"
        Resource = awscc_bedrock_guardrail_version.this.guardrail_arn
      }
    ]
  })
}

resource "awscc_bedrock_agent" "this" {
  agent_name              = "MyAgent"
  agent_resource_role_arn = aws_iam_role.bedrock_agent_this.arn
  auto_prepare            = true
  description             = "My Agent"
  foundation_model        = data.aws_bedrock_foundation_model.this.model_id
  guardrail_configuration = {
    guardrail_identifier = awscc_bedrock_guardrail_version.this.guardrail_arn
    guardrail_version    = awscc_bedrock_guardrail_version.this.version
  }
  instruction = "You are an assistant that provides information about world history. You are allowed to use general knowledge that you already possess to answer any history-related questions."
}

As you can see, the Terraform configuration maintains the familiar usage of resources and data sources in the AWS Provider. A quick validation in the Amazon Bedrock console shows that the agent does indeed filter the violent event about Julius Caesar, while it correctly answers another question about the history of wheels.

Summary

In this blog post, we looked at the pros and cons of the AWS Provider and the AWS CC Provider. As it stands, there is still a long way until AWS CC Provider has the quality and feature parity necessary to replace the AWS Provider, so both are here to stay for the foreseeable future.

If you're managing complex AWS infrastructure, now is the time to experiment with both providers. Use the AWS CC Provider for cutting-edge features, and rely on the AWS Provider for tried-and-true solutions. By blending both providers, you’ll have the best of both worlds in your Terraform configurations. You can follow this tutorial or find more information here.

For other tips and walkthroughs on AWS and Terraform, be sure to check out the Avangards Blog. Thanks for reading!

AWS has recently revamped their certification lineup to align with the growing AI/ML trend. Among them is the AWS Certified AI Practitioner (AI1-C01) exam, which is currently in beta. Beta exams validate exam questions to finalize the content before wide release. As an early adopter, you get a discount on the exam fee in exchange for your contribution to test out the exam for AWS. Given that this is an intriguing proposition and it aligns with my focus on AWS, I decided to write this exam and share my experience with the community.

A Bit About My Background

For context, I've been working exclusively with AWS for about three years and completed a couple of professional/specialty level exams, so I am quite familiar with AWS in general. My work also involves generative and conversational AI (although it has mostly been on the Microsoft side), so I have knowledge on concepts such as LLM and RAG.

I am interested in generative AI on AWS, so I've been experimenting on my own and has written a few blog post about Agents for Amazon Bedrock and Knowledge Bases for Amazon Bedrock. However, my knowledge of Amazon SageMaker is limited to understanding the ML workflow and some hands-on experience from AWS Workshops.

Overall I'd say that I know a bit more about AI/ML than the average Joe, so I was able to expedite my study somewhat. As you read about my exam prep, consider your own knowledge and experience to adjust your approach.

How I Studied for the Exam

In the past, I've always used A Cloud Guru to study for AWS and Azure exams. Recently I've been given an AWS Skill Builder subscription by my company, so I've decided to use it as the primary source of study material.

The official AWS Certified AI Practitioner webpage recommends a 4-step exam prep plan which I followed over two days. To test my knowledge prior to studying, I went through the official practice question set and got 85% which boosted my confidence (but eventually turned out to be a trap). I then proceeded with the enhanced exam prep course that included bonus practice questions and flashcards over the standard (free) version.

The course itself is well-organized and touches upon all topics. The instructor spoke very slowly in the video, so I watched the videos at 2x speed for a more reasonable pace. However, I found that certain concepts aren't explained very well and the level of details isn't representative of what the exam itself demands. I suppose the additional resources would have been good supplements, however I would prefer if the course itself is more in-depth.

The bonus questions provided additional practice opportunities and the flashcards were helpful for memorizing key concepts. I transferred the flashcards into a Word document for offline review before the exam. That being said, they did not cover all important concepts and I had to supplement them with my own research on Google (such as a list of ML algorithms).

Following my typical study regimen, I sought more practice right before the exam. Aside from the practice questions from AWS, I quickly went through Tutorial Dojo's free practice exams sampler given that they don't yet have a full practice exam package yet. (I later found out that Stephane Maarek does.)

How the Exam Went

I took the exam online at the comfort of my home in a quiet evening. The check-in process went smoothly and took about 10 minutes. The exam questions were more difficult than I expected of a foundational-level exam. On a scale of 1 to 10, I’d rate it a 4 in difficulty.

Although the exam guide lists case study as a question type, there wasn't any for me so your mileage may vary. There were many questions on machine learning concepts such as algorithms and performance metrics. Amazon Bedrock also featured a lot. There was also the expected mix of questions on Amazon SageMaker features, generative AI, and responsible AI. While I wouldn't say that the questions were very different from the official practice question set or the bonus questions in the exam prep enhanced course, they seemed more in-depth and specific.

It took me about 75 minutes to complete the exam, with the first pass completed in about an hour which left 27 out of 85 questions flagged for review. This is more questions than I usually flag in AWS exams (including the DOP exam), and after the review I was only confident on my final answer to about half of those questions. I finished the exam feeling I would pass, though not with full confidence.

A Retrospective on My Approach

I received the results by early morning and scored only 729, which admittedly is lower than I expected. Nonetheless, a pass is a pass. According to the report, I didn't do as well in the "guidelines for responsible AI" domain which was a bit surprising as I did well during practice. In any case, here is my reflection on my experience based on a typical retrospective framework.

What Went Well

• Setting an exam date motivated me to be disciplined and keep to a set study schedule amid other priorities in life. The long weekend afforded me enough time to study, socialize, and do chores.

• Following the official exam prep plan and the exam prep course helped structure my study. I've always studied by following a course be it from A Cloud Guru or AWS Skill Builder, and it's been proven to be a sound strategy.

What Didn't Go Well

• I was overly confident of my hands-on experience with services such as Amazon Bedrock, when there were numerous features and other services that I've not have enough exposure to. Consequently I did not allocate time to watch tutorial videos or do hands-on labs to gain the necessary familiarity and "muscle memory".

• I didn’t spend enough time on Step 2 of the 4-step plan to explore additional courses outside the AWS Skill Builder exam prep course. I've retroactively looked at some of the recommended courses and they would have improved my overall knowledge for this exam.

What Could Be Improved

• Although I was able to get by with one full day of study, it would have been better to spread the study across multiple days for a better pace.

• I would have done better with additional research on general AI/ML concepts such as algorithms and performance metrics, which the exam seemed to have more focus on.

• Although the AWS Skill Builder exam prep enhanced course was decent, it was not fully adequate as the sole study material. Investing in additional courses and practice exams, such as those from Stephane Maarek, would probably have helped boost my score.

Summary

I hope this blog post gives you a sense of what to expect from the AWS Certified AI Practitioner (AI1-C01) Beta Exam. It's certainly not an exam that you can wing, unless you are well-exposed to AI/ML or are already studying for other certifications such as AWS Certified Machine Learning Engineer - Associate. However with the right material and a few days of study, it is very much achievable.

Check out the Avangards Blog for more articles on AWS, Terraform, and other topics. Best of luck with your studies, and I hope you’ll soon be certified!

Over the past two months, I have published numerous blog posts on managing different AWS security services in AWS Organizations using Terraform. In this blog post, I will cover one remaining AWS service, AWS Inspector, for native vulnerability management. The Terraform resources for Inspector are a bit quirky, so I will show some slightly more advanced techniques to keep the configuration neat and configurable. With that said, let's review the objective.

About the use case

Amazon Inspector is a vulnerability management service that continuously scans AWS workloads for software vulnerabilities and unintended network exposure. Supported compute services include Amazon EC2 instances, container images in Amazon ECR, and AWS Lambda functions.

Similar to other AWS security services, Inspector supports managing multiple accounts with AWS Organizations via the delegated administrator feature. Once an account in the organization is designated as a delegated administrator, it can manage member accounts and view aggregated findings.

Since it is increasingly common to establish an AWS landing zone using AWS Control Tower, we will use the standard account structure in a Control Tower landing zone to demonstrate how to configure Inspector in Terraform:

The relevant accounts for our use case in the landing zone are:

• The Management account for the organization where AWS Organizations is configured. For details, refer to Managing multiple accounts in Amazon Inspector with Organizations.

• The Audit account where security and compliance services are typically centralized in a Control Tower landing zone.

The objective is to delegate Inspector administrative duties from the Management account to the Audit account, after which all organization configurations are managed in the Audit account. Let's walk through how to do this using Terraform.

Designating an Inspector administrator account

The Inspector delegated administrator is configured in the Management account, so we need a provider associated with it in Terraform. To keep things simple, we will take a multi-provider approach by defining two providers, one for the Management account and another for the Audit account, using AWS CLI profiles as follows:

provider "aws" {
  alias   = "management"
  # Use "aws configure" to create the "management" profile with the Management account credentials
  profile = "management" 
}

provider "aws" {
  alias   = "audit"
  # Use "aws configure" to create the "audit" profile with the Audit account credentials
  profile = "audit" 
}

We can designate the delegated administrator using the aws_inspector2_delegated_admin_account resource. However, this does not enable Inspector in the delegated administrator account, so we also need to use the aws_inspector2_enabler resource. What I learned from testing the aws_inspector2_enabler resource is that you cannot provide both the delegated account and the member accounts in the account_ids argument, so we need a dedicated aws_inspector2_enabler resource for the Audit account. According to the resource source code, this is to address a legacy Inspector issue.

The resulting Terraform configuration should look like the following (pay special attention to the provider argument in each resource):

data "aws_caller_identity" "audit" {
  provider = aws.audit
}

resource "aws_inspector2_enabler" "audit" {
  provider       = aws.audit
  account_ids    = [data.aws_caller_identity.audit.account_id]
}

resource "aws_inspector2_delegated_admin_account" "audit" {
  provider   = aws.management
  account_id = data.aws_caller_identity.audit.account_id
  depends_on = [aws_inspector2_enabler.audit]
}

Configuring Inspector activation for new member accounts

To allow more control over which scan types are enabled, we can define the following variables and use them with the relevant resources:

# Variable definition (.tfvars)

variable "enable_ec2" {
  description = "Whether Amazon EC2 scans should be enabled for both existing and new member accounts in the organization."
  type        = bool
  default     = true
}

variable "enable_ecr" {
  description = "Whether Amazon ECR scans should be enabled for both existing and new member accounts in the organization."
  type        = bool
  default     = true
}

variable "enable_lambda" {
  description = "Whether Lambda Function scans should be enabled for both existing and new member accounts in the organization."
  type        = bool
  default     = true
}

variable "enable_lambda_code" {
  description = "Whether Lambda code scans should be enabled for both existing and new member accounts in the organization."
  type        = bool
  default     = true
}

In an organizational setup, Inspector can auto-enable on new member accounts. In Terraform, this can be configured using the aws_inspector2_organization_configuration resource. Leveraging the variables above, the resource can be defined as follows:

resource "aws_inspector2_organization_configuration" "this" {
  provider = aws.audit
  auto_enable {
    ec2         = var.enable_ec2
    ecr         = var.enable_ecr
    lambda      = var.enable_lambda
    lambda_code = var.enable_lambda_code && var.enable_lambda
  }
  depends_on = [aws_inspector2_delegated_admin_account.audit]
}

Note that for AWS Lambda code scanning (lambda_code), AWS Lambda standard scanning (lambda) is a prerequisite, so we need to check both variables to enable it.

Now let's address the existing member accounts.

Activating scanning for existing member accounts

Unlike GuardDuty, the Inspector organization configuration does not support auto-enablement for existing member accounts, so we need to separately manage the member accounts. The strategy is to get the list of active member accounts from the organization, which we can use with the Inspector Terraform resources, including the aws_inspector2_enabler resource. We can exclude the Audit account since that is managed separately. To get the list of member accounts in the organization, we can use the aws_organizations_organization data source.

Furthermore, the aws_inspector2_enabler resource's resource_types argument takes a list of strings that represent the scan types to enable. Since the variables we defined earlier are boolean variables, we need a bit of function magic to create the list of scans to enable based on the variables.

The Terraform configuration that addresses the above requirements can be defined as follows:

data "aws_organizations_organization" "this" {
  provider = aws.management
}

locals {
  enabler_resource_types = compact([
    var.enable_ec2 ? "EC2" : null,
    var.enable_ecr ? "ECR" : null,
    var.enable_lambda ? "LAMBDA" : null,
    var.enable_lambda_code && var.enable_lambda ? "LAMBDA_CODE" : null,
  ])

  member_account_ids = [for account in data.aws_organizations_organization.this.accounts : account.id if account.status == "ACTIVE" && account.id != data.aws_caller_identity.audit.account_id]
}

Member accounts are not automatically associated with the delegated administrator account, so they must first be associated using the aws_inspector2_member_association resource.

Using the for_each meta-argument, we can define a single resource to associate all member accounts with the previously defined member_account_ids local value:

resource "aws_inspector2_member_association" "members" {
  provider   = aws.audit
  for_each   = toset(local.member_account_ids)
  account_id = each.key
  depends_on = [aws_inspector2_delegated_admin_account.audit]
}

Lastly, we can enable Inspector scans in the member accounts using the aws_inspector2_enabler resource. Although the account_ids argument can take the list of member accounts, it is more flexible to have one resource per account. Thus, using for_each and the local values, the resource can be defined as follows:

resource "aws_inspector2_enabler" "members" {
  provider       = aws.audit
  for_each       = toset(local.member_account_ids)
  account_ids    = [each.key]
  resource_types = local.enabler_resource_types
  depends_on     = [aws_inspector2_member_association.members]
}

Now that the Terraform configuration is fully defined, you can apply it to establish the Audit account as the delegated administration and centrally manage Inspector settings for both new and existing accounts.

Caveats about deactivating Inspector in member accounts

Among the AWS security services, Inspector has the least sophisticated API for organizational management. The mix between auto-enablement for new member accounts and explicit enablement for existing member accounts complicates how they are managed in Terraform, particularly if you are trying to disable Inspector via terraform destroy.

Consider the case where a new member account is added and auto-enablement is applied to this account. If you run terraform destroy as-is, Terraform is not aware of the new member account, and thus Inspector cannot be deactivated in this account. You must manually deactivate the account in each applied region.

Alternatively, you can first run terraform apply so that the aws_inspector2_member_association and aws_inspector2_enabler resource instances are created, then run terraform destroy to properly clean up. While this method works, you must keep track of when new member accounts are added so that you know when to run terraform apply to reconcile the Terraform resources with the updated organization.

In any case, be aware of this caveat and take one of the two approaches if you ever need to clean up Inspector resources.

Summary

In this blog post, you learned how to manage Amazon Inspector in AWS Organizations using Terraform. With a delegated administrator, Inspector can be auto-enabled for new member accounts, while existing member accounts are dynamically associated and configured with the desired scan types. If you have also configured AWS Security Hub to operate at the organization level, you can manage Inspector findings across accounts and regions, thereby streamlining your security operations.

If you are interested in this type of content, be sure to read other posts on the Avangards Blog, where I share tips and deep dives on AWS, Terraform, and beyond. Thank you, and enjoy the rest of your day!

In the previous blog post, Adding an Amazon Bedrock Knowledge Base to the Forex Rate Assistant, I explained how to create a Bedrock knowledge base and associate it with a Bedrock agent using the AWS Management Console, with a forex rate assistant as the use case example.

We also covered how to manage Bedrock agents with Terraform in another blog post, How To Manage an Amazon Bedrock Agent Using Terraform. In this blog post, we will extend that setup to also manage knowledge bases in Terraform. To begin, we will first examine the relevant AWS resources in the AWS Management Console.

Taking inventory of the required resources

Upon examining the knowledge base we previously built, we find that it comprises the following AWS resources:

1. The knowledge base itself;

2. The knowledge base service role that provides the knowledge base access to Amazon Bedrock models, data sources in S3, and the vector index;

   

3. The OpenSearch Serverless policies, collection, and the vector index;

   

4. The S3 bucket that acts as the data source

   

With this list of resources, along with those required by the agent to which the knowledge base will be attached, we can begin creating the Terraform configuration. Before diving into the setup, let's first take care of the prerequisites.

Defining variables for the configuration

For better manageability, we define some variables in a variables.tf file that we will reference throughout the Terraform configuration:

variable "kb_s3_bucket_name_prefix" {
  description = "The name prefix of the S3 bucket for the data source of the knowledge base."
  type        = string
  default     = "forex-kb"
}

variable "kb_oss_collection_name" {
  description = "The name of the OSS collection for the knowledge base."
  type        = string
  default     = "bedrock-knowledge-base-forex-kb"
}

variable "kb_model_id" {
  description = "The ID of the foundational model used by the knowledge base."
  type        = string
  default     = "amazon.titan-embed-text-v1"
}

variable "kb_name" {
  description = "The knowledge base name."
  type        = string
  default     = "ForexKB"
}

Defining the S3 and IAM resources

The knowledge base requires a service role, which can be created using the aws_iam_role resource as follows:

data "aws_caller_identity" "this" {}
data "aws_partition" "this" {}
data "aws_region" "this" {}

locals {
  account_id            = data.aws_caller_identity.this.account_id
  partition             = data.aws_partition.this.partition
  region                = data.aws_region.this.name
  region_name_tokenized = split("-", local.region)
  region_short          = "${substr(local.region_name_tokenized[0], 0, 2)}${substr(local.region_name_tokenized[1], 0, 1)}${local.region_name_tokenized[2]}"
}

resource "aws_iam_role" "bedrock_kb_forex_kb" {
  name = "AmazonBedrockExecutionRoleForKnowledgeBase_${var.kb_name}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "bedrock.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = local.account_id
          }
          ArnLike = {
            "aws:SourceArn" = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:knowledge-base/*"
          }
        }
      }
    ]
  })
}

With the service role in place, we can now proceed to define the corresponding IAM policy. As we define the configuration for creating resources that the knowledge base service role needs to access, we will consequently define the corresponding IAM policy using the aws_iam_role_policy resource. First, we create the IAM policy that provides access to the embeddings model. Since the foundation model is not created but referenced, we can use the aws_bedrock_foundation_model data source to obtain the ARN which we need:

data "aws_bedrock_foundation_model" "kb" {
  model_id = var.kb_model_id
}

resource "aws_iam_role_policy" "bedrock_kb_forex_kb_model" {
  name = "AmazonBedrockFoundationModelPolicyForKnowledgeBase_${var.kb_name}"
  role = aws_iam_role.bedrock_kb_forex_kb.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = "bedrock:InvokeModel"
        Effect   = "Allow"
        Resource = data.aws_bedrock_foundation_model.kb.model_arn
      }
    ]
  })
}

Next, we create the Amazon S3 bucket that acts as the data source for the knowledge base using the aws_s3_bucket resource. To adhere to security best practices, we also enable S3-SSE using the aws_s3_bucket_server_side_encryption_configuration resource and bucket versioning with the aws_s3_bucket_versioning resource as follows:

resource "aws_s3_bucket" "forex_kb" {
  bucket        = "${var.kb_s3_bucket_name_prefix}-${local.region_short}-${local.account_id}"
  force_destroy = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "forex_kb" {
  bucket = aws_s3_bucket.forex_kb.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_versioning" "forex_kb" {
  bucket = aws_s3_bucket.forex_kb.id
  versioning_configuration {
    status = "Enabled"
  }
  depends_on = [aws_s3_bucket_server_side_encryption_configuration.forex_kb]
}

Now that the S3 bucket is available, we can create the IAM policy that gives the knowledge base service role access to files for indexing:

resource "aws_iam_role_policy" "bedrock_kb_forex_kb_s3" {
  name = "AmazonBedrockS3PolicyForKnowledgeBase_${var.kb_name}"
  role = aws_iam_role.bedrock_kb_forex_kb.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "S3ListBucketStatement"
        Action   = "s3:ListBucket"
        Effect   = "Allow"
        Resource = aws_s3_bucket.forex_kb.arn
        Condition = {
          StringEquals = {
            "aws:PrincipalAccount" = local.account_id
          }
      } },
      {
        Sid      = "S3GetObjectStatement"
        Action   = "s3:GetObject"
        Effect   = "Allow"
        Resource = "${aws_s3_bucket.forex_kb.arn}/*"
        Condition = {
          StringEquals = {
            "aws:PrincipalAccount" = local.account_id
          }
        }
      }
    ]
  })
}

Defining the OpenSearch Serverless policy resources

The Bedrock console offers a quick create option that provisions an OpenSearch Serverless vector store on our behalf as the knowledge base is created. Since the documentation for creating the vector index in OpenSearch Serverless is a bit open-ended, we can refer to the resources from the quick create option to supplement.

First, we configure permissions by defining a data access policy for the vector search collection. The data access policy from the quick create option is defined as follows:

This data access policy provides read and write permissions to the vector search collection and its indices to the knowledge base execution role and the creator of the policy.

Using the corresponding aws_opensearchserverless_access_policy resource, we can define the policy as follows:

resource "aws_opensearchserverless_access_policy" "forex_kb" {
  name = var.kb_oss_collection_name
  type = "data"
  policy = jsonencode([
    {
      Rules = [
        {
          ResourceType = "index"
          Resource = [
            "index/${var.kb_oss_collection_name}/*"
          ]
          Permission = [
            "aoss:CreateIndex",
            "aoss:DeleteIndex",
            "aoss:DescribeIndex",
            "aoss:ReadDocument",
            "aoss:UpdateIndex",
            "aoss:WriteDocument"
          ]
        },
        {
          ResourceType = "collection"
          Resource = [
            "collection/${var.kb_oss_collection_name}"
          ]
          Permission = [
            "aoss:CreateCollectionItems",
            "aoss:DescribeCollectionItems",
            "aoss:UpdateCollectionItems"
          ]
        }
      ],
      Principal = [
        aws_iam_role.bedrock_kb_forex_kb.arn,
        data.aws_caller_identity.this.arn
      ]
    }
  ])
}

Note that aoss:DeleteIndex was added to the list because this is required for cleanup by Terraform via terraform destroy.

Next, we need an encryption policy that assigns an encryption key to a collection for data protection at rest. The encryption policy from the quick create option is defined as follows:

This encryption policy simply assigns an AWS-owned key to the vector search collection. Using the aws_opensearchserverless_security_policy resource with an encryption type, we can define the policy as follows:

resource "aws_opensearchserverless_security_policy" "forex_kb_encryption" {
  name = var.kb_oss_collection_name
  type = "encryption"
  policy = jsonencode({
    Rules = [
      {
        Resource = [
          "collection/${var.kb_oss_collection_name}"
        ]
        ResourceType = "collection"
      }
    ],
    AWSOwnedKey = true
  })
}

Lastly, we need a network policy which defines whether a collection is accessible publicly or privately. The network policy from the quick create option is defined as follows:

his network policy allows public access to the vector search collection's API endpoint and dashboard over the internet. Using the aws_opensearchserverless_security_policy resource with an network type, we can define the policy as follows:

resource "aws_opensearchserverless_security_policy" "forex_kb_network" {
  name = var.kb_oss_collection_name
  type = "network"
  policy = jsonencode([
    {
      Rules = [
        {
          ResourceType = "collection"
          Resource = [
            "collection/${var.kb_oss_collection_name}"
          ]
        },
        {
          ResourceType = "dashboard"
          Resource = [
            "collection/${var.kb_oss_collection_name}"
          ]
        }
      ]
      AllowFromPublic = true
    }
  ])
}

With the prerequisite policies in place, we can now create the vector search collection and the index.

Defining the OpenSearch Serverless collection and index resources

Creating the collection in Terraform is straightforward using the aws_opensearchserverless_collection resource:

resource "aws_opensearchserverless_collection" "forex_kb" {
  name = var.kb_oss_collection_name
  type = "VECTORSEARCH"
  depends_on = [
    aws_opensearchserverless_access_policy.forex_kb,
    aws_opensearchserverless_security_policy.forex_kb_encryption,
    aws_opensearchserverless_security_policy.forex_kb_network
  ]
}

The knowledge base service role also needs access to the collection, which we can provide using the aws_iam_role_policy similar to before:

resource "aws_iam_role_policy" "bedrock_kb_forex_kb_oss" {
  name = "AmazonBedrockOSSPolicyForKnowledgeBase_${var.kb_name}"
  role = aws_iam_role.bedrock_kb_forex_kb.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = "aoss:APIAccessAll"
        Effect   = "Allow"
        Resource = aws_opensearchserverless_collection.forex_kb.arn
      }
    ]
  })
}

Creating the index in Terraform is however more complex, since it is not an AWS resource but an OpenSearch construct. Looking at CloudTrail events, there wasn't any event that correspond to an AWS API call that would create the index. However, observing the network traffic in the Bedrock console did reveal a request to the OpenSearch collection's API endpoint to create the index. This is what we want to port to Terraform.

Luckily, there is an OpenSearch Provider maintained by OpenSearch that we can use. To connect to the vector search collection, we provide the endpoint URL and credentials in the provider block. The provider has first-class support for AWS, so credentials can be provided implicitly similar to the Terraform AWS Provider. The resulting provider definition is as follows:

provider "opensearch" {
  url         = aws_opensearchserverless_collection.forex_kb.collection_endpoint
  healthcheck = false
}

Note that the healthcheck argument is set to false because the client health check does not really work with OpenSearch Serverless.

To get the index definition, we can examine the collection in the OpenSearch Service Console:

We can create the index using the opensearch_index resource with the same specifications:

resource "opensearch_index" "forex_kb" {
  name                           = "bedrock-knowledge-base-default-index"
  number_of_shards               = "2"
  number_of_replicas             = "0"
  index_knn                      = true
  index_knn_algo_param_ef_search = "512"
  mappings                       = <<-EOF
    {
      "properties": {
        "bedrock-knowledge-base-default-vector": {
          "type": "knn_vector",
          "dimension": 1536,
          "method": {
            "name": "hnsw",
            "engine": "faiss",
            "parameters": {
              "m": 16,
              "ef_construction": 512
            },
            "space_type": "l2"
          }
        },
        "AMAZON_BEDROCK_METADATA": {
          "type": "text",
          "index": "false"
        },
        "AMAZON_BEDROCK_TEXT_CHUNK": {
          "type": "text",
          "index": "true"
        }
      }
    }
  EOF
  force_destroy                  = true
  depends_on                     = [aws_opensearchserverless_collection.forex_kb]
}

Note that the dimension is set to 1536, which is the value required for the Titan G1 Embeddings - Text model.

Before we move on, you must know about an issue with the Terraform OpenSearch provider that caused me a lot of headache. When I was testing the Terraform configuration, the opensearch_index resource kept failing because the provider could not seemingly authenticate against the collection's endpoint URL. After a long debugging session, I was able to find a GitHub issue in the Terraform OpenSearch Provider repository that mentions the cryptic "EOF" error that was present. The issue mentions that the bug is related to OpenSearch Serverless and an earlier provider version, v2.2.0, does not have the problem. Consequently, I was able to work around the problem by using this specific version of the provider:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.48"
    }
    opensearch = {
      source  = "opensearch-project/opensearch"
      version = "= 2.2.0"
    }
  }
  required_version = "~> 1.5"
}

Hopefully letting you in on this tip will save you hours of troubleshooting.

Defining the knowledge base resource

With all dependent resources in place, we can now proceed to create the knowledge base. However, there is the matter of eventual consistency with IAM resources that we first need to address. Since Terraform creates resources in quick succession, there is a chance that the configuration of the knowledge base service role is not propagated across AWS endpoints before it is used by the knowledge base during its creation, resulting in temporary permission issues. What I observed during testing is that the permission error is usually related to the OpenSearch Serverless collection.

To mitigate this, we add a delay using the time_sleep resource in the Time Provider. The following configuration will add a 20-second delay after the IAM policy for the OpenSearch Serverless collection is created:

resource "time_sleep" "aws_iam_role_policy_bedrock_kb_forex_kb_oss" {
  create_duration = "20s"
  depends_on      = [aws_iam_role_policy.bedrock_kb_forex_kb_oss]
}

Now we can create the knowledge base using the aws_bedrockagent_knowledge_base resource as follows:

resource "aws_bedrockagent_knowledge_base" "forex_kb" {
  name     = var.kb_name
  role_arn = aws_iam_role.bedrock_kb_forex_kb.arn
  knowledge_base_configuration {
    vector_knowledge_base_configuration {
      embedding_model_arn = data.aws_bedrock_foundation_model.kb.model_arn
    }
    type = "VECTOR"
  }
  storage_configuration {
    type = "OPENSEARCH_SERVERLESS"
    opensearch_serverless_configuration {
      collection_arn    = aws_opensearchserverless_collection.forex_kb.arn
      vector_index_name = "bedrock-knowledge-base-default-index"
      field_mapping {
        vector_field   = "bedrock-knowledge-base-default-vector"
        text_field     = "AMAZON_BEDROCK_TEXT_CHUNK"
        metadata_field = "AMAZON_BEDROCK_METADATA"
      }
    }
  }
  depends_on = [
    aws_iam_role_policy.bedrock_kb_forex_kb_model,
    aws_iam_role_policy.bedrock_kb_forex_kb_s3,
    opensearch_index.forex_kb,
    time_sleep.aws_iam_role_policy_bedrock_kb_forex_kb_oss
  ]
}

Note that time_sleep.aws_iam_role_policy_bedrock_kb_forex_kb_oss is in the depends_on list - this is how the aforementioned delay is enforced before the knowledge base is created by Terraform.

We also need to add the data source to the knowledge base using the aws_bedrock_data_source resource as follows:

resource "aws_bedrockagent_data_source" "forex_kb" {
  knowledge_base_id = aws_bedrockagent_knowledge_base.forex_kb.id
  name              = "${var.kb_name}DataSource"
  data_source_configuration {
    type = "S3"
    s3_configuration {
      bucket_arn = aws_s3_bucket.forex_kb.arn
    }
  }
}

Voila! We have created a stand-alone Bedrock knowledge base using Terraform! All that remains is to attach the knowledge base to an agent (the forex assistant in our case) to extend the solution.

Integrating the knowledge base and agent resources

For your convenience, you can use the Terraform configuration from the blog post How To Manage an Amazon Bedrock Agent Using Terraform to create the rate assistant. It can be found in the 1_basic directory in this GitHub repository.

Once you incorporate this Terraform configuration with the knowledge base you’ve been developing, we use the new aws_bedrockagent_agent_knowledge_base_association resource to associate the knowledge base with the agent:

resource "aws_bedrockagent_agent_knowledge_base_association" "forex_kb" {
  agent_id             = aws_bedrockagent_agent.forex_asst.id
  description          = file("${path.module}/prompt_templates/kb_instruction.txt")
  knowledge_base_id    = aws_bedrockagent_knowledge_base.forex_kb.id
  knowledge_base_state = "ENABLED"
}

For better organization, we will keep the knowledge base description in a text file called kb_instruction.txt in the prompt_templates folder. The file contains the following text:

Use this knowledge base to retrieve information on foreign currency exchange, such as the FX Global Code.

Lastly, we explained in the previous blog post that the agent must be prepared after changes are made. We used a null_resource to trigger the prepare action, so we will continue to use the same strategy for the knowledge base association by adding an explicit dependency:

resource "null_resource" "forex_asst_prepare" {
  triggers = {
    forex_api_state = sha256(jsonencode(aws_bedrockagent_agent_action_group.forex_api))
    forex_kb_state  = sha256(jsonencode(aws_bedrockagent_knowledge_base.forex_kb))
  }
  provisioner "local-exec" {
    command = "aws bedrock-agent prepare-agent --agent-id ${aws_bedrockagent_agent.forex_asst.id}"
  }
  depends_on = [
    aws_bedrockagent_agent.forex_asst,
    aws_bedrockagent_agent_action_group.forex_api,
    aws_bedrockagent_knowledge_base.forex_kb
  ]
}

Testing the configuration

Now, the moment of truth. We can apply the full Terraform configuration and make sure that it is working properly. My run took several minutes, with the majority of the time spent on creating the OpenSearch Serverless collection. Here is an excerpt of the output for reference:

In the Bedrock console, we can see that the agent ForexAssistant is ready for testing. But we first need to upload the FX Global Code PDF file to the S3 bucket and do a data source sync. For details on these steps, refer to the blog post Adding an Amazon Bedrock Knowledge Base to the Forex Rate Assistant.

Using the test chat interface, I asked:

What is the FX Global Code?

It responded with an explanation that contains citations, indicating that the information was obtained from the knowledge base.

For good measure, we will also ask the forex assistant for an exchange rate:

What is the exchange rate from US Dollar to Canadian Dollar?

It responded with the latest exchange rate as expected:

And that's a wrap! Don't forget to run terraform destroy when you are done, since there is a running cost for the OpenSearch Serverless collection.

Summary

In this blog post, we developed the Terraform configuration for the knowledge base that enhances the forex rate assistant which we created interactively in the blog post Adding an Amazon Bedrock Knowledge Base to the Forex Rate Assistant. I hope the explanations on key points and solutions to various issues in this blog post help you fast-track your IaC development for Amazon Bedrock solutions.

I will continue to evaluate different features of Amazon Bedrock, such as Guardrails for Amazon Bedrock, and streamlining the data ingestion process for knowledge bases. Please look forward for more helpful content on this topic as well as many others in the Avangards Blog. Happy learning!

Since I began covering the implementation of security controls in AWS, I have provided walkthroughs on configuring Amazon GuardDuty and AWS Security Hub in a centralized setup using Terraform. In this blog post, we will explore another security service: AWS IAM Access Analyzer. This service helps identify unintended external access or unused access within your organization. Setting up IAM Access Analyzer is simpler than the other services, so let's dive right in!

About the use case

AWS Identity and Access Management (IAM) Access Analyzer is a feature of AWS IAM that identifies resources shared with external entities and detects unused access, enabling you to mitigate any unintended or obsolete permissions.

IAM Access Analyzer can be used in AWS Organizations, allowing analyzers that use the organization as the zone of trust to be managed by either the management account or a delegated administrator account. This enables the consolidation of findings, which can then be ingested by AWS Security Hub in a centralized setup.

Since it is increasingly common to establish an AWS landing zone using AWS Control Tower, we will use the standard account structure in a Control Tower landing zone to demonstrate how to configure IAM Access Analyzer in Terraform:

The relevant accounts for our use case in the landing zone are:

1. The Management account for the organization where AWS Organizations is configured. For details, refer to Settings for IAM Access Analyzer.

2. The Audit account where security and compliance services are typically centralized in a Control Tower landing zone.

The objective is to delegate IAM Access Analyzer administrative duties from the Management account to the Audit account, after which all organization configurations are managed in the Audit account. With that said, let's see how we can achieve this using Terraform!

Designating an IAM Access Analyzer administrator account

The IAM Access Analyzer delegated administrator is configured in the Management account, so we need a provider associated with it in Terraform. To simplify the setup, we will use a multi-provider approach by defining two providers: one for the Management account and another for the Audit account. We will use AWS CLI profiles as follows:

provider "aws" {
  alias   = "management"
  # Use "aws configure" to create the "management" profile with the Management account credentials
  profile = "management" 
}

provider "aws" {
  alias   = "audit"
  # Use "aws configure" to create the "audit" profile with the Audit account credentials
  profile = "audit" 
}

Unlike other security services that have specific Terraform resources for designating a delegated administrator, this is done using the more general aws_organizations_delegated_administrator resource as follows:

data "aws_caller_identity" "audit" {
  provider = aws.audit
}

resource "aws_organizations_delegated_administrator" "this" {
  provider          = aws.management
  account_id        = data.aws_caller_identity.audit.account_id
  service_principal = "access-analyzer.amazonaws.com"
}

With the Audit account designated as the IAM Access Analyzer administrator, we can now create the analyzers for the organization.

Creating analyzers with organizational zone of trust

As mentioned earlier, there are two types of analyzers: external access and unused access. To make the setup more configurable, we will add some variables and keep them in a separate file called variables.tf. To create the external access analyzer with the organization as the zone of trust, we can define the Terraform configuration as follows:

# Defined in variables.tf

variable "org_external_access_analyzer_name" {
  description = "The name of the organization external access analyzer."
  type        = string
  default     = "OrgExternalAccessAnalyzer"
}

# Defined in main.tf

resource "aws_accessanalyzer_analyzer" "org_external_access" {
  provider      = aws.audit
  analyzer_name = var.org_external_access_analyzer_name
  type          = "ORGANIZATION"
  depends_on    = [aws_organizations_delegated_administrator.this]
}

Since the unused access analyzer is a paid feature, we ought to make it optional. The Terraform configuration can be defined in the following manner:

# Defined in variables.tf

variable "org_unused_access_analyzer_name" {
  description = "The name of the organization unused access analyzer."
  type        = string
  default     = "OrgUnusedAccessAnalyzer"
}

variable "eanble_unused_access" {
  description = "Whether organizational unused access analysis should be enabled."
  type        = bool
  default     = false
}

variable "unused_access_age" {
  description = "The specified access age in days for which to generate findings for unused access."
  type        = number
  default     = 90
}

resource "aws_accessanalyzer_analyzer" "org_unused_access" {
  provider      = aws.audit
  count         = var.eanble_unused_access ? 1 : 0
  analyzer_name = var.org_unused_access_analyzer_name
  type          = "ORGANIZATION_UNUSED_ACCESS"
  configuration {
    unused_access {
      unused_access_age = var.unused_access_age
    }
  }
  depends_on = [aws_organizations_delegated_administrator.this]
}

With the complete Terraform configuration, you can now apply it with the appropriate variable values to establish the Audit account as the delegated administrator and create the analyzers with the organization as the zone of trust.

Additional considerations

IAM Access Analyzer is a regional service, so you must create an analyzer in each region. However, it primarily applies to external access analysis, which examines the policies of regional resources such as S3 buckets and KMS keys. Since unused access analysis works with IAM users and roles, which are global resources, creating multiple unused access analyzers would only increase costs without adding value. Therefore, it is recommended to create one external access analyzer per region and only one unused access analyzer in the home region.

Another consideration is that there are times when the organizational zone of trust is not desirable. For example, if you wish to have full segregation of member accounts because they represent different tenants, then you would actually want analyzers created in each member account with itself as the zone of trust. This unfortunately would have to be managed at a per-account level.

Summary

In this blog post, you learned how to manage IAM Access Analyzer in AWS Organizations using Terraform by defining a delegated administrator and using analyzers with the organization as the zone of trust. If you have also configured AWS Security Hub to operate at the organization level, you can manage IAM Access Analyzer findings across accounts and regions, thereby streamlining your security operations.

I hope you find this blog post helpful. Be sure to keep an eye out for more how-to articles on configuring other AWS security services in Terraform, or learn about other topics like generative AI, on the Avangards Blog.

In the blog post Developing a Generic Streamlit UI to Test Amazon Bedrock Agents, I shared the design and source code of a basic yet functional UI for testing Bedrock agents. Since then, I've explored Knowledge Bases for Amazon Bedrock and shared my insights in another blog post, Adding an Amazon Bedrock Knowledge Base to the Forex Rate Assistant. If you haven't checked it out yet, I highly recommend doing so.

The Bedrock console offers additional features for testing agents that are integrated with knowledge bases, including citations in the responses and trace information about the retrieved results from knowledge bases:

With a bit of work, I have added similar support to the generic test UI and I am happy to share the updates in the GitHub repository.

Design overview

With the latest update, citations are now added to the response in a manner similar to how they are displayed in the Bedrock console:

The Agent for Bedrock Runtime API provides, for each citation, the start and end index of the text that references the source as well as the document location in S3. Through string manipulation, citation numbers are incorporated into the response text, and references are appended to the end of the text. Spacing is a bit finicky due to the use of markdown, so some HTML markups are used.

Another feature is the inclusion of citation details in the Trace section of the left pane:

Each citation block provides the raw output from the API response and includes the information mentioned earlier, as well as the raw results retrieved by the knowledge base.

Additionally, the trace blocks from the original test UI can now provide more detailed information for knowledge base invocations. You will find all references retrieved from the knowledge base, which form a superset of the citations in the final response after the model processes them.

Summary

With the improvements to the generic test UI outlined in this post, you should now be able to test any Bedrock agents with attached knowledge bases. I hope you find this update helpful and look forward to further enhancements as I continue on my Amazon Bedrock journey.

Be sure to explore other posts on the Avangards Blog to learn more about generative AI in AWS, Terraform, and other technical topics. Have a great day!

In our journey of experimenting with Amazon Bedrock up to this point, we have built a basic forex assistant as the basis for further enhancements to evaluate various Bedrock features and generative AI (gen AI) techniques. Our next step is to integrate a knowledge base to the agent, so that it can provide information about foreign currency exchange in general.

In this blog post, we will define a representative use case for a RAG scenario for the forex rate agent, build a forex knowledge base, and attach it to the agent. Accuracy and performance of a gen AI application is also essential, so we'll conduct some tests and discuss challenges associated with RAG workflows.

About Knowledge Bases for Amazon Bedrock

Knowledge Bases for Amazon Bedrock is a service that provides managed capability for implementing Retrieval Augmented Generation (RAG) workflows. Knowledge bases can be integrated with Bedrock agents to seamlessly enable RAG functionality, or be used as a component in custom AI-enabled applications using API.

Knowledge Bases for Amazon Bedrock automates the ingestion of source documents, by generating embeddings with a foundation model, such as Amazon Titan or Cohere Embed, and storing them in a supported vector store as depicted in the following diagram:

To keep things simple, the service provides a quick start option that provisions on your behalf an Amazon OpenSearch Serverless vector database for its use.

Aside from native integration into a Bedrock agent, the Agents for Amazon Bedrock Runtime API offers both the ability to perform raw text and semantic search, and the ability to retrieve and generate a response with a foundation model, on knowledge bases. The latter allows the community to provide tighter integration in frameworks such as LangChain and LlamaIndex to simplify RAG scenarios. The runtime flow is shown in this diagram:

Enhancing the forex rate assistant use case

In the blog post Building a Basic Forex Rate Assistant Using Agents for Amazon Bedrock, we created a basic forex assistant that helps users look up the latest forex rates. It would be helpful if the assistant can also answer other questions on the broader topic.

While information about the history of forex would be useful, the Claude models already possess such knowledge so it would not make a great use case for knowledge bases. As I searched for more specific subtopics, I found the FX Global Code, a set of common guidelines developed by the Global Foreign Exchange Committee (GFXC) which establishes universal principles to uphold integrity and ensure the effective operation of the wholesale FX market. The FX Global Code is conveniently available in PDF format, which is perfect for ingestion by the knowledge base.

Requesting model access and creating the S3 bucket for document ingestion

Let's start with the prerequisites by requesting for model access. For the forex knowledge base, we will be using the Titan Embeddings G1 - Text model. You can review the model pricing information here.

Next, we need to create the S3 bucket from which the knowledge base will ingest source documents. We can quickly do so in the S3 Console with the following settings:

• Bucket name: forex-kb-<region>-<account_id> (such as forex-kb-use1-123456789012)

• Block all public access: Checked (by default)

• Bucket versioning: Enable

• Default encryption: SSE-S3 (by default)

Once the S3 bucket is created, download the FX Global Code PDF file and upload it to the bucket:

This is sufficient for our purpose. For more information on other supported document formats and adding metadata for the filtering feature, refer to the Amazon Bedrock user guide.

Creating a knowledge base along with a vector database

Next, we can create the knowledge base in the Amazon Bedrock console following the steps below:

1. Select Knowledge bases in the left menu.

2. On the Knowledge bases page, click Create knowledge base.

3. In Step 1 of the Create knowledge base wizard, enter the following information and click Next:

   • Knowledge base name: ForexKB

   • Knowledge base description: A knowledge base with information on foreign currency exchange.

4. In Step 2 of the wizard, enter the following information and click Next:

   • Data source name: ForexKBDataSource

   • S3 URI: Browse and select the S3 bucket that we created earlier

5. In Step 3 of the wizard, enter the following information and click Next:

   • Embeddings model: Titan Embeddings G1 - Text v1.2

   • Vector database: Quick create a new vector store

6. In Step 4 of the wizard, click Create knowledge base.

The knowledge base and the vector database, which is an Amazon OpenSearch Server collection, will take a few minutes to create. When they are ready, you'll be directed to the knowledge base page, where you will be prompted to synchronize sync the data source. To do so, scroll down to the Data source section, select the radio button beside the data source name, and click Sync:

It will take less than a minute to complete, since we only have a single moderately-sized PDF document. Now the knowledge base is ready for some validation.

Testing the knowledge base

A knowledge base must provide accurate results, so we need to validate it against information we know exists out of the source documents. This can be done using the integrated test pane. To simulate an end-to-end test for the RAG scenario, configure the test environment in the Bedrock console as follows:

1. Enable the Generate response option (which should already be enabled by default).

2. Click Select model. In the new dialog, select the Claude 3 Haiku model and click Apply.

3. Click on the button with the three sliders, which opens the Configuration page. This should expand the test pane so you have more screen real estate to work with.

I've prepared a couple of questions after skimming through the FX Global Code PDF file. Let's start by asking a basic question:

What is the FX Global Code?

The knowledge base responded with an answer that's consistent with the text on page 3 of the document.

To see the underlying search results which the model used to generate the response, click Show source details. Similar to agent trace, we can view the source chunks that are related to our question, and the associated raw text and metadata (which is mainly the citation information). Some source chunks refer to the table of contents, which some refers to the same passage from page 3 of the document.

Next, let's ask something more specific, namely to give me information on a specific principle such as principle 15 which is on page 33 of the PDF file:

What is principle 15 in the FX Global Code?

Interestingly, the knowledge base doesn't seem to know the answer:

If I force the knowledge base to use hybrid search, which combines both sematic and text search for better response, some source chucks were fetched but it does not seem to include one with the text from page 33.

Since there are exactly five results, I figured that it might be limited by the maximum number. After increasing it to an arbitrary 20, the knowledge base finally returned a good response with the default search option:

This goes to show that just like agents, knowledge bases must be tested and fine-tuned extensively to improve accuracy. The embedding model as well as the underlying vector store may also play a part in the overall behavior of the knowledge base.

In any case, you've successfully created a knowledge base using Knowledge Bases for Amazon Bedrock, which can be integrated with your gen AI applications. To complete our experimentation, let's now integrate it with our forex rate assistant.

Integrating the knowledge base to the forex rate assistant

If you haven't already done so, please follow the blog post Building a Basic Forex Rate Assistant Using Agents for Amazon Bedrock to create the forex rate assistant manually, or use the Terraform configuration from the blog post How To Manage an Amazon Bedrock Agent Using Terraform to deploy it.

Once the agent is ready, we can associate the knowledge base to it using the steps below in the Bedrock Console:

1. Select Agents in the left menu.

2. On the Agents page, click ForexAssistant to open it.

3. On the agent page, click Edit in Agent Builder.

4. On the Agent builder page, scroll down to the Knowledge bases section and click Add.

5. On the Add knowledge base page, enter the following information and click Add:

   • Select knowledge base: ForexKB

   • Knowledge base instructions for Agent: Use this knowledge base to retrieve information on foreign currency exchange, such as the FX Global Code.

     

6. Click Save and exit.

Once the knowledge base is added, prepare the agent and ask the same first question as before to validate the integration:

What is the FX Global Code?

The agent responded with a decent answer. In the trace, we can see that the agent invoked the knowledge base as part of its toolset and retrieved the results for its use.

We also want to ask the agent to fetch some exchange rate to ensure that the functionality is still working:

What is the exchange rate from EUR to CAD?

The agent responded with the rate fetched from the ForexAPI action group, which is what we expected.

However, we run into issues when asking the second question from before:

What is principle 15 in the FX Global Code?

The agent responded with the inferior answer since we did not adjust the maximum number of retrieval results for the knowledge base.

Unfortunately, our issue now is that there is no way that I know of to provide the knowledge base configuration in the agent, so we are stuck. At this point, there's nothing we can do other than opening an AWS support case to inquire about the lack of support... That being said, another angle to look at the problem is that the quality of the source could also affect the knowledge base's search accuracy, which brings us to the topic of common RAG challenges.

Common RAG challenges

Let's examine the source chunk from the correct answer for the "principle 15" question from our knowledge base test:

This is also the text that was extracted from the PDF for embedding. Comparing it to the corresponding page in the PDF file, notice the following:

1. The chunk text includes information such as headers and "breadcrumbs" that are not related to the main content.

2. The text does not capture the context of the elements in the passage, such as the principle title in the red box and the principle summary in italic.

It's fair to think that undesirable artifacts and lack of structural context would impact search accuracy, performance, and ultimately cost. Consequently, it makes sense to perform some data pre-processing before passing the source documents to the RAG workflow. Third-party APIs and tools, such as LlamaParse and LayoutPDFReader, can help with pre-processing PDF data, however keep in mind that source documents may take any forms and there is no one-size-fits-all solution. You may have to resort to developing custom processes for pre-processing and search your unique data.

There are other challenges in building RAG-based LLM applications and proposed solutions which you should be aware of. However, some of them cannot be implemented in a managed solution such as Knowledge Bases for Amazon Bedrock, in which case you may need to build a custom solution yourself if you have a genuine need to address them. Such is the eternal quest to balance between effort and quality.

Don't forget to delete the OpenSearch Serverless collection

Be aware that Knowledge Bases for Amazon Bedrock does not delete the vector database for you. Since the OpenSearch Serverless collection consumes at least one OpenSearch Compute Unit (OCU) which is charged by the hour, you will incur a running cost for as long as the collection exists. Consequently, ensure that you manually delete the collection after you have deleted the knowledge base and other associated artifacts.

Summary

In this blog post, we created a knowledge base using Knowledge Bases for Amazon Bedrock which we integrate into the forex rate assistant to allow it to answer questions about the FX Global Code. Through some testing of the solution, we experienced some common challenges for RAG solutions and potential mitigation strategies. Although some of them are not applicable to Bedrock knowledge bases since it abstracts the implementation details, thus highlighting the potential need for a custom solution for more demanding scenarios.

My next step is to enhance the Terraform configuration for the forex rate assistant to provision and integrate the knowledge base, and to enhance the Streamlit test app to display citations from knowledge base searches. Be sure to follow the Avangards Blog as I continue my journey on building gen AI applications using Amazon Bedrock and other AWS services. Thanks for reading and stay curious!

Earlier I've published the blog post How To Manage Amazon GuardDuty in AWS Organizations Using Terraform which is essential in establishing threat detection as part of a security baseline, such as the AWS Security Baseline which I covered extensively in the blog series How to implement the AWS Startup Security Baseline (SSB) using Terraform.

Similarly, a good security baseline must include the means to manage the security posture, achieved using Security Hub in AWS. In this blog post, I will walk you through the steps to configuring Security Hub with central configuration in Terraform.

About the use case

AWS Security Hub is a security service that helps you manage security posture by collecting security data from AWS and third-party sources, and enabling analysis and remediation of security issues that are found.

Late last year, AWS introduced new central configuration capabilities in AWS Security Hub in the form of Security Hub configuration policies (SHCPs). With SHCPs, we can customize many aspects of the Security Hub configuration which can be consistently applied to all members of the organization. This addresses many challenges with managing Security Hub across an organization which I experienced first hand last year. It was practically futile to build Security Hub enablement into AWS Control Tower Account Factory for Terraform (AFT)! As this is the new best practice, we'll be using this feature.

Since it is increasingly common to establish an AWS landing zone using AWS Control Tower, we will use the standard account structure in a Control Tower landing zone to demonstrate how to configure Security Hub in Terraform:

The relevant accounts for our use case in the landing zone are:

1. The Management account for the organization where AWS Organizations is configured. For details, refer to Integrating Security Hub with AWS Organizations.

2. The Audit account where security and compliance services are typically centralized in a Control Tower landing zone.

The objective is to delegate Security Hub administrative duties from the Management account to the Audit account, after which all organization configurations are managed in the Audit account. With that said, let's see how we can achieve this using Terraform!

Designating a Security Hub administrator account

Security Hub delegated administrator is configured in the Management account, so we need a provider associated with it in Terraform. To keep things simple, we will take a multi-provider approach by defining two providers, one for the Management account and another for the Audit account, using AWS CLI profiles as follows:

provider "aws" {
  alias   = "management"
  # Use "aws configure" to create the "management" profile with the Management account credentials
  profile = "management" 
}

provider "aws" {
  alias   = "audit"
  # Use "aws configure" to create the "audit" profile with the Audit account credentials
  profile = "audit" 
}

We can then use the aws_securityhub_organization_admin_account resource to set the delegated administrator. However, I noticed the following in the Audit account:

• After this resource is created, Security Hub will be enabled with the default standards (AWS Foundational Security Best Practices v1.0.0 and CIS AWS Foundations Benchmark v1.2.0).

• When the resource is deleted, Security Hub remains enabled.

These side effects are undesirable since ideally, we want full control over the lifecycle and configuration of Security Hub in Terraform. To address this issue, we will preemptively enable Security Hub in the Audit account using the aws_securityhub_account resource. Later we will also apply the same configuration policy that will be associated to the organization.

data "aws_caller_identity" "audit" {
  provider = aws.audit
}

resource "aws_securityhub_account" "audit" {
  provider                 = aws.audit
  enable_default_standards = false
}

resource "aws_securityhub_organization_admin_account" "this" {
  provider         = aws.management
  admin_account_id = data.aws_caller_identity.audit.account_id
  depends_on       = [aws_securityhub_account.audit]
}

With the Audit account designated as the Security Hub administrator, we can now manage the organization configuration.

Configuring cross-region aggregation

Security Hub provides a cross-region aggregation feature that centralizes findings, finding updates, insights, control compliance statuses, and security scores from multiple regions into a single region. Being able to review all findings in one place is incredibly useful for security analysts. We can enable this feature for all regions using the aws_securityhub_finding_aggregator resource in Terraform as follows:

resource "aws_securityhub_finding_aggregator" "this" {
  provider     = aws.audit
  linking_mode = "ALL_REGIONS"
  depends_on   = [aws_securityhub_account.audit]
}

Enabling central configuration

First, we need to apply the organization configuration to enable central configuration. Since the settings are defined in an configuration policy, we need to disable all settings that are related to local configuration. We will achieve this using the aws_securityhub_organization_configuration resource:

resource "aws_securityhub_organization_configuration" "this" {
  provider              = aws.audit
  auto_enable           = false
  auto_enable_standards = "NONE"
  organization_configuration {
    configuration_type = "CENTRAL"
  }
  depends_on = [
    aws_securityhub_organization_admin_account.this,
    aws_securityhub_finding_aggregator.this
  ]
}

Creating and associating a configuration policy

With the organization configuration primed, we can now create and associate a configuration policy. This can be done with the aws_securityhub_configuration_policy resource and the aws_securityhub_configuration_policy_association resource.

For illustration, let's assume that we want to enable only the CIS AWS Foundations Benchmark v1.4.0 standard across the organization. We also want to disable the control [IAM.6] Hardware MFA should be enabled for the root user.

The configuration policy can be defined in Terraform as follows:

data "aws_region" "audit" {
  provider = aws.audit
}

data "aws_partition" "audit" {
  provider = aws.audit
}

resource "aws_securityhub_configuration_policy" "this" {
  provider    = aws.audit
  name        = "ExamplePolicy"
  description = "This is an example SHCP."
  configuration_policy {
    service_enabled       = true
    enabled_standard_arns = ["arn:${data.aws_partition.audit.partition}:securityhub:${data.aws_region.audit.name}::standards/cis-aws-foundations-benchmark/v/1.4.0"]
    security_controls_configuration {
      disabled_control_identifiers = ["IAM.6"]
    }
  }
  depends_on = [aws_securityhub_organization_configuration.this]
}

Lastly, we will associate this configuration policy to the entire organization:

data "aws_organizations_organization" "this" {
  provider = aws.management
}

resource "aws_securityhub_configuration_policy_association" "org" {
  provider = aws.audit
  target_id = data.aws_organizations_organization.this.roots[0].id
  policy_id = aws_securityhub_configuration_policy.this.id
}

Before you apply the Terraform configuration, there is one issue which I found while cleaning up my environment that should be addressed in the Terraform configuration.

Addressing a state-related issue which causes policy deletion to fail

While cleaning up my environment, I encountered the following state-related error when attempting to destroy the aws_securityhub_configuration_policy resource:

aws_securityhub_configuration_policy_association.org: Destroying... [id=r-lzgl]
aws_securityhub_configuration_policy_association.org: Destruction complete after 2s
aws_securityhub_configuration_policy.this: Destroying... [id=f7bf343f-af38-4b1d-9116-73f43cfb5d61]
╷
│ Error: deleting Security Hub Configuration Policy (f7bf343f-af38-4b1d-9116-73f43cfb5d61): operation error SecurityHub: DeleteConfigurationPolicy, https response error StatusCode: 409, RequestID: 06f4448f-4133-412a-b89b-bda896f7fa08, ResourceConflictException: Policy f7bf343f-af38-4b1d-9116-73f43cfb5d61 is associated with one or more accounts or organizational units. You must disassociate the policy before you can delete it.

However, you can see in the first two lines in the output that the configuration policy association is already destroyed before the attempt to destroy the policy.

After examining the Terraform resource code and the AWS API contract, I found that the StartConfigurationPolicyDisassociation API action does not report the disassociation status, nor is there another API action that can query the status. So this is not a Terraform AWS Provider bug per se and having the issue addressed upstream seems unlikely.

As a workaround, I turned to the time_sleep resource that can add a wait time for resource destruction. Through trial and error, I learned that 10 seconds is sufficient for the state to be updated. So we can update the Terraform configuration as follows:

# Some wait time is needed to account for state changes after the configuration policy is disassociated
resource "time_sleep" "aws_securityhub_configuration_policy_this" {
  destroy_duration = "10s"
  depends_on       = [aws_securityhub_configuration_policy.this]
}

resource "aws_securityhub_configuration_policy_association" "org" {
  provider   = aws.audit
  target_id  = data.aws_organizations_organization.this.roots[0].id
  policy_id  = aws_securityhub_configuration_policy.this.id
  depends_on = [time_sleep.aws_securityhub_configuration_policy_this]
}

With this change, the full Terraform configuration can be destroyed successfully.

With the complete Terraform configuration, you can now apply it to establish the Audit account as the delegated administrator and apply the SHCP to all accounts and all regions (as per the finding aggregator settings).

Caveats about disabling Security Hub in member accounts

Due to the design of the Security Hub API and the Terraform resources, Security Hub will not be disabled in the member accounts when you run terraform destroy. Normally this wouldn't be a problem for a production landing zone. However, if you are only testing, this could lead to unexpected costs especially when left running in all accounts and all regions.

Since it would be difficult to disable Security Hub in individual account, a smarter way would be to disable Security Hub using the SHCP. This can be done by changing the aws_securityhub_configuration_policy.this resource definition to the following:

resource "aws_securityhub_configuration_policy" "this" {
  provider    = aws.audit
  name        = "ExamplePolicy"
  description = "This is an example SHCP."
  configuration_policy {
    service_enabled = false
  }
  depends_on = [aws_securityhub_organization_configuration.this]
}

After you re-apply the Terraform configuration, Security Hub should be disabled in all accounts and all regions. Then you can safely run terraform destroy to remove the remaining Security Hub resources and configuration.

Summary

In this blog post, you learned how to implement central configuration to manage AWS Security Hub in AWS Organizations using Terraform. By consolidating all management work of all accounts in an organization and all regions into a delegated administrator account, you now have a single pane of glass to review and manage your cloud security posture.

For more tips and walkthroughs on AWS, Terraform, and more, please check out the Avangards Blog. Thanks for reading!

Introduction

In the earlier blog post Building a Basic Forex Rate Assistant Using Agents for Amazon Bedrock, I walked readers through the process of building and testing a Bedrock agent in the AWS Management Console. While the built-in test interface is great for validation as changes are made in the Bedrock console, it is not scalable to other team members such as testers who often don't have direct access to AWS.

Meanwhile, a developer workflow that does not require access to AWS Management Console may provide a better experience. As a developer, I appreciate having an integrated development environment (IDE) such as Visual Studio Code where I can code, deploy, and test in one place.

To address these two challenges, I decided to build a basic but functional UI for testing Bedrock agents. In this blog post, I share with readers the end product and some details about its design.

Design and implementation overview

The following is the list of requirements that I defined for the test UI:

• The design should be minimal but functional, since the focus is not on the UI but on being able to validate the business logic of the agents.

• The solution must provide the basic features as the Bedrock console, including trace.

• The solution must be adaptable to any Bedrock agents with no to minimal changes.

• The solution must run both locally and as a shared webapp for different workflows.

I decided to use Streamlit to build the UI as it is a popular and fitting choice. Streamlit is an open-source Python library used for building interactive web applications specially for AI and data applications. Since the application code is written only in Python, it is easy to learn and build with.

The Agents for Amazon Bedrock Runtime API can be used to interact with a Bedrock agent. Since the Streamlit app is developed in Python, we will naturally use the AWS SDK for Python (Boto3) for the integration. The AWS SDK Code Examples code library provides an example on how to use the AgentsforBedrockRuntime.Client.invoke_agent function to call the Bedrock agent. The function documentation was essential to determine the response format and the information.

The UI design is rather minimal as you can see in the following screenshot:

In the left sidebar, I include elements related to troubleshooting such as a session reset button and trace information similar to the Bedrock console. The main pane is a simple chat interface with the messages and the input. I made the favicon and page title configurable using environment variables for a white label experience.

About the repository structure

You can find the source code for the test UI in the acwwat/amazon-bedrock-agent-test-ui GitHub repository. The repository structure follows a standard structure as recommended by Mark Douthwaite for Streamlit projects. For a detailed explanation of the structure, refer to the getting started documentation. The only tweak I made is that I put the backend integration code into bedrock_agent_runtime.py in a services directory.

├── services
│   ├── bedrock_agent_runtime.py
├── .gitignore
├── app.py
├── Dockerfile
├── LICENSE
├── README.md
├── requirements.txt

Configuring and running the app locally

To run the Streamlit app locally, you just need to have the AWS CLI and Python 3 installed. Then you can clone the acwwat/amazon-bedrock-agent-test-ui GitHub repository and follow the steps below:

1. Run the following command to install the dependencies:
   pip install -r requirements.txt

2. Configure the environment variables for the AWS CLI and Boto3. You would typically configure the AWS CLI to create a named profile, then set the AWS_PROFILE environment variable to refer to it.

3. Set the following environment variables as appropriate:

   • BEDROCK_AGENT_ID - The ID of the Bedrock agent, which you can find in the Bedrock console or by running the aws bedrock-agent list-agents command.

   • BEDRROCK_AGENT_ALIAS_ID - The ID of the agent alias, which you can find in the Bedrock console or by running the aws bedrock-agent list-agent-aliases command. If this environment variable is not set, the default test alias ID TSTALIASID will be used.

   • BEDROCK_AGENT_TEST_UI_TITLE - (Optional) The page title. If this environment is not set, the generic title in the above screenshot will be used.

   • BEDROCK_AGENT_TEST_UI_ICON - (Optional) The favicon code, such as :bar_chart:. If this environment is not set, the generic icon in the above screenshot will be used.

4. Run the following command to start the Streamlit app:
   streamlit run app.py --server.port=8080 --server.address=localhost

Once the app is started, you can access it in your web browser at http://localhost:8080.

As an example, here is the list of bash commands I run in bash inside VS Code to start the app for testing my forex rate agent (which you can learn how to build or deploy using my Terraform configuration):

cd amazon-bedrock-agent-test-ui
pip install -r requirements.txt
# Use a named profile created by the "aws configure sso" command
export AWS_PROFILE=AWSAdministratorAccess-<redacted>
export BEDROCK_AGENT_ID=WENOOVMMEK
export BEDROCK_AGENT_TEST_UI_TITLE="Forex Rate Assistant"
export BEDROCK_AGENT_TEST_UI_ICON=":currency_exchange:"
# Log in via the browser when prompted
aws sso login
streamlit run app.py --server.port=8080 --server.address=localhost

To stop the app, send an INT signal (Ctrl+C) in the prompt where you are running the streamlit command.

Next steps

While the Streamlit app serves my purpose as it is, there are a few missing features which I will continue to add over time:

• Support for the use of Knowledge Bases for Amazon Bedrock in an agent, such as displaying citations

• Support for returning control to the agent developer

• Ability to switch between agents and aliases within the app

I also have not shown how to build and deploy the Streamlit app as a container in AWS, which I will perhaps demonstrate in another future blog post.

You are encountered to fork or copy the repository and built upon the existing code to suit your needs.

Summary

In this blog post, I provided an introduction to a generic Streamlit UI that I built to facilitate more efficient testing of agents built with the Agents for Amazon Bedrock service. You can clone the repository and follow the instructions to run it locally, and improve upon the baseline as you see fit.

I will be adding more features and fixing bugs over time, so be sure to check out the repository from time to time. Be sure to follow the Avangards Blog as I continue my journey with building generative AI applications using Amazon Bedrock.

Thanks for checking in!

In the previous blog post Building a Basic Forex Rate Assistant Using Agents for Amazon Bedrock, I demonstrated how to create a Bedrock agent in the AWS Management Console and outlined some ideas on improving the solution. Before further experimentation, it makes sense to automate the deployment of the solution to enable quicker updates as we go through trail and error in fine-tuning an agent.

In this blog post, we will automate the deployment of the basic forex rate assistant in Terraform using the resources that were recently released in v5.47.0 of the Terraform AWS Provider. Let's start by looking at the AWS resources in the AWS Management Console.

Taking inventory of the required resources

By examining the agent we previously built, we see that it is comprised of the following AWS resources:

1. The agent itself

2. The agent resource role which is an IAM service role that provides the agent with access to other AWS services and resources

   

3. The action group that defines API actions that the agent can perform

   

4. The Lambda function associated with the action group, which itself requires an execution role and a resource policy that allows the agent to invoke the function

   

With the list of resources we need to provision, we can begin creating the Terraform configuration starting with the resources that the agent depends on.

Defining resources for the IAM and Lambda dependencies

For the agent resource role, the documentation already provides the trust policy and the permissions required. It also specifies that the prefix AmazonBedrockExecutionRoleForAgents_ must be used for the role name.

The permission requires the foundational model's ARN, so we need at least the model's ID, which in our case is anthropic.claude-3-haiku-20240307-v1:0 for Claude 3 Haiku. For consistency, we will use the aws_bedrock_foundational_model data source to look up its ARN. Thus we can define the Terraform configuration for the agent resource role as follows using the aws_iam_role resource and the aws_iam_role_policy resource:

# Use data sources to get common information about the environment
data "aws_caller_identity" "this" {}
data "aws_partition" "this" {}
data "aws_region" "this" {}
locals {
  account_id = data.aws_caller_identity.this.account_id
  partition  = data.aws_partition.this.partition
  region     = data.aws_region.this.name
}

data "aws_bedrock_foundation_model" "this" {
  model_id = "anthropic.claude-3-haiku-20240307-v1:0"
}
# Agent resource role
resource "aws_iam_role" "bedrock_agent_forex_asst" {
  name = "AmazonBedrockExecutionRoleForAgents_ForexAssistant"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "bedrock.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = local.account_id
          }
          ArnLike = {
            "aws:SourceArn" = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:agent/*"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "bedrock_agent_forex_asst" {
  name = "AmazonBedrockAgentBedrockFoundationModelPolicy_ForexAssistant"
  role = aws_iam_role.bedrock_agent_forex_asst.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = "bedrock:InvokeModel"
        Effect   = "Allow"
        Resource = data.aws_bedrock_foundation_model.this.model_arn
      }
    ]
  })
}

Next, we will define the Lambda execution role which just needs the basic permissions to write logs to CloudWatch that the AWS-managed IAM policy AWSLambdaBasicExecutionRole provides. The Terraform configuration for this IAM role can be defined as follows:

data "aws_iam_policy" "lambda_basic_execution" {
  name = "AWSLambdaBasicExecutionRole"
}

# Action group Lambda execution role
resource "aws_iam_role" "lambda_forex_api" {
  name = "FunctionExecutionRoleForLambda_ForexAPI"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = "${local.account_id}"
          }
        }
      }
    ]
  })
  managed_policy_arns = [data.aws_iam_policy.lambda_basic_execution.arn]
}

We will then define the Terraform configuration for the Lambda function and its resource policy. Here is the source code for the Forex API Lambda function from the previous blog post for reference:

import json
import urllib.parse # urllib is available in Lambda runtime w/o needing a layer
import urllib.request

def lambda_handler(event, context):
    agent = event['agent']
    actionGroup = event['actionGroup']
    apiPath = event['apiPath']
    httpMethod =  event['httpMethod']
    parameters = event.get('parameters', [])
    requestBody = event.get('requestBody', {})

    # Read and process input parameters
    code = None
    for parameter in parameters:
        if (parameter["name"] == "code"):
            # Just in case, convert to lowercase as expected by the API
            code = parameter["value"].lower()

    # Execute your business logic here. For more information, refer to: https://docs.aws.amazon.com/bedrock/latest/userguide/agents-lambda.html
    apiPathWithParam = apiPath
    # Replace URI path parameters
    if code is not None:
        apiPathWithParam = apiPathWithParam.replace("{code}", urllib.parse.quote(code))

    # TODO: Use a environment variable or Parameter Store to set the URL
    url = "https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1{apiPathWithParam}.min.json".format(apiPathWithParam = apiPathWithParam)

    # Call the currency exchange rates API based on the provided path and wrap the response
    apiResponse = urllib.request.urlopen(
        urllib.request.Request(
            url=url,
            headers={"Accept": "application/json"},
            method="GET"
        )
    )
    responseBody =  {
        "application/json": {
            "body": apiResponse.read()
        }
    }

    action_response = {
        'actionGroup': actionGroup,
        'apiPath': apiPath,
        'httpMethod': httpMethod,
        'httpStatusCode': 200,
        'responseBody': responseBody

    }

    api_response = {'response': action_response, 'messageVersion': event['messageVersion']}
    print("Response: {}".format(api_response))

    return api_response

We will save this source code into a file called index.py in the lambda/forex_api directory in the same directory as the Terraform configuration, which will be packaged as a zip file using the archive_file data source to pass as an argument to the aws_lambda_function resource.

Here is the Terraform configuration for the Lambda function based on my battle-tested templates:

# Action group Lambda function
data "archive_file" "forex_api_zip" {
  type             = "zip"
  source_file      = "${path.module}/lambda/forex_api/index.py"
  output_path      = "${path.module}/tmp/forex_api.zip"
  output_file_mode = "0666"
}

resource "aws_lambda_function" "forex_api" {
  function_name = "ForexAPI"
  role          = aws_iam_role.lambda_forex_api.arn
  description   = "A Lambda function for the forex API action group"
  filename      = data.archive_file.forex_api_zip.output_path
  handler       = "index.lambda_handler"
  runtime       = "python3.12"
  # source_code_hash is required to detect changes to Lambda code/zip
  source_code_hash = data.archive_file.forex_api_zip.output_base64sha256
}

Lastly, we will set the Lambda resource policy using the aws_lambda_permission resource according to the specifications in the documentation:

resource "aws_lambda_permission" "forex_api" {
  action         = "lambda:invokeFunction"
  function_name  = aws_lambda_function.forex_api.function_name
  principal      = "bedrock.amazonaws.com"
  source_account = local.account_id
  source_arn     = "arn:aws:bedrock:${local.region}:${local.account_id}:agent/*"
}

Defining the agent and action group resources

With the dependencies out of the way, we can now define the Terraform resource for the agent with the new aws_bedrockagent_agent resource, which is rather straightforward:

resource "aws_bedrockagent_agent" "forex_asst" {
  agent_name              = "ForexAssistant"
  agent_resource_role_arn = aws_iam_role.bedrock_agent_forex_asst.arn
  description             = "An assisant that provides forex rate information."
  foundation_model        = data.aws_bedrock_foundation_model.this.model_id
  instruction             = "You are an assistant that looks up today's currency exchange rates. A user may ask you what the currency exchange rate is for one currency to another. They may provide either the currency name or the three-letter currency code. If they give you a name, you may first need to first look up the currency code by its name."
}

The action group can be defined in the agent using the aws_bedrockagent_action_group resource. We will need the OpenAPI schema YAML file from the previous blog post, which is included below for reference:

openapi: 3.0.0
info:
  title: Currency API
  description: Provides information about different currencies.
  version: 1.0.0
servers:
  - url: https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1
paths:
  /currencies:
    get:
      description: |
        List all available currencies
      responses:
        "200":
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                description: |
                  A map where the key refers to the lowercase three-letter currency code and the value to the currency name in English.
                additionalProperties:
                  type: string
  /currencies/{code}:
    get:
      description: |
        List the exchange rates of all available currencies with the currency specified by the given currency code in the URL path parameter as the base currency
      parameters:
        - in: path
          name: code
          required: true
          description: The lowercase three-letter code of the base currency for which to fetch exchange rates
          schema:
            type: string
      responses:
        "200":
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                description: |
                  A map where the key refers to the three-letter currency code of the target currency and the value to the exchange rate to the target currency.
                additionalProperties:
                  type: number
                  format: float

We will save the file as schema.yaml in the lambda/forex_api directory for the Lambda function, since they somewhat go together. Since we are providing the OpenAPI schema in-line, the Terraform resource can be defined as follows:

resource "aws_bedrockagent_agent_action_group" "forex_api" {
  action_group_name          = "ForexAPI"
  agent_id                   = aws_bedrockagent_agent.forex_asst.id
  agent_version              = "DRAFT"
  description                = "The currency exchange rates API"
  skip_resource_in_use_check = true
  action_group_executor {
    lambda = aws_lambda_function.forex_api.arn
  }
  api_schema {
    payload = file("${path.module}/lambda/forex_api/schema.yaml")
  }
}

Testing the configuration

Now that the full Terraform configuration is developed, we can apply it and make sure that it is working correctly. For me it took less than a minute to complete - here is the output for reference:

aws_iam_role.bedrock_agent_forex_asst: Creating...
aws_iam_role.lambda_forex_api: Creating...
aws_iam_role.bedrock_agent_forex_asst: Creation complete after 0s [id=AmazonBedrockExecutionRoleForAgents_ForexAssistant]
aws_iam_role_policy.bedrock_agent_forex_asst: Creating...
aws_bedrockagent_agent.forex_asst: Creating...
aws_iam_role.lambda_forex_api: Creation complete after 1s [id=FunctionExecutionRoleForLambda_ForexAPI]
aws_lambda_function.forex_api: Creating...
aws_iam_role_policy.bedrock_agent_forex_asst: Creation complete after 1s [id=AmazonBedrockExecutionRoleForAgents_ForexAssistant:AmazonBedrockAgentBedrockFoundationModelPolicy_ForexAssistant]
aws_bedrockagent_agent.forex_asst: Creation complete after 4s [id=LTR1P1OJUC]
aws_lambda_function.forex_api: Still creating... [10s elapsed]
aws_lambda_function.forex_api: Creation complete after 14s [id=ForexAPI]
aws_lambda_permission.forex_api: Creating...
aws_bedrockagent_agent_action_group.forex_api: Creating...
aws_lambda_permission.forex_api: Creation complete after 0s [id=terraform-20240430193700768300000002]
aws_bedrockagent_agent_action_group.forex_api: Creation complete after 0s [id=W1PDUUCT8P,LTR1P1OJUC,DRAFT]

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

In the Bedrock console, we can see that the agent ForexAssistant is ready for testing. Using the test chat interface, I asked:

What is the exchange rate from US Dollar to Canadian Dollar?

However, I got the following unexpected answer:

I apologize, but I am unable to look up the current exchange rate between US Dollar and Canadian Dollar. There seems to be an issue with the function call format that I am unable to resolve. I cannot provide the exchange rate information you requested.

Looking at the trace, it seems that the agent was not given the tool list and it tried to make up random functions to call, leading to errors:

On closer look, it seems that this is because there are pending changes in the agent which is requires preparation as indicated in the Bedrock console:

This tells me that Terraform is not performing the preparation. In any case, once I click Prepare and ask the same question again in a new session, the agent responds with the currency exchange rate I asked for:

The exchange rate from US Dollar (USD) to Canadian Dollar (CAD) is 1 USD = 1.36660199 CAD.

This is also confirmed in the trace which I will not show for brevity. Now we are one step away from an end-to-end IaC solution for the forex rate assistant, so let's try to address the issue.

Workaround for agent preparation using a null resource

Looking at the Terraform AWS Provider documentation, I couldn't find any resource that supports preparation. As well, the aws_bedrockagent_agent resource and the aws_bedrockagent_action_group resource don't seem to have any argument that controls the preparation behavior. To be fair, the action is implemented as a separate API action called PrepareAgent in the Agents for Bedrock API, which does not directly fit into the resource concept in Terraform.

While I opened an issue in the hashicorp/terraform-provider-aws GitHub repository, one quick workaround I can think of is to use a null resource with the local-exec provisioner to run the equivalent AWS CLI command for the PrepareAgent API, which is the aws bedrock-agent prepare-agent command.

Our objective is to trigger this null resource to be rerun (technically replaced) every time there are changes to the agent, which also extends to the action group. It is inefficient to simply prepare every time you apply the Terraform configuration, and if anything it is just one more moving part that can break. With that in mind, I devised the following resource that serve the purpose well.

resource "null_resource" "forex_asst_prepare" {
  triggers = {
    forex_asst_state = sha256(jsonencode(aws_bedrockagent_agent.forex_asst))
    forex_api_state  = sha256(jsonencode(aws_bedrockagent_agent_action_group.forex_api))
  }
  provisioner "local-exec" {
    command = "aws bedrock-agent prepare-agent --agent-id ${aws_bedrockagent_agent.forex_asst.id}"
  }
  depends_on = [
    aws_bedrockagent_agent.forex_asst,
    aws_bedrockagent_agent_action_group.forex_api
  ]
}

As you can see, I am using the triggers argument in the null resource to control when the resource should be replaced. We target the two main sources of change, which is the agent and the action group. Since trigger requires a string, a good candidate is to use the two resource's state somehow, as long as it doesn't contain any attributes that change every time Terraform is run. To keep the string short, we simply derive the SHA256 checksum from the resource state JSON as the triggers. The local-exec provisioner simply calls the AWS CLI command with the agent ID from aws_bedrockagent_agent.forex_asst.

With this change, we will run terraform destroy and then terraform apply to ensure full validity of the re-test. After Terraform completes successfully, we first check the agent in the Bedrock console to ensure that the Prepare button is no longer shown. As well, we ask our question to hopefully receive an expected result, which we did:

So there you have it, a functional Terraform configuration to deploy a basic forex rate assistant implemented using Agents for Amazon Bedrock!

Current limitations (it's brand new after all)

It is not unexpected that we encounter some issues with brand new features, such as what we encountered in this blog post with the Agents for Amazon Bedrock resources. I myself dove a bit deeper and found a few more issues which I reported. I encourage you to report any issues that you see as you work more with the Terraform resources.

Meanwhile, there are still a couple resources related to Knowledge bases for Amazon Bedrock still under development. I plan to integrate knowledge bases to our forex rate assistant, so I will eagerly wait for the Terraform resources to be ready for my next step in my Bedrock journey.

Summary

In this blog post, we developed the Terraform configuration for the basic forex rate assistant that we created interactively in the blog post Building a Basic Forex Rate Assistant Using Agents for Amazon Bedrock. While we encountered some issues, we were able to work around it as the community continues to build out the features in the Terraform AWS Provider. For now, I will pivot to enhancing the forex rate agent to add new capabilities and to address some of its known shortcomings.

If you like this blog post, please be sure to check out other helpful articles on AWS, Terraform, and other DevOps topics in the Avangards Blog.

With the prevalence of generative AI (gen AI), I've been keeping abreast on AWS' AI offerings for the past while. My journey started with Amazon Q Business, a fully managed service for building gen AI assistants. While the idea is great, it seems to be too basic as it is today and lacks the advanced features to improve the user experience in practice.

I then ventured into the more advanced use cases using Amazon Bedrock and went through many workshops such as Building with Amazon Bedrock and LangChain. The challenge I find is that these workshops still tend to be basic, and they don't answer my questions about complex use cases. I came to learn about agents while going through LangChain literatures, but developing a full workflow felt like a daunting task when my full-time job is DevOps, not software development. Things always seem too simple that it doesn't provide enough business value, or too complex that it becomes too costly.

After attending a recent AWS PartnerCast webinar on building intelligent enterprise apps using gen AI on AWS, I learned about Agents for Amazon Bedrock and some recent new features added to the service. The service seems to be within the Goldilocks zone matching my current skillsets, so I decided to dive heads-first to learn all about it. I decided to build something realistic and figured that I should share my journey with folks in this blog post.

About Agents for Amazon Bedrock

Agents for Amazon Bedrock is a service that enables gen AI applications to execute multi-step tasks across company systems and data sources. It is effectively a managed service for agents and retrieval-augmented generation (RAG), which are common patterns to extend the capabilities of large language models (LLMs).

Agents for Amazon Bedrock assumes the complexity of orchestrating the interactions between different components in such workflows, which must otherwise be programmed into your gen AI application. While you can use frameworks such as LangChain or LlamaIndex to develop these workflows, Agents for Amazon Bedrock makes it much more efficient for common use cases. Agents can also integrate with knowledge bases to enable RAG, as shown in the following diagram from the AWS documentation:

Coming up with a basic but representative use case

To help with brainstorming ideas for an agent, I decided to on these principles:

1. The idea must be practical and with real-life data.

2. Follow the KISS principle.

For inspirations on what type of agents I should build, I turned to the Public APIs GitHub repository which has a curated lists of free APIs. I narrowed my search for an API that does not require sign-up or an API key and returns useful information. I ultimately decided to use the Free Currency Exchange Rates API, which seemed promising upon some basic testing.

Naturally, the idea was steered towards a forex rate assistant which helps users look up rates from the API. The API supports lookup by dates, however to keep it simple I decided to limit the lookup to only the latest rates for now. This also leaves some room for enhancing the agent later.

Requesting for model access

Agents for Amazon Bedrock is a relatively new feature, so it is supported only in limited regions with limited model support. At the time of writing this blog post, it is only supported in US East (N. Virginia) (us-east-1 ) and US West (Oregon) (us-west-2) and only supports Anthropic models. We will use the us-west-2 region for our evaluation.

You should also be aware of the pricing for different Anthropic models. With the recent addition of the Claude 3 model family, Haiku emerges as highly competitive with great price-to-performance balance. Thus we will use Haiku as the model for our agent.

When you first use Amazon Bedrock, you must request for access to the models. This can be done in the Amazon Bedrock console using the Model access page which can be opened in the left menu. On that page, you will see the list of base models by vendor and their access status similar to the following:

To request for access, do the following:

1. Click on the Manage model access button.

2. On the Request model access page, scroll down to the Anthropic models in the list.

3. If this is the first time you are request access to Anthropic models, you will be required to submit use case details. Click on the Submit use case details button to open the form, then fill it in as appropriate and click Submit.

   

4. Check the box next to the models to which you wish to request access. Since we might compare different Anthropic models, let's check the box next to Anthropic to request access to all of them. Lastly, click Request model access at the end of the page.

   

The access status should now show "In progress" and the request will only take a few minutes to be approved if all goes well. Once available, the access status should change to "Access granted".

Creating the OpenAPI schema for the currency exchange API

In our agent, we will be using an action group that defines an action that the agent can help the user perform by calling APIs via a Lambda function. Consequently, the action group in our agent requires the following:

1. An OpenAPI schema that provides the specifications of the API

2. A Lambda function to which the action group makes API requests

That is also to say, the Lambda function is effectively a "proxy" API that calls the actual APIs, which in our case is the free currency exchange rates API. Based on the API documentation, we know the following:

• Since we will only support the latest exchange rate, the base URI for our API would be https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1 .

• We need to use the /currencies.min.json API, which gets the list of available currencies in minified JSON format. This helps minimize the number of tokens (and thus cost and limit) processed by the model.

• We also need to use the /currencies/{code}.min.json API, gets the currency exchange rates with {code} as the base currency.

Since this API does not provide the OpenAPI schema, we need to create it ourselves. I figured that this might be a regular exercise if I start testing Bedrock agents with different APIs, so I started looking for a tool that can generate OpenAPI schema, such as those listed in in OpenAPI.Tools. One category of tools seems to use network traffic, often in the HAR format, to generate the OpenAPI schema. I tried the OpenAPI DevTools which is a Chrome extension, however it did not work for the currency exchange rates API.

After wrestling with it for a bit and eventually giving up, I instead turned to ChatGPT to see if it is smart enough for the task. With my free plan, I asked ChatGPT 3.5 the following:

Can you generate the OpenAPI spec YAML from this API GET URL: https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1/currencies.min.json

To my surprise, it did generate a somewhat decent API spec:

While it is not usable as-is because the URL is missing the /v1 part and it is lacking some descriptions, it has almost everything that I need. However, it struck me as odd that the response has uppercase currency code which is NOT what the API returns. So I started a new ChatGPT session and ask the same question, only to get a very different spec:

At this point, I was certain that ChatGPT is not calling the API to generate the spec but rely on what its knowledge to generate an answer. It is probably experiencing hallucination, but it is good enough as a starting point 🤷

I did the same for the other API and adjusted the spec using the Swagger Editor. Specifically, I added detailed descriptions that should help the agent understand the API usages. The resulting OpenAPI YAML file is as follows:

openapi: 3.0.0
info:
  title: Currency API
  description: Provides information about different currencies.
  version: 1.0.0
servers:
  - url: https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1
paths:
  /currencies.min.json:
    get:
      description: |
        List all available currencies
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                description: |
                  A map where the key refers to the three-letter currency code and the value to the currency name in English.
                additionalProperties:
                  type: string
  /currencies/{code}.min.json:
    get:
      description: |
        List the exchange rates of all available currencies with the currency specified by the given currency code in the URL path parameter as the base currency
      parameters:
        - in: path
          name: code
          required: true
          description: The three-letter code of the base currency for which to fetch exchange rates
          schema:
            type: string
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                description: |
                  A map where the key refers to the three-letter currency code of the target currency and the value to the exchange rate to the target currency.
                additionalProperties:
                  type: number
                  format: float

Creating the agent

Now let's create the agent in the Amazon Bedrock console following the steps below:

1. Select Agents in the left menu.

2. On the Agents page, click Create Agent.

3. In the Create Agent dialog, enter the following information and click Create:

   • Name: ForexAssistant

   • Description: An assistant that provides forex rate information.

4. On the Agent builder page, enter the following information and click Save:

   • Agent resource role: Create and use a new service role

   • Select model: Anthropic, Claude 3 Haiku

   • Instructions for the Agent: You are an assistant that looks up today's currency exchange rates. A user may ask you what the currency exchange rate is for one currency to another. They may provide either the currency name or the three-letter currency code. If they give you a name, you may first need to first look up the currency code by its name.

Note that I try to provide concise instructions for the agent to help it reason up front. Depending on the test results, we might need to adjust it later with more prompt engineering.

Creating the action group

While still in the agent builder, we will create the action group that calls our APIs. Let's perform the following steps:

1. In the Action groups section, click Add.

2. On the Create Action group page, enter the following information and click Create:

   • Enter Action group name: ForexAPI

   • Description: The currency exchange rates API

   • Action group type: Define with API schemas

   • Action group invocation: Quick create a new Lambda function

   • Action group schema: Define via in-line schema editor

   • In-line OpenAPI schema:*Copy and paste the OpenAPI YAML from previous section*

After 15 seconds or so, you should receive a success message and be returned to the agent builder page. A dummy Lambda function should have been created, so our next step would be to add the logic to call the actual currency exchange rates API.

Updating the Lambda function to call the API

Let's go back into the action group page by clicking on the name of the action group (i.e. ForexAPI) in the list. In the edit page, click on the View button near the Select Lambda function field, which should take you to the function page in the Lambda console.

On the function page, you will see the code template that has been generated for you, which provides some basic processing of the input event and the response event.

After examining the input event format, we will recognize that the attributes that we need to use are:

• apiPath, which should provide the path to the API as defined in the OpenAPI YAML (namely /currencies.min.json or /currencies/{code}.min.json).

• httpMethod, which should always be get in our case. We thus won't make use of this attribute directly in our example.

• parameters, which we need to provide for the rate lookup API which expects the code URI path parameter to be a three-level currency code.

I will spare you the gory details on writing the Lambda function, so here's is the code and some implementation details provided in comments:

import json
import urllib.parse # urllib is available in Lambda runtime w/o needing a layer
import urllib.request

def lambda_handler(event, context):
    agent = event['agent']
    actionGroup = event['actionGroup']
    apiPath = event['apiPath']
    httpMethod =  event['httpMethod']
    parameters = event.get('parameters', [])
    requestBody = event.get('requestBody', {})

    # Read and process input parameters
    code = None
    for parameter in parameters:
        if (parameter["name"] == "code"):
            # Just in case, convert to lowercase as expected by the API
            code = parameter["value"].lower()

    # Execute your business logic here. For more information, refer to: https://docs.aws.amazon.com/bedrock/latest/userguide/agents-lambda.html
    apiPathWithParam = apiPath
    # Replace URI path parameters
    if code is not None:
        apiPathWithParam = apiPathWithParam.replace("{code}", urllib.parse.quote(code))

    # TODO: Use a environment variable or Parameter Store to set the URL
    url = "https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1{apiPathWithParam}".format(apiPathWithParam = apiPathWithParam)

    # Call the currency exchange rates API based on the provided path and wrap the response
    apiResponse = urllib.request.urlopen(
        urllib.request.Request(
            url=url,
            headers={"Accept": "application/json"},
            method="GET"
        )
    )
    responseBody =  {
        "application/json": {
            "body": apiResponse.read()
        }
    }

    action_response = {
        'actionGroup': actionGroup,
        'apiPath': apiPath,
        'httpMethod': httpMethod,
        'httpStatusCode': 200,
        'responseBody': responseBody

    }

    api_response = {'response': action_response, 'messageVersion': event['messageVersion']}
    print("Response: {}".format(api_response))

    return api_response

You can copy and paste this code into the editor and click Deploy to update it. At this point, we should test the Lambda function before returning to the Amazon Bedrock console. To do this, you can use the following event template to test the /currencies.min.json API (note that some irrelevant fields are omitted):

{
    "messageVersion": "1.0",
    "agent": {
        "name": "TBD",
        "id": "TBD",
        "alias": "TBD",
        "version": "TBD"
    },
    "inputText": "TBD",
    "sessionId": "TBD",
    "actionGroup": "TBD",
    "apiPath": "/currencies.min.json",
    "httpMethod": "get"
}

You should see the success response with the list of currencies:

You can then use the following event template to test the /currencies/{code}.min.json API:

{
    "messageVersion": "1.0",
    "agent": {
        "name": "TBD",
        "id": "TBD",
        "alias": "TBD",
        "version": "TBD"
    },
    "inputText": "TBD",
    "sessionId": "TBD",
    "actionGroup": "TBD",
    "apiPath": "/currencies/{code}.min.json",
    "httpMethod": "get",
    "parameters": [
        {
            "name": "code",
            "type": "string",
            "value": "usd"
        }
    ]
}

You should see the success response with the list of exchange rates from US dollar to other currencies:

With the Lambda function verified, we can close the Lambda console and return to the Bedrock console to test the agent.

Testing the agent

It is imperative that we test the agent thoroughly to ensure that it provides accurate answers. Back to the agent builder, we need to click on the Prepare button to prepare it, which is required whenever the agent is changed. We can then test the agent using the built-in chat interface to the right of the console using the following prompt:

What is the forex rate from US Dollar to Japanese Yen?

Interestingly, I got the following response from the agent:

Sorry, I do not have the capability to look up the current forex rate from US Dollar to Japanese Yen. I can only provide a list of available currencies, but cannot retrieve the specific exchange rate you requested.

This is seemingly implying that the agent only knows of one API but not the other. So we need to troubleshoot the problem, which is where the ever-important trace feature come into play. The trace helps you follow the agent's reasoning that leads it to the response it gives at that point in the conversation.

When we show the trace using the link below the agent response, we can see the traces for each orchestration steps. There are four traces under the Orchestration and knowledge base tab:

• Trace step 1 indicates the agent's rationale of first getting the currency code from the list then calling the /currencies/{code}.min.json API to get the rate, which seems correct. It is also able to call the /currencies.min.json API to get the list of currencies to look up the code. So far so good.

  

• Trace step 2 indicates that it was able to get the currency code for US Dollar as USD, however we are not sure why it's in uppercase. It also indicates that get::ForexAPI::/currencies/USD.min.json is not a valid function, which is not true. It is unclear about the logic behind the decision.

  

• Trace step 3 indicates that it is calling the /currencies.min.json API again for whatever reason. Lastly trace step 4 indicates that it cannot get the currency exchange rate and therefore gave up with the response we saw in the chat.

  

Since LLM is for the most part a black box, unfortunately we likely won't be able to get to the root cause. The only wild guess I could make is that the .min.json part is throwing it off because it doesn't resemble a normal RESTful API, so perhaps we can try to adjust the API specifications to remove that part.

Adjusting the API specs and re-testing

Let's make the adjustment in the OpenAPI YAML by stripping out the .min.json part from both API URLs:

openapi: 3.0.0
info:
  title: Currency API
  description: Provides information about different currencies.
  version: 1.0.0
servers:
  - url: https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1
paths:
  /currencies:
    get:
      description: |
        List all available currencies
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                description: |
                  A map where the key refers to the three-letter currency code and the value to the currency name in English.
                additionalProperties:
                  type: string
  /currencies/{code}:
    get:
      description: |
        List the exchange rates of all available currencies with the currency specified by the given currency code in the URL path parameter as the base currency
      parameters:
        - in: path
          name: code
          required: true
          description: The three-letter code of the base currency for which to fetch exchange rates
          schema:
            type: string
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                description: |
                  A map where the key refers to the three-letter currency code of the target currency and the value to the exchange rate to the target currency.
                additionalProperties:
                  type: number
                  format: float

This will cause the agent to pass the API URL without the .min.json part to the Lambda function in the event, so we need to add it to the URL before calling the currency exchange rates API in line 27. The resulting Lambda code is thus:

import json
import urllib.parse # urllib is available in Lambda runtime w/o needing a layer
import urllib.request

def lambda_handler(event, context):
    agent = event['agent']
    actionGroup = event['actionGroup']
    apiPath = event['apiPath']
    httpMethod =  event['httpMethod']
    parameters = event.get('parameters', [])
    requestBody = event.get('requestBody', {})

    # Read and process input parameters
    code = None
    for parameter in parameters:
        if (parameter["name"] == "code"):
            # Just in case, convert to lowercase as expected by the API
            code = parameter["value"].lower()

    # Execute your business logic here. For more information, refer to: https://docs.aws.amazon.com/bedrock/latest/userguide/agents-lambda.html
    apiPathWithParam = apiPath
    # Replace URI path parameters
    if code is not None:
        apiPathWithParam = apiPathWithParam.replace("{code}", urllib.parse.quote(code))

    # TODO: Use a environment variable or Parameter Store to set the URL
    url = "https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1{apiPathWithParam}.min.json".format(apiPathWithParam = apiPathWithParam)

    # Call the currency exchange rates API based on the provided path and wrap the response
    apiResponse = urllib.request.urlopen(
        urllib.request.Request(
            url=url,
            headers={"Accept": "application/json"},
            method="GET"
        )
    )
    responseBody =  {
        "application/json": {
            "body": apiResponse.read()
        }
    }

    action_response = {
        'actionGroup': actionGroup,
        'apiPath': apiPath,
        'httpMethod': httpMethod,
        'httpStatusCode': 200,
        'responseBody': responseBody

    }

    api_response = {'response': action_response, 'messageVersion': event['messageVersion']}
    print("Response: {}".format(api_response))

    return api_response

Once you updated both, you can prepare and test the agent again. Interestingly, we now get a proper response:

The current forex rate from US Dollar (USD) to Japanese Yen (JPY) is 1 USD = 158.3147925 JPY.

To ensure that the number is from the API and not other sources, we can review the agent's rationale from the trace like before. In trace step 2, we can see the right rationale and the invocation of the /currencies/{code} API with USD as the code parameter (again, not sure why it's in uppercase) as expected:

Hurray, we have successfully build a basic forex rate assistant using Agents for Amazon Bedrock! Naturally, you should test the agent extensively since LLMs are sometimes unpredictable and may require adjustments.

Testing reveals more limitations

For sake of experimentation, let's see what happens when we ask the assistant to do the reverse conversion. We will continue with the conversation in the same chat session and enter the following prompt:

What about the other way around?

The agent responded with the following:

The current forex rate from Japanese Yen (JPY) to US Dollar (USD) is 1 JPY = 0.0063163 USD.

However, a quick check shows that the number is a bit off. The response from https://cdn.jsdelivr.net/npm/@fawazahmed0/currency-api@latest/v1/currencies/jpy.json (at the time of writing) shows 0.0063165291 which is also what I got from the calculator for 1 / 158.3147925. Again, we will need to review the trace to see what the agent is up to. The trace revealed that the agent is doing an inverse calculation but the calculation is incorrect for some reason:

My expectation is that the agent should do another lookup from the API to get the right number. If the API were developed for a business and has a spread between the two exchange rates for profit, the agent would have given the wrong information. Putting that aside, the calculation is simply wrong.

After doing some reading online, it seems that LLMs in general are bad at math because their design is to predict words and not performing computations. So the exchange right 0.0063163 might just be a predication by Haiku based on the data that it was trained with.

Additional thoughts and summary

While we have built a functional forex rate assistant using Agents for Amazon Bedrock, it is certainly not production grade since it is not super accurate and it is a bit slow. Improving its accuracy is where the bulk of the effort for gen AI lies. AWS recommends the following strategies which developers should sequentially employ to improve their gen AI application:

For instance, my next iteration of improvement could start with adjusting the model inference parameters and prompt engineering, perhaps to ensure that it always calls the API instead of trying to do calculations. We also ought to look at why the LLM provide uppercase currency codes. Prompt engineering is admittedly more of an art and will require many rounds of trial and error, so be prepared for that.

I hope you learn something new from this blog post and has a better understanding of the features, potentials, and limitations of Agents for Amazon Bedrock. We are only scratching the surface here, so you are encouraged to use this forex agent as a start point for more improvements or develop your own agent. You would also need to expose the agent to end-users with a new frontend or an existing application. For me, the next step is to look into how to manage Bedrock agents using Terraform with the hot-off-the-press resources.

If you enjoyed this blog post, please be sure to check out other contents related to AWS and DevOps in the Avangards Blog. Thanks for your time and have fun with gen AI!

Since I released the blog series How to implement the AWS Startup Security Baseline (SSB) using Terraform recently, I've received some feedback and questions on it. In particular, there were some questions around setting up GuardDuty in an organization using Terraform. Since the configuration involves multiple accounts and there are some quirks with the resources, I decided to write a separate blog post on how to properly implement it with explanation on each step.

About the use case

Amazon GuardDuty is a managed threat detection service that continuously monitors AWS accounts and workloads for malicious or unauthorized activity using machine learning, anomaly detection, and integrated threat intelligence.

GuardDuty supports managing multiple accounts with AWS Organizations via the delegated administrator feature, with which you designate an AWS account in the organization to centrally manage GuardDuty for all members. This is great for managing a multi-account landing zone by centralizing management of GuardDuty settings in a consistent manner.

Since it is increasingly common to establish an AWS landing zone using AWS Control Tower, we will use the standard account structure in a Control Tower landing zone to demonstrate how to configure GuardDuty in Terraform:

The relevant accounts for our use case in the landing zone are:

1. The Management account for the organization where AWS Organizations is configured. For details, refer to Managing GuardDuty accounts with AWS Organizations.

2. The Audit account where security and compliance services are typically centralized in a Control Tower landing zone.

The objective is to delegate GuardDuty administrative duties from the Management account to the Audit account, after which all organization configurations are managed in the Audit account. With that said, let's see how we can achieve this using Terraform!

Designating a GuardDuty administrator account

GuardDuty delegated administrator is configured in the Management account, so we need a provider associated with it in Terraform. To keep things simple, we will take a multi-provider approach by defining two providers, one for the Management account and another for the Audit account, using AWS CLI profiles as follows:

provider "aws" {
  alias   = "management"
  # Use "aws configure" to create the "management" profile with the Management account credentials
  profile = "management" 
}

provider "aws" {
  alias   = "audit"
  # Use "aws configure" to create the "audit" profile with the Audit account credentials
  profile = "audit" 
}

We can then use the aws_guardduty_organization_admin_account resource to set the delegated administrator. However, I noticed the following in the Audit account:

• After this resource is created, GuardDuty will be enabled with both the foundational data sources and all protection plans enabled.

• When the resource is deleted, GuardDuty remains enabled.

These side effects are not desirable since we would ideally want full control over the lifecycle and configuration of GuardDuty in Terraform. To address this issue, we will preemptively enable GuardDuty in the Audit account using the aws_guardduty_detector resource. We will also manage the protection plans using the aws_guardduty_detector_feature resource in subsequent steps after we define the org-wide settings.

The resulting Terraform configuration should be defined as follows (pay special attention to the provider argument in each resource):

data "aws_caller_identity" "audit" {
  provider = aws.audit
}

resource "aws_guardduty_detector" "audit" {
  provider = aws.audit
}

resource "aws_guardduty_organization_admin_account" "this" {
  provider         = aws.management
  admin_account_id = data.aws_caller_identity.audit.account_id
  depends_on       = [aws_guardduty_detector.audit]
}

With the Audit account designated as the GuardDuty administrator, we can now manage the organization configuration.

Configuring organization auto-enable preferences

GuardDuty distinguishes the foundational data sources settings from the protection plans settings. The former is managed using the aws_guardduty_organization_configuration resource. In our case, we want to manage GuardDuty for all accounts (i.e. both new and existing accounts). The resulting Terraform configuration should thus look like the following:

resource "aws_guardduty_organization_configuration" "this" {
  provider                         = aws.audit
  auto_enable_organization_members = "ALL"
  detector_id                      = aws_guardduty_detector.audit.id
  depends_on                       = [aws_guardduty_organization_admin_account.this]
}

Next, let's manage the protection plan configuration. For illustration, let's assume that we only want to enable only EKS Audit Log Monitoring. To ensure full configurability, we will define the setting for all protection plans using a variable:

# Terraform configuration (.tf)

variable "guardduty_features" {
  description = "An object map that defines the GuardDuty organization configuration."
  type = map(object({
    auto_enable = string
    name        = string
    additional_configuration = optional(list(object({
      auto_enable = string
      name        = string
    })))
  }))
}

# Variable definition (.tfvars)

guardduty_features = {
  s3 = {
    auto_enable = "NONE"
    name        = "S3_DATA_EVENTS"
  }
  eks = {
    auto_enable = "ALL"
    name        = "EKS_AUDIT_LOGS"
  }
  eks_runtime_monitoring = {
    # EKS_RUNTIME_MONITORING is deprecated and should thus be explicitly disabled
    auto_enable = "NONE"
    name        = "EKS_RUNTIME_MONITORING"
    additional_configuration = [
      {
        auto_enable = "NONE"
        name        = "EKS_ADDON_MANAGEMENT"
      },
    ]
  }
  runtime_monitoring = {
    auto_enable = "NONE"
    name        = "RUNTIME_MONITORING"
    additional_configuration = [
      {
        auto_enable = "NONE"
        name        = "EKS_ADDON_MANAGEMENT"
      },
      {
        auto_enable = "NONE"
        name        = "ECS_FARGATE_AGENT_MANAGEMENT"
      },
      {
        auto_enable = "NONE"
        name        = "EC2_AGENT_MANAGEMENT"
      }
    ]
  }
  malware = {
    auto_enable = "NONE"
    name        = "EBS_MALWARE_PROTECTION"
  }
  rds = {
    auto_enable = "NONE"
    name        = "RDS_LOGIN_EVENTS"
  }
  lambda = {
    auto_enable = "NONE"
    name        = "LAMBDA_NETWORK_LOGS"
  }
}

We can then use this variable with the for_each meta-argument with the aws_guardduty_organization_configuration_feature resource as follows:

resource "aws_guardduty_organization_configuration_feature" "this" {
  provider    = aws.audit
  for_each    = var.guardduty_features
  auto_enable = each.value.auto_enable
  detector_id = aws_guardduty_detector.audit.id
  name        = each.value.name
  dynamic "additional_configuration" {
    for_each = try(each.value.additional_configuration, [])
    content {
      auto_enable = additional_configuration.value.auto_enable
      name        = additional_configuration.value.name
    }
  }
  depends_on = [aws_guardduty_organization_admin_account.this]
}

Lastly, we will circle back to recalibrating the protection plan settings for the Audit account itself. Let's piggyback on the same variable and use the aws_guardduty_detector_feature resource to achieve this:

resource "aws_guardduty_detector_feature" "audit" {
  provider    = aws.audit
  for_each    = var.guardduty_features
  detector_id = aws_guardduty_detector.audit.id
  name        = each.value.name
  status      = each.value.auto_enable == "NONE" ? "DISABLED" : "ENABLED"
  dynamic "additional_configuration" {
    for_each = try(each.value.additional_configuration, [])
    content {
      status = additional_configuration.value.auto_enable == "NONE" ? "DISABLED" : "ENABLED"
      name   = additional_configuration.value.name
    }
  }
}

With the complete Terraform configuration, you can now apply it to establish the Audit account as the delegated administrator and apply organization settings to all accounts in the target region. Note that it will take up to 24 hours for GuardDuty to automatically enable it in all accounts. YMMV, but it took about 3 hours in the evening in the Eastern time zone.

Caveats about suspending GuardDuty in member accounts

Due to limitations with the GuardDuty Terraform resources, GuardDuty is unfortunately not automatically disabled when you run terraform destroy. Normally this wouldn't be a problem for a production landing zone. However, if you are only testing, this could lead to unexpected costs especially when GuardDuty is a somewhat costly service.

As a workaround, I would recommend using the AWS CLI or AWS SDK to at least suspend GuardDuty for all members using the StopMonitoringMembers API. For your convenience, you can use the following shell script to do so before running terraform destroy:

#!/bin/bash

# Note: Make sure that you set the AWS_PROFILE environment variable to "audit" before running the script

# Get the GuardDuty detector ID
DETECTOR_ID=$(aws guardduty list-detectors --query DetectorIds[0] --output text)

# Disable auto-enable organization members
aws guardduty update-organization-configuration --detector-id $DETECTOR_ID --auto-enable-organization-member NONE

# Loop through each member account and disable GuardDuty
MEMBER_ACCOUNTS=$(aws guardduty list-members --detector-id $DETECTOR_ID --query Members[*].AccountId --output text)
for MEMBER_ACCOUNT in $MEMBER_ACCOUNTS
do
  echo "Suspending GuardDuty for account $MEMBER_ACCOUNT"
  aws guardduty stop-monitoring-members --account-ids $MEMBER_ACCOUNT --detector-id $DETECTOR_ID
done

Summary

In this blog post, you learned how to manage Amazon GuardDuty in AWS Organizations using Terraform. While there are some caveats, this allows you to streamline the setup of a security baseline for your AWS landing zone. The centralized approach to detective security can help you ensure compliance and timely reaction to security incidents.

I hope you found this blog post helpful. If you are interested in this type of content, be sure to check out other blog posts in the Avangards Blog. Thank you and have a great one!