Search for a command to run...
Learn how to deploy a self-hosted AI chat platform on AWS using LibreChat, MongoDB Atlas, Amazon ECS, and Amazon Bedrock, provisioned via Terraform.
No comments yet. Be the first to comment.
In this series, I will explore how to leverage different AWS services to power and manage all aspects of LibreChat.
Explore AWS Control Tower proactive controls, why they don’t work with Terraform, and a proof of concept that attempts to make them work. Introduction As a Terraform advocate and an AWS consultant who
I recently passed the Terraform Authoring and Operations Professional Exam. Here are my practical tips to help you prepare more effectively.
Learn how to monitor data ingestion jobs for Amazon Bedrock Knowledge Bases and enable notifications using CloudWatch Logs.
Learn how to enable logging for Amazon Bedrock Knowledge Bases, which uses the log delivery feature of Amazon CloudWatch Logs, with Terraform.
Generative AI has fundamentally shifted how we approach work, from writing and coding to research and problem-solving. For the past year, I used ChatGPT Business almost daily at work to improve my writing and research. However, I noticed limitations like fabrication and confirmation bias, so I wanted to explore how other non-OpenAI models perform. Additionally, my organization is consolidating on Microsoft 365 Copilot, which doesn't match ChatGPT's capabilities for my needs. This led me to search for a self-hosted, ChatGPT-like platform with flexibility in model choices.
I also needed it to be web-based for team members to access. As an AWS advocate, I wanted to leverage a diverse set of foundational models that Amazon Bedrock has to offer, and to host the platform using primarily AWS services. Based on my research, the three main options are LibreChat, Open WebUI, and AnythingLLM. Given that LibreChat is more feature-rich, customizable, and seemingly easier to deploy, I decided to give it a try and share my experience.
Without further ado, let's walk through the solution architecture and how it addresses my requirements.
The main design principle of the solution is to be cost-effective initially while allowing flexibility to scale in the future. Minimizing cost also means reducing operational overhead, not just service charges.
While cramming all components into an Amazon Lightsail instance is the cheapest option, it would need to be re-architected to scale horizontally. Deploying to an Amazon EC2 instance provides more flexibility, but it requires manual LibreChat installation and VM management. Ultimately, I decided to take a more modern approach and adopt a componentized architecture depicted in the following diagram:
This architecture uses the following technologies:
MongoDB Atlas - The LibreChat database runs on MongoDB Atlas using the Free (M0) cluster tier. It's technically free, runs in AWS, and is sufficient as a starter database engine as recommended by LibreChat.
Amazon ECS with AWS Fargate - The LibreChat application runs as a container with 512 (0.5) vCPU and 1 GB memory using 64-bit ARM architecture, which is sufficient when not enabling too many LibreChat features. Secrets are stored in AWS Systems Manager (SSM) Parameter Store (free), and configurations are stored in an Amazon S3 bucket to avoid additional shared storage services.
Application Load Balancer (ALB) - The public-facing endpoint. Although there is a running cost, its simpler TLS setup and support for scaling and AWS WAF integration makes it worthwhile.
fck-nat - Provides NAT gateway functionality using a pair of EC2 t4g.nano instances instead of AWS-managed NAT Gateways. This significantly reduces NAT gateway data transfer charges, making it a cost-effective option for modest traffic volumes.
The monthly cost of this architecture should be about $50 USD in us-east-1 including moderate Bedrock model use. To prevent surprise bills, set up a budget in AWS Budgets and create a cost monitor in AWS Cost Anomaly Detection.
Now that we have a good understanding of the architecture, let's go through the LibreChat concepts and prerequisites before we look at the Terraform configuration.
Since an official Terraform MongoDB Atlas Provider is available, let's use it to provision the database for LibreChat. If you have not already done so, sign up for a new account using the Get Started button on the MongoDB website.
Once you've completed sign-up, log in to the MongoDB Atlas Console. MongoDB Atlas automatically creates an organization and a sample project named Project 0. Since we'll use Terraform to create a new project, feel free to delete this sample project. You may also edit the organization name in Organizational Settings as needed.
Next, create an API key at the organization level for the Terraform provider. In the MongoDB Atlas Console, go to the organization level view and select Identity & Access > Applications in the left menu. On the Application page, select the Service Accounts tab and click Create service account:
On the Create Service Account page, enter a name (for example, "terraform") and a description (for example, "Service account for Terraform"), keep the client secret expiration as recommended, select the Organization Project Creator permission, and click Create. Copy both the client ID and secret from the next page for use with Terraform:
We're now ready to write the Terraform configuration to:
Create a new project
Create a new Free (M0) cluster with AWS as the backing provider
Create a database user for LibreChat
Add the NAT Gateway's public IP to the project IP Access List for security
To avoid hardcoding credentials, set the service account credentials as environment variables before running Terraform using MONGODB_ATLAS_CLIENT_ID and MONGODB_ATLAS_CLIENT_SECRET as per the provider configuration.
The following Terraform configuration provisions these MongoDB Atlas resources. Note that some attributes are provided as variables for flexibility, and the IP access list CIDR block refers to the NAT service elastic IPs (since egress traffic from LibreChat container goes through the fck-nat instances or NAT gateways):
resource "mongodbatlas_project" "librechat" {
org_id = var.atlas_org_id
name = var.atlas_project_name
}
resource "mongodbatlas_advanced_cluster" "librechat" {
project_id = mongodbatlas_project.librechat.id
name = var.atlas_cluster_name
cluster_type = "REPLICASET"
replication_specs = [
{
region_configs = [
{
electable_specs = {
instance_size = "M0"
}
provider_name = "TENANT"
backing_provider_name = "AWS"
region_name = var.atlas_region_name
priority = 7
}
]
}
]
}
resource "mongodbatlas_project_ip_access_list" "librechat" {
for_each = var.use_fck_nat ? aws_eip.fck_nat : aws_nat_gateway.zonal
project_id = mongodbatlas_project.librechat.id
cidr_block = "${each.value.public_ip}/32"
comment = "ECS egress CIDR for LibreChat"
}
resource "random_password" "atlas_db" {
length = 16
}
resource "mongodbatlas_database_user" "librechat" {
project_id = mongodbatlas_project.librechat.id
username = var.atlas_db_username
password = random_password.atlas_db.result
auth_database_name = "admin"
roles {
role_name = "readWriteAnyDatabase"
database_name = "admin"
}
}
Although using an IAM role to authenticate the Atlas database user would be more secure, it unfortunately doesn't work with the off-the-shelf LibreChat Docker image because it lacks the aws4 module required by the Mongoose library for AWS authentication. To avoid rebuilding a new container image, we'll stick with password authentication for now.
LibreChat uses two main sources of configuration: environment variables and LibreChat YAML. Environment variables are typically provided via a .env file created from the example in LibreChat's GitHub repository. Most LibreChat configuration is done using environment variables, and the LibreChat YAML file references these variables using the ${} notation for values such as API keys.
Additionally, the LibreChat YAML file (librechat.yaml) is typically placed in the application folder or mounted as an override in a containerized environment. However, it can also be provided in other locations by specifying the configuration path using the CONFIG_PATH environment variable.
Running LibreChat as a container introduces some challenges:
Building custom images is inefficient - While it's possible to build a LibreChat container image with .env and librechat.yaml baked in, rebuilding the image for every configuration change is inefficient. It's ideal to use the official LibreChat image from Docker Hub.
Managing many environment variables is difficult - Setting environment variables directly without using .env is hard to manage due to the sheer number of variables to configure, even when omitting those irrelevant to your use case.
Security concerns - Providing security-sensitive information as plain-text environment variables is not a security best practice.
ECS provides features to address these concerns:
Environment variable files - ECS supports passing environment variables via an environment variable file stored in S3. Since LibreChat already uses this format, it's a perfect fit.
Secrets management - AWS Secrets Manager or AWS Systems Manager (SSM) Parameter Store can securely pass sensitive data to containers as environment variables, avoiding hardcoded credentials.
To keep costs low, we will define all sensitive data as SSM Parameter Store parameters of type SecureString, while keeping all other configuration in a .env file provided to the container via ECS.
Providing the LibreChat YAML file to the container is more complicated. While the standard approach uses persistent storage options like Amazon EFS, that's cost-prohibitive for managing a single file. A better approach is using a sidecar container to write the file to a task-level shared volume before the main container starts.
For this, I implemented a sidecar container using busybox to decode a base64-encoded librechat.yaml (provided as an environment variable) and write it to /config/librechat.yaml in the task-level shared volume. The LibreChat container references this path using the CONFIG_PATH environment variable. The resulting container definitions (defined in the ECS task definition) are shown below:
container_definitions = jsonencode([
{
name = "init-librechat-config"
image = "public.ecr.aws/docker/library/busybox:1.36"
essential = false
command = [
"sh",
"-lc",
"mkdir -p /config && printf '%s' \"$LIBRECHAT_YAML_B64\" | base64 -d > /config/librechat.yaml"
]
environment = [
{
name = "LIBRECHAT_YAML_B64"
value = filebase64("${path.module}/librechat/${var.librechat_version}/librechat.yaml")
}
]
mountPoints = [
{
sourceVolume = "librechat-config"
containerPath = "/config"
readOnly = false
}
]
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.librechat.name
"awslogs-region" = var.region
"awslogs-stream-prefix" = "librechat"
}
}
},
{
name = "librechat"
image = "librechat/librechat:v0.8.4"
essential = true
dependsOn = [
{
containerName = "init-librechat-config"
condition = "SUCCESS"
}
]
portMappings = [
{
containerPort = 3080
protocol = "tcp"
}
]
secrets = [
{
name = "CREDS_KEY"
valueFrom = aws_ssm_parameter.librechat_creds_key.arn
},
{
name = "CREDS_IV"
valueFrom = aws_ssm_parameter.librechat_creds_iv.arn
},
{
name = "JWT_SECRET"
valueFrom = aws_ssm_parameter.librechat_jwt_secret.arn
},
{
name = "JWT_REFRESH_SECRET"
valueFrom = aws_ssm_parameter.librechat_jwt_refresh_secret.arn
},
{
name = "MEILI_MASTER_KEY"
valueFrom = aws_ssm_parameter.librechat_meili_master_key.arn
},
{
name = "MONGO_URI"
valueFrom = aws_ssm_parameter.librechat_mongo_uri.arn
}
]
environment = [
{
name = "CONFIG_PATH"
value = "/config/librechat.yaml"
}
]
environmentFiles = [
{
value = aws_s3_object.librechat_dot_env.arn
type = "s3"
}
]
mountPoints = [
{
sourceVolume = "librechat-config"
containerPath = "/config"
readOnly = true
}
]
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.librechat.name
"awslogs-region" = var.region
"awslogs-stream-prefix" = "librechat"
}
}
}
])
Now that we have a strategy for managing LibreChat configuration, let's define the minimal set of environment variables and LibreChat YAML as a starting point.
LibreChat provides starter files .env.example and librechat.example.yaml that we'll use as the basis for our configuration. Let's start with the environment file.
Copy the .env.example file to a local folder and comment out any sensitive values that will be provided as SSM Parameter Store environment variables specified below. Although individually defined environment variables take precedence over variables in environment files per the Amazon ECS Developer Guide, commenting them out avoids confusion.
In addition to MONGO_URI (the MongoDB connection string), LibreChat recommends adjusting any "secret" values from their default value for added security:
CREDS_IV - 16-byte Initialization Vector (IV) (32 characters in hex) for securely storing credentials
CREDS_KEY - 32-byte key (64 characters in hex) for securely storing credentials
JWT_SECRET - 32-byte key (64 characters in hex) as the JWT secret key
JWT_REFRESH_SECRET - 32-byte key (64 characters in hex) as the JWT refresh secret key
MEILI_MASTER_KEY - 16-byte key (32 characters in hex) as the MeiliSearch master key (required only if message and conversation search is enabled)
LibreChat provides a Credentials Generator to generate cryptographically secure random values for these secrets. Store them as SSM Parameter Store SecureString parameters or generate and store them directly using Terraform.
Next, adjust these environment variables for proper and secure operation:
HOST - Set to 0.0.0.0 to listen on all network interfaces, allowing ALB access
CONSOLE_JSON - Set to true to write logs to CloudWatch in JSON format for easier querying
ALLOW_REGISTRATION - Set to false to disable self-registration (we will create users with the create user script instead)
SEARCH - Set to false to disable message and conversation search (we're only demonstrating a minimal setup)
For the AI provider, we'll enable AWS Bedrock. Set ENDPOINTS to bedrock (the .env.example enables all pre-configured endpoints except Bedrock). Then configure the Bedrock endpoint by setting the following environment variables:
BEDROCK_AWS_DEFAULT_REGION - Set to your Bedrock region (e.g., us-east-1)
BEDROCK_AWS_MODELS - Set to us.amazon.nova-2-lite-v1:0 to use Amazon Nova Lite as a starting point (referring to the US Nova Lite system-defined inference profile)
OPENAI_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to OpenAI APIs even if the endpoint is disabled via ENDPOINTS.
ANTHROPIC_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to Anthropic APIs even if the endpoint is disabled via ENDPOINTS.
GOOGLE_KEY - Comment out to ensure that LibreChat does not try to make any calls to Google APIs even if the endpoint is disabled via ENDPOINTS.
ASSISTANTS_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to OpenAI Assistants APIs even if the endpoint is disabled via ENDPOINTS.
Note that we'll be using the ECS task IAM role to allow LibreChat to seamlessly call Bedrock APIs, so we don't need to set BEDROCK_AWS_ACCESS_KEY_ID nor BEDROCK_AWS_SECRET_ACCESS_KEY.
Copy the librechat.example.yaml file to a local folder. For this minimal setup, we won't need to define custom endpoints or configure advanced settings. However, to prevent custom endpoints like Groq and Mistral AI from appearing in the UI, comment out the custom key in the endpoints block and set it to an empty array [].
With all environment variables set in the environment file or SSM Parameter Store, and the librechat.yaml file prepared, we're ready to tie everything together with Terraform.
Since this blog post is getting quite long, let's focus on the key design elements of the Terraform configuration. You can find the complete Terraform configuration and source code in the 1_ecs_basic directory in this GitHub repository. Here are the descriptions for each Terraform configuration file:
atlas.tf - Defines the MongoDB Atlas resources, as explained in the earlier section of this blog post.
vpc.tf - Defines the VPC infrastructure for the solution. Key design elements include:
The VPC design follows a three-tier architecture across two AZs. Although the database subnets are currently not in use, they can be utilized for AWS cache and database services in the future.
There is built-in support for either fck-nat (default) or NAT Gateways, depending on your preference.
s3.tf - Defines the S3 bucket that hosts the LibreChat files and the S3 object for the .env file. Key design elements include:
A ready-to-use .env file is included in the librechat/v0.8.4 folder. You can edit this file if using the same version, or upload a new .env file when you upgrade LibreChat (be sure to change the librechat_version variable).
This S3 bucket may be used in the future as the LibreChat file storage backend, hence the .env file is placed in the config subfolder for better separation.
ecs.tf - Defines all ECS and related resources to run LibreChat as an ECS service. Key design elements include:
A ready-to-use librechat.yaml file is included in the librechat/v0.8.4 folder. You can edit this file if using the same version, or upload a new librechat.yaml file when you upgrade LibreChat (be sure to change the librechat_version variable).
The LibreChat credentials are generated by first creating a random password using the random_password ephemeral resource, then defining an aws_ssm_parameter resource with the write-only value storing the hash of the password (SHA1 or SHA256). You can replace this logic if you prefer to manually store the generated credentials in SSM Parameter Store first.
The IAM policy for the ECS task role, aws_iam_role_policy.ecs_task_librechat, contains permissions to invoke Bedrock models and manage marketplace subscription of third-party models.
Service auto-scaling is defined for future use, but for now, it is scaled to 1 for cost control.
The LibreChat container runs on TCP port 3080 by default.
CloudWatch logs are streamed to the /ecs/librechat log group.
alb.tf - Defines the ALB resources as the public-facing endpoint for LibreChat. Key design elements include:
A HTTPS listener is defined with HTTP redirect to ensure security. Consequently, you must import or create a TLS certificate in AWS Certificate Manager (ACM), then pass the certificate's ARN using the alb_certificate_arn variable.
HTTPS also requires a custom host name, so you must pass the DNS name using the librechat_dns_name variable.
The target group checks the ECS task health using LibreChat's /health endpoint.
After cloning the GitHub repository, deploy the solution as follows:
From the root of the cloned GitHub repository, navigate to 1_ecs_basic.
Set the MONGODB_ATLAS_CLIENT_ID and MONGODB_ATLAS_CLIENT_SECRET to the Atlas service account credentials created in the earlier section.
Configure your AWS credentials using aws configure for IAM, or aws configure sso for IAM Identity Center. The profile name will be provided as a Terraform variable.
Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.
Run terraform init and terraform apply.
It is advised to check the ECS service status in the AWS Management Console to ensure that the tasks run successfully. Failed tasks will cause a perpetual restart due to the required capacity of 1, and overlooking this may lead to unexpected cost consequences. Check the task logs in the CloudWatch log group /ecs/librechat for errors if needed.
Once the configuration is applied, create the CNAME record for the ALB DNS name. If you manage your domain/subdomain using a public hosted zone in Amazon Route 53, you can also create an alias record pointing to the ALB. Lastly, go to the custom host name for LibreChat and ensure that the login page loads successfully.
Since self-registration is disabled, we need to use the create user script to create the first user. The easiest way is to use ECS Exec in the ECS console to run the script in the librechat container in the task configuration. Here's a screenshot showing where to find the Connect button to open an interactive session:
Once CloudShell opens and connects to the container's shell, run the following command to start the user creation wizard:
npm run create-user
Enter the user's information as prompted, and a user will be created in LibreChat's database. Here's an example of creating a user for John Doe:
Now you're ready to log in. Open your LibreChat application URL and log in using the credentials you just created. Upon successful login, accept LibreChat's terms of service. You should see the Nova 2 Lite model already selected at the top, since it's the only configured model.
Let's test the setup with a simple prompt:
Tell me a joke about AWS.
If LibreChat responds with a joke, you've successfully completed the setup! Here's the joke I received, which honestly suggests that the Nova model could use some additional training with a better comedy dataset...
Congratulations, you now have your own LibreChat instance in AWS! You're now ready to start exploring and expanding its capabilities. While this setup gives you a functional chat interface, there's much more you can do to enhance its features. Here are some examples:
Configure more Bedrock models to unlock diverse capabilities and customization, balancing functionality, performance, and cost
Enable web search to allow LibreChat to search the internet and retrieve relevant information, enhancing conversations
Build custom AI assistants using AI Agents and integrate with various built-in and MCP tools to elevate capabilities and user experience
As your LibreChat usage grows, it's imperative to align your architecture with the AWS Well-Architected Framework. Here are some examples to improve security and operational robustness:
Streamline authentication by integrating with an Identity Provider (IDP)
Enable caching, session storage, and horizontal scaling in LibreChat using Redis and compute auto-scaling
Scale file storage with an Amazon S3 storage backend
Given the vast possibilities, I will start a new blog series about LibreChat and how to best use AWS services and best practices to enable these capabilities. If you enjoyed this post, stay tuned for new content at the Avangards Blog. Thanks so much for reading, and I hope you have fun chatting with LibreChat!