Cloud-managed database

Suggest Edits

Introduction

In case you're using a cloud-managed database, such as AWS's RDS, you can use Terraform modules to deploy the database. An example is presented in one of Bunnyshell's Templates.

For seeding, you can leverage the create from snapshot cloud feature to seed the database. As an alternative, you can import the database from a dump file in a Bunnyshell runner.


Component Example

Let's consider the code for the Component below. You can find the full bunnyshell.yaml on GitHub.

-
    kind: Terraform
    name: db-rds
    gitRepo: 'https://github.com/bunnyshell/templates.git'
    gitBranch: terraform-aws-rds/v0.1.0
    gitApplicationPath: /components/tf-aws-rds
    runnerImage: 'hashicorp/terraform:1.5.1'
    deploy:
        - 'cd components/tf-aws-rds'
        - '/bns/helpers/terraform/get_managed_backend > zz_backend_override.tf'
        - 'terraform init -input=false -no-color'
        - |
            terraform apply -input=false -auto-approve -no-color \
              -var="prefix=bns-{{ env.unique }}" \
              -var="instance_class=db.t3.micro" \
              -var="allocated_storage=20" \
              -var="engine=postgres" \
              -var="engine_version=15.3" \
              -var="db_name={{ env.vars.DB_NAME }}" \
              -var="username=bunnyshell" \
              -var="db_port=5432" \
              -var="vpc_cidr=10.0.0.0/16" \
              -var='allowed_cidrs=["54.246.59.74/32", "34.83.72.16/32"]'\
              -var="publicly_accessible=true"
        - 'BNS_TF_STATE_LIST=`terraform show -json`'
        - |
            DB_HOST=$(terraform output --raw rds_endpoint | awk -F: '{print $1}')
        - 'DB_PORT=`terraform output --raw rds_port`'
        - 'DB_USERNAME=`terraform output --raw rds_username`'
        - 'DB_PASSWORD=`terraform output --raw rds_password`'
    destroy:
        - 'cd components/tf-aws-rds'
        - '/bns/helpers/terraform/get_managed_backend > zz_backend_override.tf'
        - 'terraform init -input=false -no-color'
        - |
            terraform destroy -input=false -auto-approve -no-color \
              -var="prefix=bns-{{ env.unique }}" \
              -var="instance_class=db.t3.micro" \
              -var="allocated_storage=20" \
              -var="engine=postgres" \
              -var="engine_version=15.3" \
              -var="db_name={{ env.vars.DB_NAME }}" \
              -var="username=bunnyshell" \
              -var="db_port=5432" \
              -var="vpc_cidr=10.0.0.0/16" \
              -var='allowed_cidrs=["54.246.59.74/32", "34.83.72.16/32"]'\
              -var="publicly_accessible=true"
    exportVariables:
        - DB_HOST
        - DB_PORT
        - DB_USERNAME
        - DB_PASSWORD
    environment:
        AWS_ACCESS_KEY_ID: your-aws-access-key-id
        AWS_REGION: your-aws-region
        AWS_SECRET_ACCESS_KEY: your-aws-secret-access-key

📘

AWS Credentials

Since this Component actually creates a database, we cannot provide any demo credentials. You will need to create your own IAM user. You can find the needed IAM permissions here.


Cloud database snapshot

In order to have the database seeded when it is created, you need to specify a snapshot to be used. The following describes just as this can be achieved.


1. Create a Snapshot

You can create a snapshot of your RDS DB instance from the AWS Management Console, AWS CLI, or the RDS API.

From AWS Management Console

Navigate to RDS > Databases, select the DB instance that you want to create a snapshot of, and choose Take snapshot.

From AWS CLI

You can use the create-db-snapshot command. Replace my-db-instance-identifier with the name of your DB instance and my-db-snapshot-identifier with the name you want to give your DB snapshot.

The database instance identifier is the name of the RDS instance in AWS. The database snapshot identifier is something you choose and then you will use in Bunnyshell when running terraform apply to have the database created from this snapshot.

aws rds create-db-snapshot \
    --db-instance-identifier my-db-instance-identifier \
    --db-snapshot-identifier my-db-snapshot-identifier

2. Get the Snapshot Identifier

After the snapshot is created, you can find it in the AWS Management Console under RDS > Snapshots.
The name of the snapshot is the snapshot identifier. If you used the AWS CLI to create the snapshot, the snapshot identifier is what you specified for db-snapshot-identifier.


3. Use the Snapshot Identifier in Terraform

Once you have the snapshot identifier, you can use it as the value for the snapshot_identifier variable in your Terraform configuration, by adding a variable to the terraform apply script under deploy.
If you set this variable, Terraform will create a new RDS DB instance from the specified snapshot.

You need to add -var="snapshot_identifier=books-demo-snapshot" to the terraform apply (see last line below).

terraform apply -input=false -auto-approve -no-color \
-var="prefix=bns-{{ env.unique }}" \
              -var="instance_class=db.t3.micro" \
              -var="allocated_storage=20" \
              -var="engine=postgres" \
              -var="engine_version=15.3" \
              -var="db_name={{ env.vars.DB_NAME }}" \
              -var="username=bunnyshell" \
              -var="db_port=5432" \
              -var="vpc_cidr=10.0.0.0/16" \
              -var='allowed_cidrs=["54.246.59.74/32", "34.83.72.16/32"]'\
              -var="publicly_accessible=true"\
              -var="snapshot_identifier=books-demo-snapshot"

Custom database dump

You could use Bunnyshell to perform the seeding, by using a GenericComponent to download the seed file and run the database import after the database has been created.

We're going to take an example that can be coupled with the Terraform Component db-rds (presented above). The component must be placed after the one which creates the database.

In the example below, bunnyshell/s3-download:0.1.0 is the image used for the runner; it has the AWS cli installed. The file is downloaded from S3, then unzipped, and the import is performed afterwards.

You can then use pg_restore if you have a file created with pg_dump, or simply run psql on a standard sql.

Passing PGPASSWORD as an environment variable is the authentication method for pg_restore.

-
    kind: GenericComponent
    name: db-seed
    runnerImage: 'bunnyshell/s3-download:0.1.0'
    deploy:
        - '. /download_s3_file.sh'
        - 'apk add --no-cache postgresql-client gzip'
        - 'gunzip /tmp/books.sql.gz'
        - '#psql -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB -f /tmp/books.sql'
        - 'pg_restore -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB /tmp/books.sql'
    environment:
        AWS_ACCESS_KEY_ID: your-aws-access-key-id
        AWS_REGION: your-aws-region
        AWS_SECRET_ACCESS_KEY: your-aws-secret-access-key
        DOWNLOAD_PATH: /tmp/
        S3_FILES: 's3://demo-books-sql-dump/books.sql.gz'
        SKIP_IF_EXISTS: '1'
        PGPASSWORD: '{{ components.db-rds.exported.DB_PASSWORD }}'
        POSTGRES_DB: '{{ env.vars.DB_NAME }}'
        POSTGRES_HOST: '{{ components.db-rds.exported.DB_HOST }}'
        POSTGRES_USER: '{{ components.db-rds.exported.DB_USERNAME }}'

:warning: However, keep in mind that large dump files will take (considerably) longer to import than using Database snapshots.

Updated 5 months ago