Data Storage Modules 0.26.0

Database backup

View SourceRelease Notes

This module, along with the lambda-share-snapshot and lambda-copy-shared-snapshot modules, can be used to backup your RDS database to another AWS account (e.g., for disaster recovery) on a configurable schedule. Under the hood, each module runs a Lambda function that instructs your database to take a snapshot (this module), share the snapshot with another account (the lambda-share-snapshot module), and make a copy of the snapshot (lambda-copy-shared-snapshot).

RDS architecture

Features

• Standalone functions for taking snapshots, sharing snapshots, and copying snapshots that can be combined in many different ways

• You can combine them to copy snapshots across AWS accounts and regions

• Configurable backup schedule (e.g., using cron expressions)

• Clean up old snapshots automatically using the lambda-cleanup-snapshots module.

Learn

Note

This repo is a part of the Gruntwork Infrastructure as Code Library, a collection of reusable, battle-tested, production ready infrastructure code. If you’ve never used the Infrastructure as Code Library before, make sure to read How to use the Gruntwork Infrastructure as Code Library!

Core concepts

• What is Amazon RDS?

• How does this differ from RDS automatic snapshots?

• RDS documentation: Amazon’s docs for RDS that cover core concepts such as the types of databases supported, security, backup & restore, and monitoring.

• Designing Data Intensive Applications: the best book we’ve found for understanding data systems, including relational databases, NoSQL, replication, sharding, consistency, and so on.

Deploy

Non-production deployment (quick start for learning)

If you just want to try this repo out for experimenting and learning, check out the following resources:

• examples folder: The examples folder contains sample code optimized for learning, experimenting, and testing (but not production usage).

Production deployment

If you want to deploy this repo in production, check out the following resources:

• rds module in the Acme example Reference Architecture: Production-ready sample code from the Acme Reference Architecture examples.

Manage

Day-to-day operations

• How to backup RDS snapshots to a separate AWS account

Major changes

• Restoring from a DB snapshot

Sample Usage

Terraform

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S LAMBDA-CREATE-SNAPSHOT MODULE
# ------------------------------------------------------------------------------------------------------

module "lambda_create_snapshot" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-data-storage.git//modules/lambda-create-snapshot?ref=v0.26.0"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The ARN of the RDS database
  rds_db_arn = <INPUT REQUIRED>

  # The identifier of the RDS database
  rds_db_identifier = <INPUT REQUIRED>

  # If set to true, this RDS database is an Amazon Aurora cluster. If set to false,
  # it's running some other database, such as MySQL, Postgres, Oracle, etc.
  rds_db_is_aurora_cluster = <INPUT REQUIRED>

  # An expression that defines how often to run the lambda function to take
  # snapshots. For example, cron(0 20 * * ? *) or rate(5 minutes).
  schedule_expression = <INPUT REQUIRED>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # Set to false to have this module skip creating resources. This weird parameter
  # exists solely because Terraform does not support conditional modules. Therefore,
  # this is a hack to allow you to conditionally decide if this module should create
  # anything or not.
  create_resources = true

  # Namespace all Lambda resources created by this module with this name. If not
  # specified, the default is var.rds_db_identifier with '-create-snapshot' as a
  # suffix.
  lambda_namespace = null

  # If the DB is not in available state when this function runs, it will retry up to
  # max_retries times.
  max_retries = 60

  # If set true, just before the lambda function finishes running, it will report a
  # custom metric to CloudWatch, as specified by
  # var.report_cloudwatch_metric_namespace and var.report_cloudwatch_metric_name.
  # You can set an alarm on this metric to detect if the backup job failed to run to
  # completion.
  report_cloudwatch_metric = false

  # The name to use for the the custom CloudWatch metric. Only used if
  # var.report_cloudwatch_metric is set to true.
  report_cloudwatch_metric_name = null

  # The namespace to use for the the custom CloudWatch metric. Only used if
  # var.report_cloudwatch_metric is set to true.
  report_cloudwatch_metric_namespace = null

  # Namespace all Lambda scheduling resources created by this module with this name.
  # If not specified, the default is var.lambda_namespace with '-scheduled' as a
  # suffix.
  schedule_namespace = null

  # The ARN of a lambda job to trigger to share the DB snapshot with another AWS
  # account. Only used if var.share_snapshot_with_another_account is set to true.
  share_snapshot_lambda_arn = null

  # The ID of an AWS account with which to share the RDS snapshot. Only used if
  # var.share_snapshot_with_another_account is set to true.
  share_snapshot_with_account_id = null

  # If set to true, after this lambda function takes a snapshot of the RDS DB, it
  # will trigger the lambda function specified in var.share_snapshot_lambda_arn to
  # share the snapshot with another AWS account.
  share_snapshot_with_another_account = false

  # The amount of time, in seconds, between retries.
  sleep_between_retries_sec = 60

  # Namespace all snapshots created by this module's jobs with this suffix. If not
  # specified, only the database identifier and timestamp are used.
  snapshot_namespace = ""

}

Terragrunt

terraform.tfvars

# Coming soon!

Reference

Inputs

Required

rds_db_arnstringrequired

The ARN of the RDS database

rds_db_identifierstringrequired

The identifier of the RDS database

rds_db_is_aurora_clusterboolrequired

If set to true, this RDS database is an Amazon Aurora cluster. If set to false, it's running some other database, such as MySQL, Postgres, Oracle, etc.

schedule_expressionstringrequired

An expression that defines how often to run the lambda function to take snapshots. For example, cron(0 20 * ? ) or rate(5 minutes).

Optional

create_resourcesbooloptional

Set to false to have this module skip creating resources. This weird parameter exists solely because Terraform does not support conditional modules. Therefore, this is a hack to allow you to conditionally decide if this module should create anything or not.

Default:true

lambda_namespacestringoptional

Namespace all Lambda resources created by this module with this name. If not specified, the default is rds_db_identifier with '-create-snapshot' as a suffix.

Default:null

max_retriesnumberoptional

If the DB is not in available state when this function runs, it will retry up to max_retries times.

Default:60

report_cloudwatch_metricbooloptional

If set true, just before the lambda function finishes running, it will report a custom metric to CloudWatch, as specified by report_cloudwatch_metric_namespace and report_cloudwatch_metric_name. You can set an alarm on this metric to detect if the backup job failed to run to completion.

Default:false

report_cloudwatch_metric_namestringoptional

The name to use for the the custom CloudWatch metric. Only used if report_cloudwatch_metric is set to true.

Default:null

report_cloudwatch_metric_namespacestringoptional

The namespace to use for the the custom CloudWatch metric. Only used if report_cloudwatch_metric is set to true.

Default:null

schedule_namespacestringoptional

Namespace all Lambda scheduling resources created by this module with this name. If not specified, the default is lambda_namespace with '-scheduled' as a suffix.

Default:null

share_snapshot_lambda_arnstringoptional

The ARN of a lambda job to trigger to share the DB snapshot with another AWS account. Only used if share_snapshot_with_another_account is set to true.

Default:null

share_snapshot_with_account_idstringoptional

The ID of an AWS account with which to share the RDS snapshot. Only used if share_snapshot_with_another_account is set to true.

Default:null

share_snapshot_with_another_accountbooloptional

If set to true, after this lambda function takes a snapshot of the RDS DB, it will trigger the lambda function specified in share_snapshot_lambda_arn to share the snapshot with another AWS account.

Default:false

sleep_between_retries_secnumberoptional

The amount of time, in seconds, between retries.

Default:60

snapshot_namespacestringoptional

Namespace all snapshots created by this module's jobs with this suffix. If not specified, only the database identifier and timestamp are used.

Default:""

Outputs

lambda_function_arn

lambda_iam_role_id