This article will share our experience importing and adapting infrastructure configurations deployed manually to AWS to the Terraform format. You may ask, what’s the point of all this? The reasons are many: ensuring fault tolerance, making horizontal and vertical scaling easier, etc. Let’s discuss them.
Why I might need that
The automated infrastructure management solutions allow you to:
- quickly recreate the infrastructure;
- maintain the changes history;
- control the software versions used;
- …
It’s not just a matter of convenience. Lack of such automated solutions directly impacts your business apps’ operation, e.g., the speed of infrastructure problems troubleshooting, its overall fault tolerance, and uptime. This revelation led to the emergence of the IaC approach and its proliferation.
The configuration management systems (Ansible, Chef, etc.) solve this problem at the software services level. However, they do not cover the lower (hardware) level, which also needs attention.
Most of our readers are probably aware of solutions covering the hardware level (e.g., Terraform). It comes in handy when you create an infrastructure from scratch. But what if our production cluster includes dozens or even hundreds of components? In this case, writing deployment scripts for each element would require tremendous effort.
In this article, we will take a different approach and try to import the existing production infrastructure. Next, we will adapt the imported configurations to enable a seamless transition to automated management.
Planning and preparing
We have examined various tools for importing configurations (Terraform itself, the AWS CLI tool, Terraforming) and settled on Terraformer. We liked the completeness and accuracy of the output data produced by Terraformer and its ease of use. The author of Terraformer pursued the goals mentioned above when he faced a similar challenge. In the end, he created a tool with the capabilities we needed.
It is worth mentioning that we planned to import and adapt the configurations in several unrelated regions. All actions had to be carried out sequentially and affect only one region at a time. Thus, we decided to implement a fully operational environment and verify its functionality in a testing environment. We will use a Docker container because it allows for quick deployment of an environment that is tied to specific software versions.
The Docker image will be built and run using the werf tool and its built-in syntax (called Stapel) instead of Dockerfile. However, our preference is motivated by a more convenient/habitual approach and is not due to any architectural constraints meaning that you can easily use plain Docker as well.
We will use a basic Ubuntu image and install Terraform and the Terraformer plugin in it. We will also need to configure Terraform and add a file with AWS credentials. Here is the Stapel manifest for werf (you can efficiently rewrite it to plain Dockerfile
if necessary):
configVersion: 1
project: terraform
---
{{- $params := dict -}}
{{- $_ := set $params "UbuntuVersion" "18.04" -}}
{{- $_ := set $params "UbuntuCodename" "bionic" -}}
{{- $_ := set $params "TerraformVersion" "0.13.6" -}}
{{- $_ := set $params "TerraformerVersion" "0.8.10" -}}
{{- $_ := set $params "WorkDir" "/opt/terraform" -}}
{{- $_ := set $params "AWSDefaultOutput" "json" -}}
{{- $_ := env "AWS_SECRET_ACCESS_KEY" | set $ "AWSSecretAccessKey" -}}
{{- $_ := env "AWS_ACCESS_KEY_ID" | set $ "AWSAccessKeyId" }}
{{- $_ := env "AWS_REGION" | set $ "AWSRegion" }}
{{- $_ := env "CI_ENVIRONMENT_SLUG" | set $ "Environment" }}
---
image: terraform
from: ubuntu:{{ $params.UbuntuVersion }}
git:
- add: /
to: "{{ $params.WorkDir }}"
owner: terraform
group: terraform
excludePaths:
- "*.tfstate"
- "*.tfstate.backup"
- "*.bak"
- ".gitlab-ci.yml"
stageDependencies:
setup:
- "regions/*"
- "states/*"
ansible:
beforeInstall:
- name: "Install essential utils"
apt:
name:
- tzdata
- apt-transport-https
- curl
- locales
- locales-all
- unzip
- python
- groff
- vim
update_cache: yes
- name: "Remove old timezone symlink"
file:
state: absent
path: "/etc/localtime"
- name: "Set timezone"
file:
src: /usr/share/zoneinfo/Europe/Tallinn
dest: /etc/localtime
owner: root
group: root
state: link
- name: "Create non-root main application user"
user:
name: terraform
comment: "Non-root main application user"
uid: 7000
shell: /bin/bash
home: {{ $params.WorkDir }}
- name: "Remove excess docs and man files"
file:
path: "{{`{{ item }}`}}"
state: absent
with_items:
- /usr/share/doc/
- /usr/share/man/
- name: "Disable docs and man files installation in dpkg"
copy:
content: |
path-exclude="/usr/share/doc/*"
dest: "/etc/dpkg/dpkg.cfg.d/01_nodoc"
- name: "Generate ru_RU.UTF-8 default locale"
locale_gen:
name: ru_RU.UTF-8
state: present
install:
- name: "Download terraform"
get_url:
url: https://releases.hashicorp.com/terraform/{{ $params.TerraformVersion }}/terraform_{{ $params.TerraformVersion }}_linux_amd64.zip
dest: /usr/src/terraform_{{ $params.TerraformVersion }}_linux_amd64.zip
mode: 0644
- name: "Install terraform"
shell: |
unzip /usr/src/terraform_{{ $params.TerraformVersion }}_linux_amd64.zip -d /usr/local/bin
terraform -install-autocomplete
- name: "Install terraformer provider"
get_url:
url: https://github.com/GoogleCloudPlatform/terraformer/releases/download/{{ $params.TerraformerVersion }}/terraformer-aws-linux-amd64
dest: /usr/local/bin/terraformer
mode: 0755
- name: "Make AWS config files"
shell: |
set -e
mkdir "{{ $params.WorkDir }}/.aws"
touch "{{ $params.WorkDir }}/.aws/credentials"
touch "{{ $params.WorkDir }}/.aws/config"
chown -R 7000:7000 "{{ $params.WorkDir }}/.aws"
chmod 0700 "{{ $params.WorkDir }}/.aws"
chmod 0600 "{{ $params.WorkDir }}/.aws/credentials"
beforeSetup:
- name: "Write AWS credentials"
shell: |
set -e
printf "[default]\noutput = %s\nregion = %s\n" "{{ $params.AWSDefaultOutput }}" "{{ $.AWSRegion }}" >> "{{ $params.WorkDir }}/.aws/config"
printf "[default]\naws_access_key_id = %s\naws_secret_access_key = %s\n" "{{ $.AWSAccessKeyId }}" "{{ $.AWSSecretAccessKey }}" >> "{{ $params.WorkDir }}/.aws/credentials"
- name: "Create region directories"
file:
path: "{{`{{ item }}`}}"
state: directory
owner: 7000
group: 7000
mode: 0775
with_items:
- "{{ $params.WorkDir }}/regions/{{ $.Environment }}"
- "{{ $params.WorkDir }}/states/{{ $.Environment }}"
setup:
- name: "Init"
shell: |
terraform init
args:
chdir: "{{ $params.WorkDir }}"
become: true
become_user: terraform
beforeSetupCacheVersion: "{{ $.AWSRegion }}-{{ $.AWSAccessKeyId }}"
setupCacheVersion: "{{ $.Environment }}-{{ $.AWSRegion }}-{{ $.AWSAccessKeyId }}"
Now it is time to build an image (do not forget to provide the AWS IAM user credentials). You can specify them manually each time you run the command or define them in the CI environment:
# werf build
CI_ENVIRONMENT_SLUG="eu" AWS_SECRET_ACCESS_KEY="XXX" AWS_ACCESS_KEY_ID="YYY" AWS_REGION="eu-central-1" werf build --stages-storage :local
Start the container using the image we built and the same variable values. Note that the difference in values will result in a configuration mismatch error. Since we interact with the existing infrastructure through abstraction layers, you must use the same parameters we used for building the image:
# werf run terraform
CI_ENVIRONMENT_SLUG="eu" AWS_SECRET_ACCESS_KEY="XXX" AWS_ACCESS_KEY_ID="YYY" AWS_REGION="eu-central-1" werf run --stages-storage :local --docker-options="--rm -ti -w /opt/terraform/ -u terraform" terraform -- /bin/bash
The implementation
Initialization
The Terraform initialization stage for installing the provider plugin can be included in the image. Yes, the size of the image will grow. However, you will get a fully functional image you can repeatedly use (which is essential for our main task).
To pre-install a plugin, create a providers.tf
file and put it to the corresponding directory in the image:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.25"
}
}
}
provider "aws" {
profile = "default"
}
Now, initialize the Terraform configuration. If the provider plugin is installed during the building stage, the init
command goes last (and you do not need to invoke it manually).
~$ terraform init
Initializing the backend...
~~~
Terraform has been successfully initialized!
Importing
Now it is time to import the resources:
~$ terraformer import aws --path-pattern="{output}/" --compact=true --regions=eu-central-1 --resources=elasticache,rds
aws importing region eu-central-1
aws importing... elasticache
~~~
aws importing... rds
~~~
aws Connecting....
aws save
aws save tfstate
In our example, we use the following keys:
aws
— the provider name;--path-pattern
— the path to the configuration files generated by the utility;--compact=true
— writing configurations for all types of resources into one file. Note that the description of resources, variables, states will be saved to different files;--regions
— the provider’s region;--resources
— types of resources to import.
Change the current directory to the one created by the import utility, and run init
in this directory (to initialize the imported resource configuration):
~$ cd generated/
~/generated$ terraform init
Initializing the backend...
~~~
Terraform has been successfully initialized!
Now it is time to do some planning with the imported configuration:
~/generated$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.terraform_remote_state.local: Refreshing state...
~~~
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# aws_db_instance.tfer--db1 will be updated in-place
~ resource "aws_db_instance" "tfer--db1" {
~~~
+ delete_automated_backups = true
~~~
+ skip_final_snapshot = false
~~~
}
# aws_db_instance.tfer--db2 will be updated in-place
~ resource "aws_db_instance" "tfer--db2" {
~~~
+ delete_automated_backups = true
~~~
+ skip_final_snapshot = false
~~~
}
Plan: 0 to add, 2 to change, 0 to destroy.
Planning revealed inconsistencies between the state detected during import and the current resource configuration. And Terraform let us know that it is going to change two aws_db_instance
resources. This can be, for example, due to a difference of available sets of instructions of the import utility and Terraform. This situation is quite normal given that the tools we use are developed by different vendors, and one is an abstraction on top of the other. Also, there may be some lag in the available functionality.
In our case, the imported resource configuration and the tfstate
file are missing the following two parameters in aws_db_instance
resources:
delete_automated_backups
skip_final_snapshot
To resolve conflicts, let’s find out the current values of these parameters using one of AWS interfaces: web-gui or aws-cli. Add these parameters to the resource config file:
~/generated$ vim resources.tf
delete_automated_backups = "false"
~~~
skip_final_snapshot = "false"
~/generated$ vim terraform.tfstate
"delete_automated_backups": "false",
~~~
"skip_final_snapshot": "false",
… and run the planning for the imported configuration once again:
~/generated$ terraform plan
Refreshing Terraform state in-memory prior to plan...
This time, the planning completes successfully:
0 to add, 0 to change, 0 to destroy
Well, the preparatory steps for the adaptation of resources are complete, and the configuration is ready to be applied.
Adding to and merging configurations
There may be situations when you need to update the resource configuration.
Let’s imagine, for example, that we need to add another type of resource to the configuration. To do this, switch to the source directory and import the configuration of this resource type. Note that the path-pattern
directory should be different from the previous one since Terraformer does not rename files used for saving the import results. In other words, it will simply overwrite the existing files if you forget to change the directory.
Let’s add the EC2 resources with the NodeRole=node
tag to our imported configuration using the --filter
key:
~/generated$ cd ../
~$ terraformer import aws --path-pattern="./ec2/" --compact=true --regions=eu-central-1 --resources=ec2_instance --filter="Name=tags.NodeRole;Value=node"
aws importing region eu-central-1
aws importing... ec2_instance
Refreshing state... aws_instance.tfer--i-002D-03f57062-node-1
Refreshing state... aws_instance.tfer--i-002D-009fc9e6-node-2
Refreshing state... aws_instance.tfer--i-002D-0b4b4c7c-node-3
aws Connecting....
aws save
aws save tfstate
Switch to the directory with the imported config and run the terraform init
command:
~$ cd ec2/
~/ec2$ terraform init
Initializing the backend...
Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding hashicorp/aws versions matching "~> 3.29.0"...
- Finding latest version of -/aws...
- Installing -/aws v3.29.0...
- Installed -/aws v3.29.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.29.0...
- Installed hashicorp/aws v3.29.0 (signed by HashiCorp)
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.
* -/aws: version = "~> 3.29.0"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Perform the planning:
~/ec2$ terraform plan
Make necessary changes (in the manner mentioned above) to the config so that the planning completes successfully:
0 to add, 0 to change, 0 to destroy
… and merge resources.tf
files by copying and pasting configuration blocks.
The tfstate
files are merged using the built-in Terraform command, state mv
. Note that Terraform can only add the state to the target file for a single resource at a time. Therefore, if you have several resources, you need to repeat this operation for each one of them.
~/ec2$ cd ../generated/
~/generated$ terraform state mv -state=../ec2/terraform.tfstate -state-out=terraform.tfstate 'tfer--i-002D-03f57062-node-1' 'tfer--i-002D-03f57062-node-1'
Applying the configuration
It’s time to apply the configuration. You must ensure that no changing or destroying (per Terrafrom’s terminology) tasks are planned. After all, even minor and harmless changes can cause the instance to restart. And in some cases, the resource can even be recreated from scratch. That is why you must strive for the following cherished line: “Plan: 0 to add, 0 to change, 0 to destroy.”
~/regions/eu$ terraform apply
~~~
Terraform will perform the following actions:
Plan: 0 to add, 0 to change, 0 to destroy.
~~~
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Migrating the state to S3
You can upload the tfstate
file to S3 storage for convenience since Terraform will handle all further actions with this state file. The beauty of storing the file in S3 is that it will be blocked during use. As a result, a team of engineers can work simultaneously on a massive infrastructure without the risk of conflicts when launching critical operations. Also, with S3, you can store the file reliably (and version it, too).
Migrating the tfstate
file starts with the creation of a separate directory. It will contain the configuration files intended for creating resources responsible for storing the state.
Let’s create a provider configuration first:
provider "aws" {
region = "eu-central-1"
}
terraform {
required_providers {
aws = {
version = "~> 3.28.0"
}
}
}
… and the aws_s3_bucket
configuration. This resource provisions the dedicated storage in S3:
resource "aws_s3_bucket" "terraform_state" {
bucket = "eu-terraform-state"
lifecycle {
prevent_destroy = true
}
versioning {
enabled = true
}
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
}
It uses the following parameters:
bucket
— the name of the storage directory;lifecycle { prevent_destroy = true }
— prevents the deletion of the resource;versioning { enabled = true }
— enables version control for theterraform.tfstate
file on the part of the S3 storage;server_side_encryption_configuration
— encryption settings.
Let’s also add the aws_dynamodb_table
resource to the config. This AWS DynamoDB table stores information about terraform.tfstate
locks:
resource "aws_dynamodb_table" "terraform_locks" {
name = "eu-terraform-locks"
billing_mode = "PAY_PER_REQUEST"
hash_key = "LockID"
attribute {
name = "LockID"
type = "S"
}
}
Here, we have:
name
— the name of the table for storing information about locks;billing_mode
— in thePAY_PER_REQUEST
billing mode, the payment amount is calculated based on the number of requests to the table;hash_key
— the primary key of the table.
Initialize the configuration, then run the plan
and apply
commands:
~$ terraform init
Initializing the backend...
~~~
Terraform has been successfully initialized!
~/states/eu$ terraform plan
Terraform will perform the following actions:
# aws_dynamodb_table.terraform_locks will be created
# aws_s3_bucket.terraform_state will be created
Plan: 2 to add, 0 to change, 0 to destroy.
~/states/eu$ terraform apply
Terraform will perform the following actions:
# aws_dynamodb_table.terraform_locks will be created
# aws_s3_bucket.terraform_state will be created
Plan: 2 to add, 0 to change, 0 to destroy.
aws_dynamodb_table.terraform_locks: Creating...
aws_s3_bucket.terraform_state: Creating…
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Switch to the directory containing the AWS resource configuration and create a backend.tf
file in it with the following lines:
~$ cd ../../regions/eu
~$ vim backend.tf
terraform {
backend "s3" {
bucket = "eu-terraform-state"
key = "terraform.tfstate"
region = "eu-central-1"
dynamodb_table = "eu-terraform-locks"
encrypt = true
}
}
Any attempt to do something with the current resource configuration will result in an error:
Backend reinitialization required. Please run "terraform init".
Reason: Initial configuration of the requested backend "s3"
To get rid of it, you need to re-run the initialization. Terraform will connect to the S3 backend created earlier and request the terraform.tfstate
file. Since the new backend is empty, it will prompt if you want to copy the local state file to S3:
~/regions/eu$ terraform init
Initializing the backend...
Do you want to copy existing state to the new backend?
Pre-existing state was found while migrating the previous "local" backend to the
newly configured "s3" backend. No existing state was found in the newly
configured "s3" backend. Do you want to copy this state to the new "s3"
backend? Enter "yes" to copy and "no" to start with an empty state.
Enter a value: yes
Releasing state lock. This may take a few moments...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Terraform has been successfully initialized!
And, lastly, plan the final version of the configuration:
~/regions/eu$ terraform plan
Refreshing Terraform state in-memory prior to plan...
~~~
No changes. Infrastructure is up-to-date.
Summary
We have successfully adapted the configuration of the target resources. You can save it to a Git repository to enable versioning and (some) decentralization. The state file is also kept in the S3 backend with versioning and session locking enabled.
All the steps discussed in this article can help you re-create the infrastructure in the shortest possible time if some incident destroys your environment.
Of course, these measures are only a tiny part of those required to fail-proof your application. You also need to keep in mind dynamic data (and their recovery), as well as the ability to migrate between different providers, among other things.
Conclusion
You can import the resource configuration of a particular cloud provider in many ways. The practice shows that any approach involves customization, and you must tailor it to suit your specific tools. Nevertheless, Terraformer helped us to complete the task. Of course, it took us time to research, implement, and debug the process. But once it was complete, we were able to import resource configurations of several clusters within the planned time frames and without any problems on the infrastructure’s side.
When planning such tasks, you need to consider the rationale (or even test the feasibility) of such an import, given the efforts involved. Chances are, for a small number of resources, you’d instead write Terraform configurations from scratch or import the resources one by one using the standard tools and then adapt them. However, automation helps when a large number of resources are involved. With it, you can perform a seemingly impossible task.
Comments